AI Gives Wrong Answers Despite Having Documents: Where Should You Investigate?
A practical guide to diagnosing wrong AI and RAG answers across source documents, ingestion, chunking, embeddings, retrieval, reranking, prompts, citations, and evaluation.

When AI answers wrongly even though the business has provided documents, it does not mean the model cannot read them. The fault may lie in the source documents, ingestion, chunking, metadata, embedding, retrieval, reranking, the prompt, permissions, memory, citations, or generation.
A common mistake is to switch models or edit the prompt immediately. If the right chunk is never retrieved and placed in the final prompt, the model has almost no chance of answering correctly.
Core principle
Before changing the model, prove that the correct document was ingested, the correct chunk was retrieved, and the final prompt actually contained the right evidence.
Investigation
Classify the failure as data, retrieval, or generation
Do not investigate the whole system at once.
Pick one specific question that was answered wrongly, establish the correct answer and the source passage that supports it, then trace the full pipeline for exactly that question.

Classification
Three main failure groups
Practical checks for isolating the exact layer that caused the wrong answer.
Data failure
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Retrieval failure
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Generation failure
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Are the documents correct, current, and consistent?
RAG cannot fix wrong data.
Source checks
What to check
Practical checks for isolating the exact layer that caused the wrong answer.
Version
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Effective date
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Scope
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Contradictions
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Parsing
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Language
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
A successful upload is not necessarily a successful index
A file can appear in the UI without any chunks in the vector store.
Check the document ID, checksum, ingestion status, chunk count, embedding status, namespace, tenant scope, and error log. You must be able to trace from the original file to every chunk in the index.
Ingestion checks
Evidence required
Practical checks for isolating the exact layer that caused the wrong answer.
Chunk count
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Embedding status
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Namespace
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Checksum
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Partial failure
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Timestamp
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
The right chunk may be split badly
Chunks that are too small lose context; chunks that are too large reduce retrieval precision.
Chunk diagnostics
Common mistakes
Practical checks for isolating the exact layer that caused the wrong answer.
Splitting mid-sentence or mid-table
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Lost headings
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Overlap too low
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Chunks too large
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Chunks too small
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Missing metadata
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
A wrong query rewrite can pull retrieval away from the intent
Log both the original question and the rewritten one.
Check query normalisation, language detection, conversation context, query expansion, and metadata filters. Pay special attention to Vietnamese typed without diacritics, abbreviations, and questions that depend on the previous turn.
Query checks
What to log
Practical checks for isolating the exact layer that caused the wrong answer.
Original query
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Rewritten query
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Conversation context
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Language
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Filters
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Is the right chunk in the Top-K?
This is the most important question when debugging RAG.

For every wrong answer, look at the retrieved chunks, similarity scores, ranks, filters, and why a chunk was excluded. If the right chunk does not appear in the Top-K, the fault lies before generation.
Retrieval checks
Variables to check
Practical checks for isolating the exact layer that caused the wrong answer.
Embedding model
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Hybrid search
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Top-K
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Threshold
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Metadata filter
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Freshness
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Retrieval can be right while the wrong context is still sent
Reranking, de-duplication, and token limits can drop an important chunk.
Context checks
Frequently missed errors
Practical checks for isolating the exact layer that caused the wrong answer.
Reranker score
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Duplicate chunks
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Token truncation
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Bad ordering
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Formatting loss
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Missing citation ID
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
The context is right but the model still answers wrongly
Only then investigate the prompt and the model.
Generation checks
Common causes
Practical checks for isolating the exact layer that caused the wrong answer.
Vague system prompt
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
No abstention rule
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Context conflict
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Temperature cao
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Prompt injection
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Synthesis error
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Stale memory or wrong permission filters can skew results
It looks like weak retrieval but has a different cause.
Check summary memory, pending state, user profile, tenant, branch, role, locale, and document ACLs. Never drop a permission filter just to increase recall.
Access checks
Scope layers to verify
Practical checks for isolating the exact layer that caused the wrong answer.
Tenant
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Branch or property
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Role
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Document ACL
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Environment
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Memory TTL
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
A relevant citation does not necessarily support the claim
Score citation correctness and answer correctness separately.
Citation checks
Evaluation criteria
Practical checks for isolating the exact layer that caused the wrong answer.
Source relevance
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Claim support
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Placement
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Authority
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Version
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Coverage
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Without a test set, errors come back
One manually corrected answer does not prove the system improved.

Metrics
Minimum evaluation set
Practical checks for isolating the exact layer that caused the wrong answer.
Answer correctness
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Recall@K
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
MRR or nDCG
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Citation correctness
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Faithfulness
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Abstention
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Leakage
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
An investigation sequence for one wrong answer
Go layer by layer so you do not fix the wrong place.
Step 1
Pick one concrete case
Practical checks for isolating the exact layer that caused the wrong answer.
Question
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Expected answer
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Expected source
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Current response
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Step 2
Pipeline monitoring
Practical checks for isolating the exact layer that caused the wrong answer.
Ingestion
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Retrieval
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Reranking
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Final prompt
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Response
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Step 3
Fix and regression-test
Practical checks for isolating the exact layer that caused the wrong answer.
Change one thing
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Rerun evaluation
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Check side effects
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Save evidence
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Add regression case
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Investigation
Fixes that usually do not work
They can hide the symptom without solving the root cause.
Anti-patterns
Mistakes to avoid
Practical checks for isolating the exact layer that caused the wrong answer.
Switching models immediately
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Raising Top-K without limit
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Stuffing in every document
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Only editing the prompt
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Not logging chunk IDs
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
No regression set
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Each investigation step should keep the query, chunk IDs, scores, prompt, response, and evidence to pinpoint where the answer first went wrong.
Investigation
How FlowNexa helps evaluate and improve RAG
Find the failing layer and prove quality with evaluation.
Services
Support scope
Practical checks for isolating the exact layer that caused the wrong answer.
RAG Quality Assessment
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Knowledge Base Review
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Retrieval Tuning
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Evaluation Framework
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Observability
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Production Remediation
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
FlowNexa's view
Before changing the model, prove that the correct document was ingested, the correct chunk was retrieved, and the final prompt actually contained the right evidence.
Investigation
Frequently asked questions
Answers focused on how to investigate in practice.
FAQ
Quick answers
Practical checks for isolating the exact layer that caused the wrong answer.
What should we check first?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Does a stronger model help?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Is a higher Top-K always better?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
What is the right chunk size?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Does a correct citation mean a correct answer?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Can FlowNexa audit our RAG system?
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
FlowNexa AI & RAG
Find the real cause before changing models
Practical checks for isolating the exact layer that caused the wrong answer.
Book a RAG consultation
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.
Request a RAG Quality Assessment
Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.


