Skip to content
Back to Blog

AI Gives Wrong Answers Despite Having Documents: Where Should You Investigate?

A practical guide to diagnosing wrong AI and RAG answers across source documents, ingestion, chunking, embeddings, retrieval, reranking, prompts, citations, and evaluation.

AI Gives Wrong Answers Despite Having Documents: Where Should You Investigate?

When AI answers wrongly even though the business has provided documents, it does not mean the model cannot read them. The fault may lie in the source documents, ingestion, chunking, metadata, embedding, retrieval, reranking, the prompt, permissions, memory, citations, or generation.

A common mistake is to switch models or edit the prompt immediately. If the right chunk is never retrieved and placed in the final prompt, the model has almost no chance of answering correctly.

Core principle

Before changing the model, prove that the correct document was ingested, the correct chunk was retrieved, and the final prompt actually contained the right evidence.

Investigation

Classify the failure as data, retrieval, or generation

Do not investigate the whole system at once.

Pick one specific question that was answered wrongly, establish the correct answer and the source passage that supports it, then trace the full pipeline for exactly that question.

Illustration: Classify the failure as data, retrieval, or generation

Classification

Three main failure groups

Practical checks for isolating the exact layer that caused the wrong answer.

Data failure

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Retrieval failure

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Generation failure

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Are the documents correct, current, and consistent?

RAG cannot fix wrong data.

Source checks

What to check

Practical checks for isolating the exact layer that caused the wrong answer.

Version

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Effective date

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Scope

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Contradictions

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Parsing

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Language

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

A successful upload is not necessarily a successful index

A file can appear in the UI without any chunks in the vector store.

Check the document ID, checksum, ingestion status, chunk count, embedding status, namespace, tenant scope, and error log. You must be able to trace from the original file to every chunk in the index.

Ingestion checks

Evidence required

Practical checks for isolating the exact layer that caused the wrong answer.

Chunk count

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Embedding status

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Namespace

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Checksum

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Partial failure

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Timestamp

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

The right chunk may be split badly

Chunks that are too small lose context; chunks that are too large reduce retrieval precision.

Chunk diagnostics

Common mistakes

Practical checks for isolating the exact layer that caused the wrong answer.

Splitting mid-sentence or mid-table

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Lost headings

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Overlap too low

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Chunks too large

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Chunks too small

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Missing metadata

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

A wrong query rewrite can pull retrieval away from the intent

Log both the original question and the rewritten one.

Check query normalisation, language detection, conversation context, query expansion, and metadata filters. Pay special attention to Vietnamese typed without diacritics, abbreviations, and questions that depend on the previous turn.

Query checks

What to log

Practical checks for isolating the exact layer that caused the wrong answer.

Original query

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Rewritten query

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Conversation context

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Language

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Filters

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Is the right chunk in the Top-K?

This is the most important question when debugging RAG.

Illustration: Is the right chunk in the Top-K

For every wrong answer, look at the retrieved chunks, similarity scores, ranks, filters, and why a chunk was excluded. If the right chunk does not appear in the Top-K, the fault lies before generation.

Retrieval checks

Variables to check

Practical checks for isolating the exact layer that caused the wrong answer.

Embedding model

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Hybrid search

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Top-K

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Threshold

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Metadata filter

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Freshness

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Retrieval can be right while the wrong context is still sent

Reranking, de-duplication, and token limits can drop an important chunk.

Context checks

Frequently missed errors

Practical checks for isolating the exact layer that caused the wrong answer.

Reranker score

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Duplicate chunks

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Token truncation

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Bad ordering

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Formatting loss

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Missing citation ID

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

The context is right but the model still answers wrongly

Only then investigate the prompt and the model.

Generation checks

Common causes

Practical checks for isolating the exact layer that caused the wrong answer.

Vague system prompt

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

No abstention rule

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Context conflict

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Temperature cao

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Prompt injection

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Synthesis error

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Stale memory or wrong permission filters can skew results

It looks like weak retrieval but has a different cause.

Check summary memory, pending state, user profile, tenant, branch, role, locale, and document ACLs. Never drop a permission filter just to increase recall.

Access checks

Scope layers to verify

Practical checks for isolating the exact layer that caused the wrong answer.

Tenant

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Branch or property

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Role

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Document ACL

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Environment

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Memory TTL

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

A relevant citation does not necessarily support the claim

Score citation correctness and answer correctness separately.

Citation checks

Evaluation criteria

Practical checks for isolating the exact layer that caused the wrong answer.

Source relevance

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Claim support

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Placement

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Authority

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Version

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Coverage

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Without a test set, errors come back

One manually corrected answer does not prove the system improved.

Illustration: Without a test set, errors come back

Metrics

Minimum evaluation set

Practical checks for isolating the exact layer that caused the wrong answer.

Answer correctness

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Recall@K

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

MRR or nDCG

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Citation correctness

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Faithfulness

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Abstention

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Leakage

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

An investigation sequence for one wrong answer

Go layer by layer so you do not fix the wrong place.

Step 1

Pick one concrete case

Practical checks for isolating the exact layer that caused the wrong answer.

Question

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Expected answer

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Expected source

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Current response

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Step 2

Pipeline monitoring

Practical checks for isolating the exact layer that caused the wrong answer.

Ingestion

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Retrieval

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Reranking

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Final prompt

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Response

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Step 3

Fix and regression-test

Practical checks for isolating the exact layer that caused the wrong answer.

Change one thing

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Rerun evaluation

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Check side effects

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Save evidence

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Add regression case

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Investigation

Fixes that usually do not work

They can hide the symptom without solving the root cause.

Anti-patterns

Mistakes to avoid

Practical checks for isolating the exact layer that caused the wrong answer.

Switching models immediately

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Raising Top-K without limit

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Stuffing in every document

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Only editing the prompt

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Not logging chunk IDs

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

No regression set

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Each investigation step should keep the query, chunk IDs, scores, prompt, response, and evidence to pinpoint where the answer first went wrong.

Investigation

How FlowNexa helps evaluate and improve RAG

Find the failing layer and prove quality with evaluation.

Services

Support scope

Practical checks for isolating the exact layer that caused the wrong answer.

RAG Quality Assessment

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Knowledge Base Review

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Retrieval Tuning

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Evaluation Framework

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Observability

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Production Remediation

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

FlowNexa's view

Before changing the model, prove that the correct document was ingested, the correct chunk was retrieved, and the final prompt actually contained the right evidence.

Investigation

Frequently asked questions

Answers focused on how to investigate in practice.

FAQ

Quick answers

Practical checks for isolating the exact layer that caused the wrong answer.

What should we check first?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Does a stronger model help?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Is a higher Top-K always better?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

What is the right chunk size?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Does a correct citation mean a correct answer?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Can FlowNexa audit our RAG system?

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

FlowNexa AI & RAG

Find the real cause before changing models

Practical checks for isolating the exact layer that caused the wrong answer.

Book a RAG consultation

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.

Request a RAG Quality Assessment

Validate this point with logs, chunk IDs, scores, prompts, citations, and regression tests.