Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

JarvisBitz

A retrieval system that worked in testing starts giving wrong answers in production, and the first instinct is to improve retrieval. Better embeddings, a reranker, smaller chunks. Teams spend a quarter on this and the wrong answers persist, sometimes worse than before, which is confusing enough that people conclude the technology is not ready.

The uncomfortable part: improving retrieval can make wrong answers more convincing rather than less. A better retriever brings back passages that are more topically relevant to the question, which makes a wrong synthesis of them read more plausibly and cite more confidently. You have improved the input and degraded the reader's ability to notice the output is wrong.

Three failures that look identical from outside

Every wrong answer looks the same to the person receiving it. Internally they are three different problems with three different fixes, and the first job is telling them apart.

Retrieval failure. The passage containing the answer was never fetched. This is the one everyone assumes and the easiest to test: take the wrong answers, find the correct source passage by hand, and check whether it was in the retrieved set. If it was, retrieval is not your problem, and every hour spent on embeddings is wasted.

Context assembly failure. The right passage was retrieved and something about how the context was assembled defeated it. It was truncated. It sat below four less-relevant chunks. It was one of three retrieved passages that contradict each other, because your corpus contains a current policy and two superseded ones and nothing in the pipeline knows which is which. This is the most common failure in real deployments and the least discussed, because it is unglamorous and specific to your data.

Faithfulness failure. Everything needed was present, correctly ordered, and the model still produced something the passages do not support. It blended two sources, carried over a detail from its own training, or answered a slightly different question than the one asked.

Why testing did not catch it

Test sets are usually written by the people who built the system, from questions they knew the corpus answered, phrased the way the corpus phrases things. Real users ask about the gap between two documents, use internal shorthand, and ask questions where the honest answer is that the corpus does not say.

That last category is the one that matters most and is almost never tested. A system that has no correct answer available will produce something anyway unless you have built it to decline, and declining is a capability that has to be designed, not a behaviour that emerges.

Score faithfulness separately, and act on it

The fix that changes outcomes most is measuring whether an answer is supported by the passages that were retrieved, as a separate score from whether retrieval found good passages. Two different numbers, computed independently, because they fail independently.

Faithfulness is checkable without a golden answer, which is what makes it practical in production. Decompose the answer into claims and check each against the retrieved text. Anything unsupported is either a fabrication or a leak from model knowledge, and both are things you want to know about a live system.

Then wire it to a behaviour rather than a dashboard. Below a threshold, the system should not answer: it should say what it could not confirm and offer what it did find. Teams resist this because abstention feels like failure, but a system that declines two percent of the time is far more usable than one that is confidently wrong two percent of the time, because the second kind cannot be trusted anywhere.

What good looks like afterwards

Worth stating, because teams fix this and cannot tell whether it worked. A healthy retrieval system has three properties, and none of them is a higher accuracy number.

It declines sometimes, at a rate you chose rather than a rate that emerged. It cites passages a reader can check, and when someone does check them the passage genuinely supports the claim. And its failures are boring: it says it could not find something, rather than inventing a policy that sounds like yours.

That last one is the real test of whether the work landed. Before, the failure mode is a confident invention that a subject expert has to catch. After, the failure mode is an honest gap that anyone can see. The second is a system a business can actually deploy, because it fails in a direction people can handle.

The specific things worth checking first

• Contradictory sources in the corpus. Current and superseded policies sitting side by side with nothing marking which is which. Extremely common after a document migration, and it makes retrieval quality irrelevant.
• Chunk boundaries cutting answers in half. Check whether the answers you get wrong sit near a boundary. Tables and lists suffer worst.
• Permissions applied after retrieval instead of before. Filtering results a user should not see, after ranking, silently changes what the model sees. We cover why this matters in RAG versus long context.
• Freshness. If the index is rebuilt nightly and the question is about this morning, the failure is architectural, not model behaviour.
• No abstention path. If there is no code path where the system says it does not know, it will never say it.

If you have a retrieval system producing confident wrong answers and no clear read on which of the three failures you have, our free AI audit includes classifying a sample of real failures, which is usually enough to redirect the work. How we build retrieval systems covers the architecture.

Related reading

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.

RAG permissions: why your assistant leaks and how to stop it

Because a vector index has no idea who is asking, and the filter is the easy half. The hard half is keeping permissions synchronised with a source of truth that changes during the working day, which is where the leak window opens and why this never shows up in testing.