Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

JarvisBitz

Every time context windows get bigger, someone declares retrieval dead. The argument is intuitive: if the model can read a million tokens, why build a pipeline to find the right few thousand? Just send everything.

For a narrow set of problems that is now genuinely the right answer, and it is worth knowing which ones so you stop over engineering them. For most business systems it is not, and the reasons have almost nothing to do with whether the data fits.

What actually changed

Long context removed the crudest reason to build retrieval, which was that a contract or a report simply would not fit. If your entire corpus is a handful of documents and it fits comfortably in the window, a retrieval layer is now overhead you do not need. Skip it. That is a real simplification and it applies to more prototypes than people admit.

What it did not change is everything that happens when a system runs continuously, for many users, over data that keeps moving.

The four constraints that keep retrieval necessary

1. You pay for the window on every single call

Context is not free storage, it is a per request charge. Sending a large corpus with every question means paying for all of it every time somebody asks anything. Retrieval sends the relevant slice instead. At one query a day that difference is invisible. At thousands of queries a day across a support team it is the dominant line on the bill, and it scales with usage rather than with the size of your data.

2. Permissions are per user, not per corpus

This is the one that quietly ends the debate for most companies. If you stuff the whole corpus into the prompt, every user shares one view of your data. The moment two people should see different things, and in any real business they should, you need a layer that filters by identity before the model ever sees the content. That layer is retrieval.

3. Data changes faster than you want to resend it

Prices, stock, policies and tickets change constantly. A retrieval index can be updated for the records that changed. A stuffed context has to be reassembled and resent, and any caching you were relying on to make it affordable is invalidated the moment anything moves.

4. Citations require knowing what you retrieved

If an answer needs to be checkable, and in regulated or high value work it does, you have to be able to point at the source. Retrieval gives you that by construction: you know exactly which passages were used. With a large stuffed window the model can cite, but you have far weaker guarantees about what it actually drew on.

So when should you just use the window?

A good rule: if you are analysing one thing deeply, use the window. If you are answering many questions across many things, use retrieval.

The hybrid most production systems actually use

In practice the two are not rivals. Long context made retrieval systems better, because you no longer have to be miserly about how much you pass through. The pattern that works:

• Retrieve generously, since you can now afford to pass more candidates than the old tight budgets allowed.
• Re-rank the candidates, because relevance still matters more than volume for answer quality.
• Pass a wider slice than you would have two years ago, and let the model do more of the sifting.
• Keep the identity filter in front of all of it, always.

That gives you the accuracy benefit of a bigger window without the cost, permission and freshness problems of abandoning retrieval entirely.

A caution about the benchmark argument

Long context benchmarks usually measure whether a model can find a fact placed somewhere in a large window. That is a genuine capability and it has improved a great deal. It is also a different task from what your users do, which is ask an ambiguous question whose answer is spread across several documents, some of which contradict each other and one of which is out of date.

Test on your own questions and your own corpus before concluding anything from a chart. The gap between benchmark retrieval and messy real retrieval is where most disappointing deployments live.

The short answer

Long context is a real improvement that made one class of problem simpler and made retrieval systems better at the same time. It did not remove the need to control who sees what, to keep costs proportional to usage rather than corpus size, to reflect data that changed this morning, or to show your work.

If you are weighing this for a specific system, read how we structure retrieval end to end, or run the free AI audit to see what your data situation actually looks like before choosing an architecture.

A note on graph based retrieval

Knowledge graph approaches come up whenever retrieval is discussed, usually framed as strictly better and strictly more expensive. The cost half of that has changed. Microsoft Research reported in November 2024 that their LazyGraphRAG variant has "data indexing costs identical to vector RAG and 0.1% of the costs of full GraphRAG", with "more than 700 times lower query cost" than GraphRAG global search at comparable answer quality.

The practical reading is that indexing cost is no longer the reason to avoid graph approaches, so the decision returns to whether your questions are actually relational. Questions that span many documents to assemble an answer benefit. Questions answered by one passage do not, and pay complexity for nothing.

Related reading

Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.

RAG permissions: why your assistant leaks and how to stop it

Because a vector index has no idea who is asking, and the filter is the easy half. The hard half is keeping permissions synchronised with a source of truth that changes during the working day, which is where the leak window opens and why this never shows up in testing.