Do you need a knowledge graph? Usually schema and filters

Usually not. Most problems that look like they need a graph are solved by a schema and metadata filters on the retrieval you already have. Graphs earn their cost on genuine multi-hop questions, where the answer requires connecting facts across documents rather than finding the right passage.

JarvisBitz

Knowledge graphs come up at a predictable moment. The retrieval system works, someone asks a question it handles badly, and a suggestion appears that the real fix is a graph. It is an appealing idea because it sounds like the grown up version of what you already have.

Worth noticing who is answering. Most of the material recommending graphs is published by companies that sell graph databases or build on them, and they are answering a question they have a structural interest in answering yes. That does not make them wrong, and it does mean the honest version of the question, whether you need one, is under-served.

In our experience the answer is usually no, and the reason is that the problem people are describing is nearly always a metadata problem wearing a graph costume.

What people actually mean when they ask

The complaints that prompt this are consistent, and most of them have a cheaper fix.

• "It returns the wrong version of the policy." That is a metadata problem. Tag documents with effective dates and status, filter on them at query time. No graph required, and if you do not fix it a graph will faithfully connect you to the superseded document.
• "It mixes up our products." Metadata. Tag by product line and filter.
• "It does not know that this customer is part of that group." This one might be a graph, and might equally be a field on a record you already have in your CRM.
• "It cannot answer questions that span documents." This is the real case, and it is the only one on the list that a graph is genuinely for.

The case where a graph earns its cost

Multi-hop questions are the genuine article. The answer is not in any one passage, it has to be assembled by following relationships: which suppliers are affected by a component from a factory in a particular region, or which contracts inherit a clause from a master agreement that was amended twice.

Neo4j put the limitation of pure vector search fairly, that it "lacks awareness of how facts are connected", which is true and worth bounding. It is a real limitation for questions where the connections are the answer, and irrelevant for questions answered by one good passage, which is most of them.

The research position has also moved in a way that matters here. An April 2026 benchmark comparing the two for agentic search found that "agentic search substantially improves dense RAG and narrows the performance gap to GraphRAG", while concluding that "GraphRAG remains advantageous for complex multi-hop reasoning, exhibiting more stable agentic search behavior when its offline cost is amortized".

Read those two findings together and the practical guidance falls out. Letting a retrieval system search iteratively, refining its query based on what it found, closes much of the gap that graphs were being recommended to close. What survives is genuine multi-hop reasoning, and even there the advantage is conditional on the setup cost being amortised, which means it depends on volume.

What a graph actually costs

The cost is not the database, which is the part people price. It is everything around it.

• Extraction. Somebody has to turn documents into entities and relationships, and doing that with a model produces a graph containing the model's mistakes, structured and confident.
• Schema design. Deciding what counts as an entity and which relationships matter is domain modelling, it is slow, and it is difficult to change once populated.
• Keeping it current. Every document change is potentially a set of edge changes. This is where graph projects quietly stall, because the maintenance burden is ongoing while the build was a project.
• Ambiguity. Deciding that two similarly named entities are or are not the same thing is unglamorous, endless, and determines whether the graph is trustworthy.

Indexing cost, the thing usually cited as the blocker, has genuinely improved, and we note the published figures in RAG versus long context. That removes one objection and leaves the four above, which are the ones that actually determine whether the project finishes.

The order to try things in

Cheapest first, because each step frequently ends the conversation.

• Add a schema. Tag documents with the handful of attributes your questions actually filter on: date, status, product, entity, owner.
• Filter on them at retrieval, before the search rather than after, which is the same discipline required for permissions.
• Let retrieval run more than once, refining based on what came back. This is where the recent gains are.
• Then, if the remaining failures are genuinely relational, build a graph for that narrow set of questions rather than for the whole corpus.

That last point is the one worth holding onto. The version of this that works is a small graph covering the relationships that matter, sitting beside ordinary retrieval, not a programme to model your entire business as a graph. The latter is where budgets go to die, and it is usually what gets proposed.

If you have a list of questions your retrieval handles badly and want to know which are metadata problems and which are genuinely relational, that classification is what our free AI audit produces, and how we build knowledge graphs covers the narrow version we recommend when the answer is yes.

Related reading

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.