How we took a SaaS AI assistant from 30 second answers to under 3, across 40,000 PDFs

Latency and accuracy in a large RAG system are mostly decided before the model is called: which intent the request has, which documents the user may see, how little context you can get away with, and what you cache. Fixing those four took first token latency from 10 to 30 seconds to under 3 and lifted accuracy to roughly 98% on the evaluated use cases.

JarvisBitz

The client is a SaaS company in the US automotive space. I will keep them anonymous, but the shape of the problem will be familiar to anyone who added an AI assistant to a product in the first wave. They had done it early, and they had done it seriously: the assistant was meant to help customers find information, work out what they wanted, respond to it, spot sales opportunities in the conversation, search a very large document library, and eventually talk out loud.

By the time we were involved it did most of those things badly. It behaved like a fixed workflow with a chat window on the front. It often misread what the customer was asking. And depending on the question, an answer could take anywhere from 10 to 30 seconds to appear, which in a chat interface feels like the product has died.

The knowledge base was the reason for the last part. More than 120 GB of material, roughly 40,000 PDF documents, and the original system was searching far too much of it for every single question.

What we found when we looked at the latency

Everyone's first instinct with a slow assistant is to blame the model. In this case the model was maybe a fifth of the problem. Most of the 10 to 30 seconds was spent before any text generation started, and it broke down into four habits that reinforced each other.

• Retrieving too much. Every question pulled a wide set of passages from across the library, because nobody had decided what kind of question it was first.
• Sending too much. All of those passages went into the prompt. Bigger prompts mean slower generation and higher cost, and they also make the model worse at picking out the passage that actually matters.
• Checking permissions late. Which documents a given user was allowed to see was handled after retrieval, which meant retrieving material only to throw it away, and in a few cases meant material that should have been filtered leaking into the context.
• Caching nothing. Two people asking the same question an hour apart paid the full cost twice.

The intent problem and the latency problem turned out to be the same problem. The system did not know what the customer wanted, so it retrieved everything, and retrieving everything is slow.

What we rebuilt

We redesigned the system around a custom retrieval augmented generation pipeline with agentic orchestration on top, running on Google Vertex AI with Gemini, in Python, with speech to text and text to speech for the voice path. The important decisions were not the tools. They were the order things happen in.

Intent first, then retrieval

The single biggest change was putting intent detection at the front. The business defined the intents that mattered to them, with example phrasings for each, and every incoming message is classified against those by meaning before anything else happens. Once the system knows it is looking at, say, a question about a specific product's specification rather than a general browse, retrieval can be targeted at the right slice of the library instead of the whole thing.

This is also where the lead generation logic lives. Certain intents carry commercial signal. When one of those fires, the assistant can move naturally from answering into asking a qualifying question or two, and route the conversation on. A customer who simply wants a fact gets the fact and is left alone.

Permissions inside the search, not after it

Different users on this platform are entitled to different documents. The original design retrieved first and filtered second, which is the common pattern and the wrong one. We moved authorization into the retrieval step itself, so the search only ever runs over documents the current user may see. It is faster, because there is less to search, and it closes a category of leak that is very hard to test for from an admin account, because admins can see everything and every answer looks fine.

Less context, chosen better

With intent known and the candidate set already narrowed by authorization, context construction becomes a question of how little you can send rather than how much. We were fairly aggressive here. The prompt carries the passages that answer the question and not much else. This is where a lot of the latency went, and it is also where a surprising amount of the accuracy came from: a model reading three relevant passages is more reliable than a model reading thirty, most of which are noise.

The numbers

Two honest notes on those figures. The 98% is measured on the set of use cases the client defined and evaluated, and I would not present it as a universal rate across every possible question. And "under three seconds" is time to the first token of the answer in typical interactions, which is what a person in a chat window actually feels; long answers still take longer to finish streaming.

The cost reduction was mostly a side effect. Smaller prompts and cached answers mean fewer tokens, and fewer tokens is the whole cost model.

The voice path

The same engine sits behind a voice interface, with speech to text on the way in and text to speech on the way out. Voice is far less forgiving of latency than text; a three second pause in a spoken conversation is an eternity. The work described above is what made a voice path viable at all. Without the intent routing and context trimming, the assistant could not have answered fast enough to hold a conversation.

What I would tell a team starting this today

• Find out what kind of question you are answering before you go looking for the answer. Intent detection is not a nice to have on top of RAG; it is what makes retrieval targeted.
• Put permissions inside retrieval from day one. Retrofitting it is painful and the leak you are preventing is one your own testing will never find.
• Measure where the time actually goes before touching the model. In our experience the model is rarely the largest slice.
• Keep humans in charge of the decisions that matter. The assistant here answers questions, collects details and routes leads on its own. It does not close deals or change records without a person in the loop, and that is a design choice, not a limitation.

The hard part of enterprise AI is rarely calling the model. It is the retrieval, the permissions, the context management, the caching, the latency controls and the verification around it. That surrounding system is what we build, and it is the difference between an assistant that demos well and one that runs inside a real product with a substantial production user base. If you have an assistant that is slow, wrong, or both, the free AI audit is a reasonable place to start, and this page describes how we approach retrieval systems in general.

Related reading

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.