Your voice agent is fast and still feels wrong. Here is why

Almost always turn-taking rather than raw speed. A fast agent that interrupts you mid sentence or leaves a beat of dead air feels worse than a slightly slower one that takes its turn correctly. Chasing the latency number first is how teams spend weeks and change nothing a caller notices.

JarvisBitz

The complaint is always the same and rarely precise. The voice agent works, it answers correctly, somebody has driven the latency down, and it still feels wrong to talk to. The team goes back to shaving milliseconds because that is the number they have, and the next version feels equally wrong while costing more to run.

In our experience the problem is almost never raw speed. It is turn-taking: deciding when the caller has finished speaking and when the agent may start. Get that wrong and a fast agent is worse than a slow one, because it interrupts.

Two failures, both called slow

Listen to recordings of a bad voice agent and you will hear the same two things, neither of which is latency.

Cutting in. The caller pauses to think, says "my account number is...", and the agent starts talking over them. This happens when end of turn is decided purely on silence duration. Human speech is full of pauses that are not turn endings, and a threshold short enough to feel responsive is short enough to catch every one of them.

Dead air. The caller finishes, clearly, and nothing happens for a beat. Then the agent replies. The gap reads as the machine not understanding, which is the impression you were trying to avoid. This is the same threshold set the other way, and teams oscillate between the two for weeks because they are tuning one number against two failures.

A silence timer cannot separate these, because the information that distinguishes them is not in the silence. It is in what was said before it. "My account number is" is obviously unfinished. "That is all, thanks" is obviously finished. Both can be followed by exactly the same 400 milliseconds of quiet.

What we measured about latency

None of which means latency is irrelevant. It matters, and it behaves worse than most teams assume, which is worth showing because it changes how you test.

We ran a live session against a realtime speech API over a websocket harness and recorded time to first audio on every turn. Over seven sequential turns, time to first audio rose from 649ms on the opening turn to 1242ms by the seventh, roughly a doubling, purely from accumulating conversation context. Enabling a sliding context window changed the curve to 726ms rising to 1037ms: slightly slower to start, considerably flatter, and the middle of the conversation stopped degrading.

The practical consequence is that a latency figure taken from turn one is not a measurement of your system, it is a measurement of your system at its best moment. If your test harness opens a session, asks one question and records the number, it will report a latency your callers rarely experience. Measure at turn ten.

What to fix, in order

• Decide end of turn on meaning, not only on silence. Use a model that considers what was said, and treat the silence timer as a fallback rather than the decision. This single change removes most cut-ins.
• Do not stack a fixed wait on top of it. Adding a safety timer over a semantic decision does not make it safer, it just adds delay to a decision that was already made.
• Handle barge-in explicitly. When the caller talks over the agent, stop speaking immediately and discard the rest of the planned utterance. An agent that finishes its sentence while being interrupted is the single most irritating behaviour in this category.
• Then compress context. Once turn-taking is right, keep latency flat across the session using a sliding window, and re-measure at turn ten rather than turn one.
• Fill the gap honestly if you cannot close it. A short acknowledgement while a slow tool call runs beats silence, provided it is true. Filler that plays on every turn becomes noise within two calls.

Why this is usually cheaper than it sounds

Teams delay this work because it sounds like a rebuild, and it rarely is. Endpointing is a component with a boundary: audio and partial transcript in, a decision out. Swapping how that decision is made touches the turn controller and nothing else, and it can be trialled on a share of calls with the old behaviour one config change away.

The genuinely expensive version is the one teams talk themselves into instead, which is replacing the speech stack, changing provider, or rearchitecting around a lower latency model because latency was the number on the dashboard. That is months, and callers do not notice.

How to know it worked

Latency numbers will not tell you, so measure the thing you actually care about. Sample real calls and count two rates: how often the agent started speaking while the caller was still going, and how often more than a set gap elapsed after the caller genuinely finished. Those two rates move in opposite directions when you tune a silence threshold, which is precisely why tuning it alone never converges, and they can both fall once the decision is made on meaning.

Also listen to ten calls yourself, end to end, before and after. The metrics tell you whether it improved; only listening tells you whether it is pleasant, and pleasantness is the thing being bought.

If you have a voice agent that is fast and still not landing, our free AI audit includes listening to real calls and separating the turn-taking problems from the latency ones, and how we build voice systems covers the architecture we use.

Related reading

Can a voice agent handle Hinglish and regional accents?

Not as well as the demo suggests, and the gap is measurable. Recognition accuracy degrades substantially on code-switched speech compared to a single language, which matters because everything downstream reads the transcript. Scope it by measuring on your own calls before anyone promises a number.

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

A2A explained: when AI agents need to talk to each other

Most businesses do not need it yet. A2A standardises how agents built by different teams or vendors delegate work to each other, which only starts to matter once you have more than one agent and the other one is not yours to change.