Can a voice agent handle Hinglish and regional accents?

Not as well as the demo suggests, and the gap is measurable. Recognition accuracy degrades substantially on code-switched speech compared to a single language, which matters because everything downstream reads the transcript. Scope it by measuring on your own calls before anyone promises a number.

JarvisBitz

Every voice platform demo handles Hindi. The demo is a clean recording, one speaker, one language, a quiet room, and a script. Your callers switch between Hindi and English inside a single sentence, from a street, on a phone, while someone else talks nearby.

The gap between those two is not a matter of degree, it is measurable and it is the thing nobody quotes you. Worth being specific about, because the whole system sits on top of the transcript.

What code-switching costs, measured

A speech corpus paper published in July 2025 states it plainly: "ASR models experience a 30-50% increase in Word Error Rate (WER) when exposed to code-switched speech compared to monolingual input". The same work reports that "standard models trained on monolingual data underperform by approximately 42% WER on our test set".

Sit with the second number for a moment. Roughly two words in five wrong, on speech that a human listener would follow without effort. That is not a system with an accent problem, it is a system reading a different sentence from the one that was spoken.

Why it breaks more than the transcript

Speech recognition errors do not stay in the transcript. Everything downstream reads it as though it were what the caller said.

• Intent classification degrades silently. The model classifies confidently on a misheard sentence, so a wrong route looks like a wrong decision rather than a wrong input.
• Entity capture fails where it matters most. Names, amounts, addresses and reference numbers are exactly the fields with the least surrounding context to self correct against, and exactly the fields you cannot get wrong.
• Retrieval misses. A misheard product name does not find the right document, and the assistant answers about something else entirely, fluently.
• The caller adapts, badly. People who are misunderstood start speaking unnaturally, slowly and in one language, which is a worse experience than a form and is frequently reported as the agent being rude.

How to scope this honestly

The mistake is accepting a vendor accuracy figure and designing as though it held for your callers. Two steps prevent most of the pain.

Measure on your own recordings before anyone promises a number. Take two hundred real calls, have a bilingual person transcribe them properly, and run your candidate systems against that. Two hundred calls is a few days of work and it replaces the entire argument, because you will have a number for your callers rather than for a benchmark.

Measure per field, not overall. Overall accuracy hides the shape of the problem, and the shape is what you design around. This is the same discipline as extraction accuracy: the headline number is nearly meaningless and the per field breakdown tells you what to build.

Designing around it rather than pretending

• Confirm the expensive fields, always. Read back the amount, the account, the address. Every voice system should do this and one handling code-switched speech must.
• Constrain where you can. Recognising a reference number against a list of plausible values is a far easier problem than recognising it from open speech, and most systems know the candidate set.
• Route on confidence, not just on intent. When recognition confidence is low the correct behaviour is to hand to a person, not to guess and proceed. This is the same threshold logic used elsewhere, applied to the transcript.
• Let people finish switching languages. Turn-taking gets harder with code-switching, because a pause mid switch looks exactly like a pause at the end of a turn. We cover why silence timers cannot tell these apart in voice agents and turn-taking.
• Offer the exit early. A caller who has repeated themselves twice should be with a human, and the count is trivial to implement.

What can actually be improved

This is not a fixed constraint, and it is worth knowing which levers exist before accepting a vendor number as the ceiling.

Models trained or adapted on code-switched speech perform materially better than models trained on either language alone, which is the reason corpora like the one above exist. Adaptation on a few hours of your own audio is frequently available and frequently overlooked, and your own calls are the most representative data anyone could train on.

Beyond the model, the cheapest gains are usually in the vocabulary. Product names, place names and the specific terms your callers use are the words that break most often and the easiest to supply in advance, because most recognition systems accept a domain vocabulary or a bias list. That change costs an afternoon and it lands on exactly the fields that matter.

Do this before concluding the technology is not ready. The gap between an off the shelf system and an adapted one on the same audio is usually larger than the gap between vendors.

What to ask a vendor

Three questions, and the answers separate the serious from the confident.

• What is your word error rate on code-switched speech specifically, and on what data was it measured?
• Can I run your system against my own recordings before signing, and what does that cost?
• What happens on low confidence: do you guess, flag, or escalate?

A vendor who has genuinely worked on this will give you a number and its conditions, and will want you to test on your own audio because that is how they win the comparison. A vendor who answers that their model supports Hindi has answered a different question.

If you are scoping a voice system for callers who switch languages and want the measurement run on your own recordings first, our free AI audit covers it, and how we build voice systems covers the confirmation and escalation design that makes the residual error survivable.

Related reading

Your voice agent is fast and still feels wrong. Here is why

Almost always turn-taking rather than raw speed. A fast agent that interrupts you mid sentence or leaves a beat of dead air feels worse than a slightly slower one that takes its turn correctly. Chasing the latency number first is how teams spend weeks and change nothing a caller notices.

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

A2A explained: when AI agents need to talk to each other

Most businesses do not need it yet. A2A standardises how agents built by different teams or vendors delegate work to each other, which only starts to matter once you have more than one agent and the other one is not yours to change.