Which LLM should we standardise on? Do not standardise

The question assumes standardising is the goal, and it is not. Benchmarks will not tell you which model is best at your task, models get retired on roughly annual cadence, and the cost of switching is what actually matters. Evaluate on your own data and keep at least two viable behind one gateway.

JarvisBitz

The question arrives sounding like good engineering discipline. We should pick one model, standardise on it, build expertise, negotiate a rate. It is how organisations have sensibly handled databases and cloud providers for twenty years, and it is the wrong instinct here.

Standardising on one model is not a decision you make once. It is a migration you have scheduled without noticing, because the model you standardise on has a retirement date you have not been told yet.

Why the usual selection process does not work

Most model selection is done by reading benchmark tables, which is the closest available thing to evidence and is nearly useless for the decision in front of you.

Benchmarks measure general capability on public tasks. You are asking a narrow question: which model is best at extracting line items from your suppliers' invoices, or handling your customers' phrasing, or calling your six tools in the right order. Two models with near identical benchmark scores routinely differ substantially on any one of those, and the ranking on your task is frequently not the ranking on the leaderboard.

The things that decide production suitability barely appear in benchmarks at all: how reliably a model produces parseable structured output, how it behaves when a tool returns an error, latency at your prompt length rather than a short one, and how often it refuses something ordinary in your domain. These are cheap to test and nobody publishes them for your workload because nobody has your workload.

The reason standardising fails, specifically

Even if you evaluated correctly and picked genuinely the best model today, the commitment decays on a clock you do not control.

We go through this in moving off a retired model: providers give reasonable notice, and models are still being retired within about sixteen months of release. A single model commitment means that when the notice arrives you are performing a migration you have never rehearsed, on someone else's deadline, with no second option already integrated.

What to do instead

Evaluate on your own data. Take a hundred real cases from your actual workload, with correct answers recorded by someone who knows the domain. Run every candidate against them. This takes a couple of days, costs very little, and produces the only ranking that matters. It also gives you the evaluation set you will need for every subsequent model decision, which is why the work compounds rather than being spent.

Measure the operational properties too. Structured output reliability across a few hundred calls, behaviour on malformed tool responses, latency at your real prompt length, refusal rate on your domain. A model that is marginally better at the task and noticeably worse at returning valid output is worse overall.

Keep at least two viable, behind one gateway. Not running both in production, but both integrated, both passing your evaluation, and switchable by configuration. The second one is insurance against a deprecation notice, a price change, an outage, and a capability gap you have not hit yet.

Route by task, not by contract. Once traffic flows through a gateway, the expensive model does not have to serve everything. Classification and extraction frequently run on a smaller model at a fraction of the cost with no measurable quality loss, and you can prove that on your own evaluation set rather than guessing. This is the same layer that makes spend attribution possible.

What standardising is actually trying to solve

The instinct is not stupid, and it is worth naming what it is protecting against, because those concerns are real and have better answers.

• Sprawl. Six teams using six providers with six billing accounts is genuinely bad. The fix is one gateway, not one model. Standardise the interface and the governance, and let the model behind it vary.
• Expertise. The transferable skill is prompting, evaluation and system design, not provider trivia. Teams that have built an evaluation set move between models comfortably; teams that have memorised one provider's quirks do not.
• Procurement effort. Real, and front loaded. Doing two vendor reviews once is less work than doing one under time pressure when a deprecation notice lands.
• Data governance. The strongest argument for fewer providers, and it is a reason to be deliberate rather than to commit to one. Each provider is a separate answer on residency and retention, which we set out in data residency.

When one model is genuinely the right answer

Two situations do justify committing, and it is worth naming them so this does not read as blanket advice.

If you are running at a volume where a negotiated commitment materially changes unit economics, the discount can outweigh the flexibility, and that is a calculation rather than a principle. Do it with the switching cost written down as a number, so the trade is explicit.

And if you are deploying into an environment where each additional provider carries a real approval burden, fewer providers is a legitimate constraint. Even then, keep the indirection layer, because the constraint is on how many vendors you contract with, not on whether your application code hardcodes a model name.

The version of the question worth asking

Replace "which model should we standardise on" with three better questions. Which model wins on our evaluation set today. What would it cost us in engineering days to switch if that changed next quarter. And do we have a second model already integrated and passing, or only a name on a slide.

If the answer to the last one is no, that is the work, and it is smaller than it sounds because the evaluation set does most of it.

If you want the evaluation built on your own cases and a gateway that makes the choice reversible, that is what our gateway work covers, and a free AI audit will tell you how deeply one provider is currently embedded in your codebase.

Related reading

LLM data residency: what to configure and what to verify

Residency is a configuration you set, not a property you inherit, and the defaults differ per vendor and change without notice. Establish four things in writing for each provider, then enforce the rest at your own gateway, because that is the only layer you control.

LLM spend: attribution before optimisation

You cannot cut what you cannot see. Most teams optimise models and prompts before they can attribute a single dollar to a team, feature or customer, which means they are guessing. Meter at the gateway first, then cache, route and cap in that order.

What to log in an LLM system so you can debug it later

A 200 response tells you almost nothing about an AI system, because the failure you care about is a confident wrong answer that returned successfully. Log the inputs, the retrieved context, every tool call and the guardrail outcomes, or you will be debugging from screenshots.