Your model is being retired. How to move without regressions

The API call will port in an afternoon. What breaks is behaviour: the same prompts producing subtly different output, and you cannot tell without something to compare against. The migration is a measurement problem, and the teams that find it easy built the measurement before the email arrived.

JarvisBitz

The email arrives with a date on it. A model you are using in production is being retired, there is a recommended replacement, and you have a few months. The engineering reaction is usually relief, because changing a model identifier is a one line change and the API is identical.

That relief is the problem. The call ports in an afternoon. What does not port is behaviour, and behaviour is what your users experience. The same prompt against a newer model produces output that is different in ways that are individually small and collectively noticeable, and nothing in your test suite will fail.

What actually changes

Model swaps produce a recognisable set of regressions, none of which raise an error.

• Format drift. The new model formats lists differently, or adds a preamble your parser did not expect, or stops emitting a field it used to include unprompted.
• Verbosity shifts. Longer answers change cost per call and change how a UI looks, and both get noticed by different people in different meetings.
• Refusal boundaries move. Content the old model handled is now declined, or the reverse. In a support context this is a visible change in behaviour toward customers.
• Tool calling changes. Argument shapes, how often it decides to call at all, how it handles an ambiguous instruction. For agents this is the largest source of surprise.
• Prompt sensitivity shifts. Instructions carried for a year because they fixed something in the old model may now do nothing, or actively hurt.

The uncomfortable part is that most of these are improvements in isolation. The new model is usually better. It is just different from the thing your prompts, parsers and users were calibrated against.

Six months is less than it sounds

Providers do give notice. OpenAI's deprecations page currently commits to "at least 6 months" for generally available models, which is genuinely reasonable and still shorter than it feels once it collides with everything else on the roadmap.

Worth looking at the actual schedule rather than the policy, because it reframes how you should build. As of 8 September 2026 that page lists a January 2027 shutdown for a realtime model and a February 2027 shutdown for a speech to text model, alongside a set of October 2026 retirements. It also lists gpt-5-2025-08-07 shutting down on 11 December 2026, which is a model retired inside about sixteen months of its own release.

The structural fix, which is boring and works

Two changes turn this from a project into a task, and both are cheap to make before you need them and expensive afterwards.

Never let a provider model identifier appear in application code. Application code should ask for a capability, something like the summarisation model or the extraction model, and one place maps that to a concrete provider and version. When the email arrives you change one mapping rather than grepping a codebase, and you can point a percentage of traffic at the new model without touching a feature. If you already route through a gateway for cost attribution, the same layer does this.

Record the model version the provider actually reported on every call, rather than the string you asked for. Aliases move underneath you, and when behaviour changes on a day you did not deploy, that field is the difference between a diagnosis and a theory. This is one line in the schema we set out in what to log.

How to actually run the migration

The method is the one we use for any behavioural change, described in full in testing an agent before production, and it works here because a model swap is exactly that.

• Replay before you switch. Take real production requests from the last few weeks and run them against both models. You are not scoring correctness yet, you are looking at where the two disagree.
• Read the disagreements by hand. Fifty of them, personally. This is the step people skip and it is the one that finds the format change nobody thought to assert on.
• Check the parsers, not just the prose. Anything downstream that expects structure is where a silent regression becomes a broken feature.
• Shadow, then canary. Run the new model alongside on live traffic with its output unused, then move a small share of real traffic with the mapping ready to revert.
• Re-check cost after, not before. Verbosity changes move token counts, and the new model being cheaper per token does not mean your bill falls.

The thing that makes all of this easy

Every step above depends on having something to compare against. Teams that find migrations routine have an evaluation set built from their own real cases, and teams that find them frightening do not, so they are reduced to asking whether it seems fine.

That set is also the asset you should own rather than rent, which we go into in what you should own when the build ends. If you have never built one, a deprecation notice with a date on it is a good reason to start, because you are going to need it again next year.

If you want the indirection layer and the replay harness in place before the next email, that is what our AI gateway work covers, and a free AI audit will tell you how many model identifiers are currently hardcoded across your codebase, which is usually more than anyone expects.

Related reading

LLM data residency: what to configure and what to verify

Residency is a configuration you set, not a property you inherit, and the defaults differ per vendor and change without notice. Establish four things in writing for each provider, then enforce the rest at your own gateway, because that is the only layer you control.

LLM spend: attribution before optimisation

You cannot cut what you cannot see. Most teams optimise models and prompts before they can attribute a single dollar to a team, feature or customer, which means they are guessing. Meter at the gateway first, then cache, route and cap in that order.

What to log in an LLM system so you can debug it later

A 200 response tells you almost nothing about an AI system, because the failure you care about is a confident wrong answer that returned successfully. Log the inputs, the retrieved context, every tool call and the guardrail outcomes, or you will be debugging from screenshots.