Changing your embedding model is a migration, not a setting

You have to re-embed. A different model produces vectors in a different space, so old and new are not comparable and cannot share an index. It looks like a configuration change and behaves like a database migration, which is why it goes wrong so predictably.

JarvisBitz

A better embedding model comes out, someone notices the config file has a model name in it, and the change looks like a one line edit and a deploy. It is not. It is a full data migration with a config file for a disguise, and the reason is geometry rather than engineering practice.

An embedding model places text at coordinates in a space it invented during training. A different model invents a different space. The same sentence lands somewhere else, and the distance between a new query vector and your existing document vectors is not a measure of similarity, it is a measure of nothing. Old and new cannot share an index, and there is no setting that makes them comparable.

The dangerous part is that nothing fails. Change the model, keep the index, and every query still returns ten documents with confident scores attached. They are simply the wrong ten, and the system reports perfect health while quietly becoming useless.

What the migration actually involves

Re-embedding every document is the unavoidable core, and around it sit the parts that surprise people.

• Cost and time proportional to corpus size, not to the size of the change. A million documents is a million embedding calls whether the model improved slightly or enormously.
• Dimensionality frequently differs, so the index schema changes too, and anything that assumed a vector width needs revisiting.
• Everything tuned against the old space becomes suspect. Similarity thresholds, the number of results retrieved, reranker settings. These were fitted to distances that no longer mean the same thing, and carrying them across is how teams conclude the new model is worse.
• Chunking may want revisiting. Models differ in how much text they handle well, and the chunk size that suited the old one is not automatically right.

The cutover that avoids downtime

Do not migrate in place. Build the new index alongside the old one and switch when it is ready, which is the standard blue green pattern and works here for the same reasons.

• Stand up a second index and embed into it while the first keeps serving.
• Run both against your evaluation set and compare properly, rather than spot checking a few queries.
• Re-tune thresholds against the new space before judging quality, because untuned is not a fair comparison.
• Shadow real traffic through both and diff the results, which surfaces the queries that regressed.
• Switch, keep the old index for a while, and be able to switch back.

The bridge, and its limits

There is a genuinely interesting middle option worth knowing about. A September 2025 paper proposes learning a small transform between the old and new embedding spaces rather than re-embedding everything, and reports that it "recovers 95-99% of the retrieval recall (Recall@10, MRR)" while adding "less than 10 microseconds of query latency" and reducing "recompute costs by over 100 times" compared with full re-indexing.

That is a real result and worth reading carefully rather than adopting reflexively. Recovering most of the recall means not recovering all of it, and the gap lands unevenly rather than uniformly. Treat it as a bridge that buys time on a very large corpus, or a way to serve traffic during a long re-embed, rather than a permanent alternative to owning your vectors in one space.

How to stop this being a recurring surprise

The structural fixes are the same ones that make provider changes survivable generally.

• Record the embedding model and version on every stored vector. Without it you cannot tell which documents were embedded with what, and a partially migrated index is worse than either state.
• Keep the source text and the pipeline that produced the chunks. If re-embedding requires re-deriving chunks from originals nobody kept, the migration becomes an archaeology project.
• Budget re-embedding as periodic maintenance, the way you budget model migration. Embedding models improve, and you will want to move at least occasionally.
• Keep thresholds in configuration, not scattered in code, because they will need refitting each time.

None of this makes the migration free. It makes it a planned week rather than an unplanned month, and it prevents the specific failure where nobody realises the index is half in one space and half in another.

The half migrated index

The worst outcome is not a failed migration, it is a partial one. A re-embed that stops halfway, through a timeout, a quota, or somebody cancelling a job, leaves an index holding vectors from two different spaces.

That index is worse than either version of itself. Queries return a blend of correctly and incorrectly placed documents, so results are plausible enough to pass a spot check and wrong often enough to matter, and the pattern of failure looks random because it depends on which space a given document happened to land in.

This is the specific reason to record the model version per vector and to build into a separate index rather than updating in place. A partial build of a new index is simply an incomplete index you have not switched to yet, which is a recoverable state. A partial update of a live one is a corruption you may not detect for weeks.

When not to bother

A better model on a public benchmark is not a reason to migrate. The question is whether it is better on your corpus and your queries, and general improvements frequently do not show up on a narrow domain corpus at all.

Run the candidate against your evaluation set before planning anything. If the improvement is marginal, staying put is the correct engineering decision, and it is worth writing down that you checked so the question does not get reopened every quarter. Where retrieval quality is genuinely the problem, the causes are more often the ones in why RAG gives wrong answers than the embedding model.

If you are weighing an embedding upgrade and want the comparison run on your own corpus before committing to a re-index, that is what our free AI audit covers, and how we build retrieval systems covers the versioning that makes the next one routine.

Related reading

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.