Should you fine-tune a model on your company documents?

Almost certainly not. Fine-tuning shapes behaviour and format, it does not reliably install facts. Most requests to train a model on internal documents are a retrieval or prompting problem wearing a more expensive hat.

JarvisBitz

The request arrives phrased as a conclusion rather than a question: we want to fine-tune a model on our documents so it knows our business. It sounds like the serious option, the one beyond prompting, and it is usually the wrong one for a reason that has nothing to do with budget.

Fine-tuning adjusts how a model behaves. It is unreliable at installing what a model knows. Those are different operations, and almost every request we receive to train on internal documents is a description of the second while asking for the first.

The distinction that saves the money

Say the sentence out loud with the emphasis moved and the answer usually appears.

"The model does not know our returns policy." That is a knowledge problem. The policy is a document, it changes, and when it changes you need the system to change with it the same day. Retrieval solves this and a trained model does not, because the policy is now baked into weights and the next revision requires another training run. We go through the constraints that keep retrieval necessary in RAG versus long context, and they all apply here.

"The model does not answer the way we answer." That is a behaviour problem, and it is what fine-tuning is genuinely for. Tone, structure, house terminology, a consistent output shape, knowing which of your five ticket categories a message belongs in. These are patterns rather than facts, and patterns are what training transfers well.

What people are usually really hitting

When a system feels like it does not know the business, three causes are more likely than model capability, and all three are cheaper to fix.

• The retrieval is finding the wrong passages, or the right ones are being assembled badly. Diagnosing this properly is its own piece: why RAG gives wrong answers that sound right.
• The prompt does not say what good looks like. Two or three real examples of a correct answer in the prompt routinely close most of the perceived gap, and cost an afternoon.
• Nobody has written down the rules. Frequently the business logic the model is failing to follow has never been stated anywhere, and the exercise of writing it down for a prompt is what actually fixes things.

Work through those before training, in that order, because each one is reversible and a trained model is a commitment.

The costs that do not appear in the quote

Fine-tuning has a price beyond the training run, and it is mostly paid later.

• Your knowledge freezes. Whatever the model absorbed is true as of the training data. Facts drift, and nothing tells you when.
• You are pinned to a base model. When a materially better one appears, you retrain to move, and you will want to move more often than you expect.
• You need an evaluation set to know if it worked, and building one honestly is often more work than the training. Without it you have a vibe.
• The dataset becomes a maintained asset. Someone owns it, curates it, and answers for what is in it, which is a permanent job rather than a project.

The combination that actually ships

Framing this as a choice is part of what makes teams pick wrong. The systems that work in production usually do both, for the two different problems, and the split is clean.

Retrieval supplies the facts, freshly, from documents that are edited by the people who own them. Training, if used at all, shapes how the model uses those facts: the format it returns, the register it writes in, when it declines. The knowledge changes daily without a training run, and the behaviour stays consistent without a longer prompt.

Ordering matters. Build retrieval first and get it right, because it is the part that makes the system correct. Only then look at whether behaviour still needs work, and by that point the answer is often that the remaining gap is small enough for a few examples in the prompt to close. Teams that train first spend the budget before they know which half of the problem they had.

When training is the right call

It is not never, and the cases are recognisable.

• A narrow, high volume classification that runs constantly, where a small trained model is faster and cheaper than a large general one and the labels are stable.
• An output format that must be exact, every time, where prompting gets you close and close is not sufficient.
• A genuinely specialist register that general models handle poorly, and where you have real examples rather than an aspiration.
• Latency or cost at scale, where a smaller trained model does one job well enough to displace a larger one, and you have measured that rather than assumed it.

Notice that none of these is about the model knowing your documents, and that all of them require labelled examples you already have. If the training data would have to be created first, that creation cost is the real project.

If you are being quoted for a fine-tune, the useful question is which of the two problems it solves, and what happens when the underlying documents change next month. Our free AI audit will tell you which half of your problem is knowledge and which is behaviour, and how we approach fine-tuning covers the cases where we recommend it.

Related reading

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.

Why your RAG gives wrong answers that sound right

Because retrieval and faithfulness are different failure modes and most teams only measure the first. A better retriever can make wrong answers more convincing, not less, so faithfulness has to be scored separately and low scoring answers routed to abstention.

RAG permissions: why your assistant leaks and how to stop it

Because a vector index has no idea who is asking, and the filter is the easy half. The hard half is keeping permissions synchronised with a source of truth that changes during the working day, which is where the leak window opens and why this never shows up in testing.