What to log in an LLM system so you can debug it later

A 200 response tells you almost nothing about an AI system, because the failure you care about is a confident wrong answer that returned successfully. Log the inputs, the retrieved context, every tool call and the guardrail outcomes, or you will be debugging from screenshots.

JarvisBitz

A 200 used to mean the request was fine. In an AI system it means the request completed, which is almost unrelated to whether it was fine. The failures that matter return successfully: a confident wrong answer, a tool called with plausible but incorrect arguments, a retrieval that found nothing and got answered anyway. None of them appear in an error rate.

Which is why the first serious production incident on an AI feature is usually investigated from screenshots a customer sent, because the system recorded that it worked. What follows is the schema we use, framework agnostic, written as fields rather than tools.

Per request

• A request id that spans everything, including every model call, tool call and retrieval underneath it. Without one you have fragments and no way to reassemble a single user interaction.
• The full prompt as sent, after templating, not the template. The difference between what you think you sent and what you sent is where a good share of bugs live.
• The full completion as received, before any parsing or post processing.
• Model identifier and version, exactly as the provider reported it back, not the string in your config. Providers move aliases and your config will lie to you eventually.
• Tokens in and out, and computed cost, which is what makes attribution possible at all.
• Latency split into queue, model and tools, because "it was slow" has three different fixes.
• The user or tenant, and the feature, as identifiers you can join on.

Per tool call

This is where most systems are thinnest and where agent investigations actually happen.

• Which tool, with the exact arguments the model produced.
• What came back, including errors, truncated with the original length recorded rather than silently cut.
• Where the result came from. Internal system, third party, user supplied content. This is the provenance field that tells you later whether a bad step was fed by something outside your control, which is the practical hook for investigating prompt injection.
• Whether it changed anything, and the identifier of what it changed. When you need to reverse an agent's afternoon, this field is the difference between an hour and a week.
• Retry count, since silent retries are how one logical action becomes four writes.

Per retrieval

• The query as issued, which is often a rewritten version of what the user typed.
• Document ids and scores of what came back, in order.
• Which chunks actually made it into the prompt after truncation, which is frequently not all of them and is the failure described in why RAG gives wrong answers.
• Index version, so you can tell whether a regression followed a reindex.

Per guardrail

Log the checks that passed, not only the ones that fired. A guardrail with no passing records is indistinguishable from a guardrail that is not running, and that is a distinction you want to make quickly.

• Which check, what it decided, and the score if it produced one.
• What happened as a result: allowed, blocked, rewritten, escalated to a human.
• For anything that went to a person, what they were shown and what they decided.

Redaction, which decides whether you can keep any of this

The schema above is only useful if you are permitted to hold it, and full prompts and completions are the most sensitive data your system touches. Redaction has to happen at capture, not at query time, because anything else means the raw values were written to disk and the redaction is cosmetic.

• Redact on the way in, in the logging layer, before anything is persisted or forwarded.
• Tokenise rather than delete where you can, so a support conversation is still traceable without holding the values.
• Two tiers of retention. Metadata, costs, latencies, decisions and ids can be kept long, because that is what trend analysis needs. Full content bodies should have a short clock, measured in days or weeks, because their value decays fast and their risk does not.
• Sample rather than keep everything, once volume is real. Full capture on a small percentage plus complete capture of anything that errored or was escalated gives you most of the investigative value at a fraction of the exposure.
• Know where the logs physically are, which is the point we make in data residency: an observability vendor is a processor of everything your model saw.

Make it queryable, not just stored

A detail that decides whether any of this pays off: these records are only useful if you can ask questions across them. Store the fields as structured columns rather than a serialised blob in one text field, because the questions worth asking are aggregate ones. Which feature regressed after Tuesday's deploy. Which tool fails most often. Whether cost per task has crept up over the month.

Teams that log everything into an unstructured blob have technically captured the data and practically cannot use it, and discover this during the incident when it matters most.

Where to start if none of this exists

Do not attempt the whole schema at once. The request id and full prompt and completion, at a sample rate, on your highest traffic feature, gets you most of the way in an afternoon. Add tool call provenance next if you run agents, because that is the field you will wish you had during the first incident. Everything else can follow.

If you want this designed once so it holds as features multiply, how we run AI systems in production covers the instrumentation we build, and a free AI audit will tell you which of these fields your current setup is missing.

Related reading

LLM data residency: what to configure and what to verify

Residency is a configuration you set, not a property you inherit, and the defaults differ per vendor and change without notice. Establish four things in writing for each provider, then enforce the rest at your own gateway, because that is the only layer you control.

LLM spend: attribution before optimisation

You cannot cut what you cannot see. Most teams optimise models and prompts before they can attribute a single dollar to a team, feature or customer, which means they are guessing. Meter at the gateway first, then cache, route and cap in that order.

Your model is being retired. How to move without regressions

The API call will port in an afternoon. What breaks is behaviour: the same prompts producing subtly different output, and you cannot tell without something to compare against. The migration is a measurement problem, and the teams that find it easy built the measurement before the email arrived.