Document extraction accuracy, and the threshold that ships it
A single accuracy number for a document is close to meaningless, because accuracy varies enormously by field type. Headers are easy, line items and long alphanumeric fields are not, and the thing that makes extraction production safe is a per field confidence threshold routing the uncertain ones to a person.
Ask how accurate AI document extraction is and you will be given a single percentage, usually high, usually measured on clean documents. It is not a useful number, and building a business case on it is how automation projects end up quietly staffed by the people they were meant to free.
Accuracy is not a property of a document, it is a property of a field. On the same invoice, the total is nearly always right and the line items are where things break. A system reported as ninety five percent accurate can be effectively unusable if the five percent is concentrated in the fields your process depends on.
Accuracy by field type, roughly in order
The pattern is consistent across document types, and it follows from how the underlying models work rather than from any particular vendor.
| Field type | Why it behaves that way | Typical outcome |
|---|---|---|
| Header fields: dates, totals, supplier name | Appear once, in predictable places, with strong surrounding context | Reliable enough to automate |
| Structured single values: PO number, invoice number | Distinctive format, usually labelled | Reliable, watch for two candidates on one page |
| Line items and tables | Repeating structure, variable row counts, columns that wrap and merge across pages | The main source of error |
| Long alphanumeric strings: IBAN, VAT, serials | Character level accuracy required, no semantic context to self correct against | Error prone in a specific, checkable way |
| Anything conditional: applies only if a clause exists | Requires reading comprehension across the document, not extraction | Least reliable, needs review |
Two of these deserve elaboration because they cause most real incidents.
Line items fail structurally, not randomly. A table that continues over a page break, a row where the description wraps to two lines, a credit line with a negative quantity. The model does not misread a character, it misunderstands the shape, and the result is a plausible table with one row missing. Nothing about the output looks wrong.
Long alphanumeric fields fail predictably. Zero against capital O, one against lowercase l, five against S. Predictability is good news: these are the fields where a checksum or format rule catches the error before a human ever sees it, and where validation beats any amount of model improvement.
The evaluation trap
Most teams evaluate extraction by having a model judge whether the output looks right. Without ground truth, that only measures plausibility, and plausibility is exactly what a wrong extraction has. A missing line item looks entirely reasonable to a judge that never saw the original table.
You need a set of documents where a human has recorded the correct values, field by field, including the awkward ones. A hundred documents chosen to include the messes, rather than a thousand clean ones, will tell you more. This is unglamorous work and it is the only thing that turns a demo into a system you can size a team around.
Report accuracy per field, never as one number. "Ninety eight percent on header fields, eighty one percent on line items, and here is the confusion pattern on account numbers" is a sentence a business can plan against. "Ninety five percent accurate" is not.
The threshold is what makes it shippable
The mistake is treating extraction as a system that must be right. Treat it instead as a system that must know when it might be wrong, and route those cases to a person. That single reframing is the difference between a pilot and a production process.
Have the extractor emit a confidence per field, then set a threshold per field type rather than one global number, because the cost of an error differs wildly across the document. A wrong supplier name is embarrassing. A wrong bank account is money gone.
- Auto accept above the threshold, with the confidence recorded for later audit.
- Route to review below it, showing the human the extracted value beside the source region so the decision takes seconds rather than minutes.
- Always review the fields where an error is expensive, regardless of confidence, and accept that permanently. This is the same reversibility and blast radius logic we use for agent approval gates.
- Validate before you trust confidence. Checksums, date sanity, totals that must equal the sum of lines. A failed arithmetic check is worth more than any confidence score.
Setting the threshold honestly
Do not choose it from intuition. Run the labelled set, and for each candidate threshold compute two things: what proportion of documents pass without review, and what proportion of auto accepted values are wrong. Those two move against each other, and the right threshold is a business decision about which error rate is tolerable, not a technical one.
Then state the result as a staffing number, because that is what everyone actually wants to know. At this threshold, this share of documents needs a human, so this is the review load at your volume. A project justified by removing a whole team, that turns out to need a third of one, is still a good project if you knew before starting.
Recheck it after a month of live traffic. Real documents are messier than any sample, and the threshold that looked right on the labelled set usually moves once.
If you want the labelled set built and the per field thresholds derived from your own documents rather than a vendor benchmark, that is what our free AI audit establishes, and how we build document pipelines covers the review workflow around it.
Related reading
Get this applied to your business.
The free AI audit measures your live setup and shows where AI would actually pay off.