Vision API or a custom model? A decision rule that holds

Use a general vision API for semantic work at modest volume, and train a specialist when the task is geometric, high throughput, real time or on device. The split is not capability in the abstract, it is whether you are asking what is in the picture or exactly where and how much.

JarvisBitz

General purpose vision models describe images remarkably well, which is why the first instinct on any new vision project is now to send a photograph to an API and read the answer. For a real class of problems that is the right call and it saves months. For another class it produces a demo that impresses everyone and then fails quietly in the field.

The split is not capability in the abstract. It is whether you are asking what is in the picture or asking exactly where and how much.

The finding that makes the rule concrete

A team at EPFL benchmarked multimodal foundation models against standard computer vision tasks, in work titled "How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks". Their summary of the pattern is the useful part: these models "perform semantic tasks notably better than geometric ones".

On object detection specifically, they report that while the best general model led the other general models, all of them performed below the specialist detection models they were measured against. That is the shape of the trade in one sentence: excellent at meaning, weaker at measurement, and a specialist still wins where precision is the product.

Semantic against geometric, in practice

A useful reformulation: if a competent person could answer from a verbal description of the image, a general model will probably do well. If they would need to look closely and measure, it will not.

The three constraints that override the rule

Even for semantic work, three things push you to a custom model regardless of how good the general one is.

• Throughput. Per image API pricing is fine at thousands per month and becomes the dominant line item at millions. This is the same arithmetic as build versus buy for agents: work out your real volume, multiply by the published per image price, and compare against a trained model serving on hardware you already have.
• Real time. If a decision has to happen in the time between one frame and the next, a network round trip is not available to you. This alone decides most production line and robotics work.
• On device. Cameras in places with poor connectivity, or images that should not leave the site at all. A small trained model runs there; an API does not.

The hybrid most teams end up with

The framing as a binary is itself part of the problem. The arrangement that tends to survive contact with production uses both, on the grounds that they fail differently.

A cheap specialist model runs on every frame doing the narrow geometric job it was trained for: find the object, measure the thing, count the items. When it is uncertain, or when something unexpected appears, that frame goes to a general model for a semantic second opinion, and from there to a person if it is still unclear. You get the precision and throughput of the specialist with a sensible fallback for the long tail it was never trained on, which is exactly where specialists are weakest.

This is the same confidence routing pattern as document extraction thresholds, and the design work is the same: decide what confidence means, and what happens below it.

The labelled data question, asked honestly

Every custom model conversation eventually reaches training data, and it is where projects quietly stall, so it is worth asking early and plainly.

The question is not whether you have images. Most operations have years of them. It is whether anyone has recorded what is in them, in the form the model needs. Photographs of finished parts are not labels. Bounding boxes drawn around defects, by someone who agrees with the next person about what counts as a defect, are labels, and creating them is usually the largest line in a custom vision budget.

A general model can help here rather than compete with it: use it to pre-label a set and have a person correct rather than draw from scratch, which is considerably faster. That is often the most valuable role for a general vision API on a project that ultimately needs a specialist.

How to decide in an afternoon

• Write the question as a sentence. If it contains a unit of measurement, you are in geometric territory.
• Multiply real monthly image volume by the current published per image price. Compare against a trained model on hardware you control.
• Ask where the camera is and whether the answer is needed before the next frame.
• Check whether you have labelled examples, or would have to create them. That creation cost is usually the real budget for a custom model.
• Prototype the semantic version first regardless. It is a day of work and it tells you whether the task is well defined, which is the more common failure.

If you want that call made against your actual images and volume, our free AI audit covers it, and how we build vision systems covers what a specialist model needs to reach production.

Related reading

Document extraction accuracy, and the threshold that ships it

A single accuracy number for a document is close to meaningless, because accuracy varies enormously by field type. Headers are easy, line items and long alphanumeric fields are not, and the thing that makes extraction production safe is a per field confidence threshold routing the uncertain ones to a person.

Should cameras run inference on the device or in the cloud?

Bandwidth and what happens when the link drops decide it, more often than model quality does. Work out what leaving the site costs before comparing accuracy, because at real frame rates the network is usually the binding constraint and it settles the argument.

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.