Should cameras run inference on the device or in the cloud?

Bandwidth and what happens when the link drops decide it, more often than model quality does. Work out what leaving the site costs before comparing accuracy, because at real frame rates the network is usually the binding constraint and it settles the argument.

JarvisBitz

The edge or cloud question usually gets argued on model quality, which is the wrong axis and produces a stalemate, because cloud hardware genuinely runs bigger models and everyone knows it. The decision is nearly always settled earlier by something less interesting: what it costs to get the pixels off the site, and what happens when you cannot.

Do that arithmetic first. In most real deployments it ends the discussion before accuracy is mentioned.

Start with the bandwidth, not the model

A camera at ordinary resolution and frame rate produces a continuous stream, and streaming it somewhere is a recurring cost that scales linearly with how many cameras you have and how long they run. Multiply your camera count by hours of operation and you have the number that usually decides this.

Two things make it worse than the first estimate. Sites with cameras are frequently sites with poor connectivity, because they are factories, warehouses, car parks and remote installations rather than offices. And upload is the direction that is always cheaper and slower on whatever connection exists.

The four things that force the edge

Where any of these holds, the general purpose model in the cloud is not a slightly worse option, it is unavailable, and the useful conversation moves straight to what a smaller model needs in training data. That conversation is the same one we set out in vision API or a custom model, and the honest answer is usually that labelling is the real budget.

What the cloud is genuinely better at

The argument is not one sided, and pretending otherwise leads teams to put hardware in places that did not need it.

• Anything low volume or intermittent. A handful of images a day does not justify hardware, installation or a maintenance visit.
• Anything semantic. Describing a scene, judging whether something looks damaged, reading a mixed document. Larger models are meaningfully better and these tasks are rarely per frame.
• Anything still changing. Updating a cloud model is a deploy. Updating a hundred devices in the field is a project, and it is the cost nobody includes.
• Anything where the site is a person with a phone, which is a camera you do not own and cannot install onto.

The arrangement that survives a real site

Most working systems we see are neither, and the split follows the constraints rather than a preference.

A small model runs on or beside the camera doing the continuous narrow job: detect, count, measure, and decide whether anything interesting happened. Almost everything it sees is uninteresting and is discarded locally, which is the entire point, because it collapses the bandwidth problem to a trickle. When something is interesting, or when the local model is uncertain, that single frame or short clip goes up for a heavier look, and from there to a person if it is still unclear.

This is confidence routing applied to compute placement, the same pattern as the review thresholds in extraction accuracy. It also gives you a graceful degradation story: when the link drops, the local model keeps deciding and queues what it would have escalated.

The pilot that decides wrongly

A pattern worth naming because it repeats. The pilot runs on two cameras in an office with good connectivity, streams to the cloud, works beautifully, and the decision is made. The rollout is forty cameras across three sites with a connection that drops most afternoons, and the architecture that was chosen cannot survive there.

Two cameras is not a small version of forty cameras, it is a different problem, because the constraint that decides this scales and the pilot did not include it. If the eventual deployment is at a site you have not tested on, the pilot has told you the task is feasible and nothing about where the compute belongs.

Run at least one pilot camera at a real site on the real connection, even if the model runs in the cloud during the trial. It is a day of work and it produces the number the whole decision turns on.

What people underestimate about the edge

Being fair to the cloud, three costs consistently get left out of edge proposals.

• Deployment and updates. Getting a new model onto devices in the field, reliably, with a way to roll back. This is a real system, and it needs building before the second model version exists rather than after.
• Observability across a fleet. Knowing that camera 47 has been quietly failing for a fortnight. On a single cloud service this is free; across a fleet it is work.
• Physical reality. Devices get unplugged, overheat, get moved, and get pointed at a wall by someone rearranging a room. Plan for a non trivial rate of that.

How to decide this week

• Count cameras, hours and resolution, and price continuous upload for a year against edge hardware for the same period.
• Ask how often the site loses connectivity, from someone who works there rather than from the network diagram.
• Write down whether the decision must happen between frames. If yes, stop here, it is edge.
• Check whether the imagery is allowed to leave the site at all, before designing anything.
• If it is genuinely marginal, prototype in the cloud, because it is faster to build and you can move it later once you know the task is well defined.

If you want that arithmetic run against your actual camera count and connectivity rather than a reference architecture, our free AI audit covers it, and how we build vision systems covers the hybrid and the fleet management around it.

Related reading

Document extraction accuracy, and the threshold that ships it

A single accuracy number for a document is close to meaningless, because accuracy varies enormously by field type. Headers are easy, line items and long alphanumeric fields are not, and the thing that makes extraction production safe is a per field confidence threshold routing the uncertain ones to a person.

Vision API or a custom model? A decision rule that holds

Use a general vision API for semantic work at modest volume, and train a specialist when the task is geometric, high throughput, real time or on device. The split is not capability in the abstract, it is whether you are asking what is in the picture or exactly where and how much.

Is RAG still needed now that context windows are huge?

Yes, for most business systems. Long context solved the cases where all your data fits and you can afford to resend it every call. Retrieval still wins on cost, freshness, permissions, and being able to show where an answer came from.