Object Recognition for Cameras: ML + VMS Guide 2026

Object recognition is the cheapest part of a camera analytics platform. Choosing where it runs, how it emits metadata, and which VMS receives the event is the expensive part — and the part vendors do not help with. This playbook is the architecture, hardware, and integration pattern we use to ship custom object recognition into camera systems at Fora Soft in 2026.

Why Fora Soft wrote this playbook

Object recognition is the computer-vision task of finding and naming objects in an image or video stream — drawing a box around each one and labeling it (person, vehicle, forklift, hard-hat). On a camera system, that means turning raw pixels into structured events your software can act on. The models are commodities now; the value is in the plumbing that carries a detection into a decision.

Fora Soft has shipped video products since 2005 — 250+ of them — and object detection on IP camera feeds has been in our stack since MobileNet-SSD on an NVIDIA Jetson TX1. We built VALT, the surveillance and observation platform now used by 770+ US organizations and 50,000+ users, and we have been its sole engineering team for more than a decade. What follows is the architecture we actually use: what we put on the camera, what we run on the gateway, what we push to the cloud, and how we make the result behave like a first-class citizen inside Milestone XProtect, Genetec Security Center, or a custom VMS.

We focus on the 2026 integration reality: YOLO and DETR models running under TensorRT, Hailo HEF, Axis ACAP, or CoreML; events flowing through MQTT or Kafka; metadata emitted as ONVIF Profile T; embeddings stored in Milvus for re-identification. If your use case already has a packaged product (vehicle counting, license plate recognition), buy it. If it has a twist, build it — with the patterns below. For the model-training side of the house, our AI for video engineering guide goes deeper.

What changed in 2024–2026

Three shifts made custom object recognition both easier and harder between 2024 and 2026.

Easier: edge silicon exploded. A Hailo-8 on an M.2 card delivers 26 TOPS at about 2.5 W. Ambarella's CV7 line puts double-digit TOPS inside a camera SoC. Sony's IMX500 runs a small classifier on the sensor itself. And NVIDIA's Jetson Orin Nano Super now does 67 TOPS for $249 — it replaced the old 40-TOPS, $499 kit in December 2024. Inference that needed a GPU server in 2022 now sits behind a PoE port.

Easier: YOLO got faster and simpler. YOLO11 (September 2024) is the stable production default in 2026, and YOLO26 (January 2026) drops non-maximum suppression for end-to-end inference tuned to edge chips, reporting up to +2.5 box AP over YOLO11 on COCO. YOLOv9-E still holds 55.6% mAP for anyone on the older toolchain. The open-source weights are almost always good enough; the model file is not the moat.

Harder: privacy law grew teeth. In 2025 alone France's data regulator, the CNIL, issued 83 sanctions totaling €486.8M; sixteen of them hit employee video surveillance (the Samaritaine department store drew €100,000 for hidden cameras in its stockrooms), and Clearview AI was fined €20M for scraped facial recognition. Custom privacy engineering — face blurring, plate masking, role-based access to raw footage — is now a hard requirement, not a nice-to-have.

Three-tier architecture: camera, gateway, cloud

Every deployment larger than a handful of cameras ends up with the same three-tier pattern. Read it top-down.

Tier 1 — On-camera inference. Lightweight detectors (YOLOv8n, YOLO11n, MobileNet-SSD, purpose-built occupancy classifiers) run inside the camera firmware via Axis ACAP, the Ambarella CVflow SDK, or an on-sensor runtime like Sony IMX500. Output: bounding boxes and class labels emitted as ONVIF Profile T metadata alongside the RTSP stream. Latency: 30–80 ms. Power: 0.5–3 W above the baseline camera draw.

Tier 2 — Gateway aggregation. NVIDIA Jetson Orin, Hailo-8 M.2 accelerators, or an Intel OpenVINO box ingest multiple RTSP streams, run heavier models (YOLOv9, RT-DETR, action recognition), and do cross-camera reasoning: tracking a person across the factory floor, counting dwell time, correlating plate reads with access control. Latency: 80–200 ms end to end. Density: 8–32 streams per Jetson Orin NX.

Tier 3 — Cloud analytics and search. The cloud is the system of record: detection metadata, embeddings, audit logs. It runs the expensive jobs (re-identification across days, forensic search by appearance, dashboards) and nothing real-time. This is also where AWS Rekognition Video, Azure Video Indexer, or Google Cloud Vision slot in when you want a managed service for one specific capability.

The split forces a decision in week one: what stays local, what gets a round trip, and what never leaves the network. Get it wrong and you are either saturating WAN links with raw HD video (the cloud-only mistake) or asking a camera to run a transformer it cannot fit in memory (the edge-only mistake).

Figure 1. The three-tier pattern every deployment past a handful of cameras converges on.

Edge silicon in 2026: Axis, Hailo, Ambarella, Jetson, Sony

Hardware choice follows three dimensions: where the chip sits, how many TOPS it delivers, and which runtime the team is willing to target.

For greenfield deployments we default to Hailo-8 M.2 cards in an off-the-shelf mini PC; they clear roughly 29 fps on YOLOv8n at 640×640 and pull about 2.5 W under load. For existing Axis estates, ACAP on the camera is the lowest-friction path and keeps the gateway simple. Jetson Orin wins when one box must run heterogeneous models (detection + pose + plate reading) across 16–32 streams at once.

Model selection: YOLO, DETR, or something smaller

Three model families cover 90% of object-recognition workloads in 2026. Pick by latency budget and whether the box has a GPU.

YOLO family (YOLOv8, YOLOv9, YOLO11, YOLO26). One-stage detectors, anchor-free. YOLOv8n or YOLO11n for edge cameras (a few MB, ~1 ms on an A100 under TensorRT), YOLO11m for a Jetson, YOLOv9-E or YOLO26 for cloud or server-class gateways. YOLO26 goes end-to-end without non-maximum suppression, which simplifies edge deployment. The Ultralytics toolchain exports cleanly to TensorRT, ONNX, HailoRT, OpenVINO, and CoreML.

DETR and variants (RT-DETR, DETA, Deformable DETR). Transformer detectors. Cleaner behavior on crowded scenes because anchor matching is replaced by set prediction. RT-DETR hits YOLO-class speed with transformer semantics. Use it when the scene holds 30+ overlapping objects, or when the downstream system wants attention maps for explainability.

Purpose-built smaller models. MobileNet-SSD, EfficientDet-Lite, or a tiny custom classifier when the task is single-class (forklift, hard-hat, fire). A fire-detection classifier under 2 MB beats YOLO on false-positive rate because its training set is narrower and better curated. Never reach for a general-purpose model when a specific one exists.

TensorRT INT8 quantization cuts latency a further 3–5× on NVIDIA hardware; Hailo's HEF compiler and Intel's OpenVINO post-training quantization produce comparable results on their targets. The gap between FP32 and INT8 is typically 1–2% mAP on YOLO-family models — rarely worth fighting.

ONVIF Profile T: the metadata contract

A detection is worthless if it does not land inside the VMS the security team already uses. ONVIF Profile T, the streaming-and-analytics profile, defines the metadata schema that Milestone, Genetec, Avigilon, Axis Camera Station, and most open-source VMS platforms consume out of the box.

The contract is simple: the camera or gateway emits an RTSP stream with a parallel metadata track. Each frame carries a MetadataStream element with one or more Object entries — each with a bounding box (normalized 0.0–1.0), a class label, a confidence, and a stable tracker ID. Timestamps must be frame-synchronized with the video; drift above 40 ms confuses the VMS event-correlation engine.

Axis ACAP apps emit ONVIF metadata natively. NVIDIA DeepStream can broker it to the VMS and to a message bus. For bespoke pipelines we usually ship a small Go or Rust service that wraps the inference output and speaks ONVIF to the VMS — roughly 600 lines of code living in the gateway.

For VMS-specific integration (Milestone MIP SDK, Genetec SDK, Avigilon Control Center SDK) we also emit a parallel webhook or SDK call, because the VMS rule engine binds to proprietary events more predictably than to generic ONVIF. Two tracks, one source of truth.

Figure 2. ONVIF Profile T is the metadata contract that carries a detection into the VMS.

The event pipeline: MQTT, Kafka, or webhooks

Detection events need a bus. Three options cover every realistic deployment.

MQTT. The default for low-count, edge-heavy deployments (under 200 cameras, under 500 events per second). Mosquitto or HiveMQ as the broker, QoS 1 for reliability. It fits cleanly on the same gateway that runs inference, and DeepStream publishes to it natively.

Kafka. The default above 500 events per second, or when several independent consumers need the stream (VMS, analytics warehouse, SIEM, alerting). Confluent Cloud, MSK, or self-hosted Strimzi. Topics per camera-group let consumers subscribe without seeing every event. A seven-day retention is typical for replay and debugging.

Webhooks. Use them when the consumer is a single SaaS (Splunk, PagerDuty, a ticketing system) and you do not want another broker in the stack. Sign every webhook with HMAC-SHA256; do not trust the source IP.

We almost always combine two: MQTT from camera and gateway up to a local aggregator, Kafka from the aggregator out to downstream consumers. That split survives a WAN outage (MQTT keeps buffering locally) and scales horizontally (Kafka consumer groups take the load).

Embeddings, re-identification, and forensic search

Detection is only half the problem. Once a box is drawn, the commercially interesting question is: is this the same person we saw yesterday at camera 12? That is a re-identification problem, and it runs on embeddings, not detections.

The pattern we ship most often: a 256- or 512-dimensional embedding per detected object, computed on the gateway by a lightweight model (OSNet for persons, a color-plus-texture model for vehicles) and written to Milvus, Weaviate, or Qdrant. Query latency at 100 million embeddings, well indexed with IVF_PQ, stays under 50 ms on a single modest VM.

Forensic search — find everyone in a red jacket who entered Zone 3 yesterday — becomes a vector query with metadata filters. For clients who need it we bolt in ElasticSearch for the metadata facet (timestamp, zone, camera) and Milvus for the vector nearest-neighbor, roughly the same split as modern retail search.

Embeddings raise the privacy stakes. An embedding is a biometric identifier under most regulations; treat it as one. Expire embeddings on the same schedule as raw footage, never longer.

Privacy and compliance baked in

GDPR Article 6 lawful-basis analysis, the CCPA's biometric provisions, and the EU AI Act classify most public-space object recognition as high-risk processing. The Act's Article 5 prohibitions have applied since 2 February 2025 (real-time remote biometric identification in public, emotion recognition at work or school, untargeted face scraping); the Act reaches full effect on 2 August 2026, and the high-risk obligations for biometric systems now bind from 2 December 2027 under the Digital Omnibus. The engineering implications are concrete.

Pixel-level anonymization at the source. Face blurring and plate masking run on the same gateway as detection. The un-anonymized frame never leaves the gateway unless an authorized investigator triggers an escrow release with a signed audit record. Libraries: OpenCV blur for faces when throughput matters, or a dedicated face-parser for segmentation-quality masks.

Role-based access to raw versus anonymized feeds. The VMS integration should surface two parallel streams. The default UI shows the anonymized one. Raw access requires an elevated role and writes an audit event to the SIEM. This is the single most-cited CNIL violation we see.

Retention windows per data class. Raw video: 7–30 days typical, 90 days max without a specific legal hold. Anonymized video: same or shorter. Detection metadata: 1–3 years for analytics, expirable on request. Embeddings: same as raw video.

Subject access and erasure. A GDPR data-subject-access-request workflow has to find every frame containing a given face or plate — the same embedding index used for forensic search. Budget that capability up front; retrofitting it once the DPO asks is a multi-sprint surprise. Our anomaly-detection guide covers the model side of the same compliance story.

Edge, gateway, or cloud: the cost tradeoff

Where inference runs is a unit-economics decision. Here is the breakeven math we run on every kickoff.

Cloud-only (AWS Rekognition Video or Azure Video Indexer). Rekognition streaming runs $0.10 per stream-minute. Analyze a camera continuously and that is $0.10 × 60 × 24 × 365 ≈ $52,560 per camera per year; a realistic motion-gated feed running a couple of active hours a day lands nearer $4,000–9,000. Either way it scales linearly with every camera and every minute. Fine for 10–40 cameras; brutal above 200.

Gateway-based (Jetson or Hailo on-prem). One $1,500 Jetson Orin NX handles 16–32 streams; amortized over five years that is under $19 per camera per year in hardware, plus roughly 30 W of draw per box. VMS licensing (Milestone XProtect and the like) is a separate line item.

Camera-native (Axis ACAP, Ambarella). Zero additional hardware. Model updates ship as signed ACAP packages or firmware. It works cleanly only when the camera already has inference silicon; you cannot retrofit an older camera.

The crossover: below 40–50 cameras, cloud-only is often simplest because you skip capex; between 50 and 200, a Jetson gateway usually wins; above 200, camera-native gets attractive because you are already specifying new hardware on a refresh cycle.

Figure 3. Per-camera inference economics: cloud scales with every minute; self-hosted amortizes toward zero.

Build vs. buy: where custom earns its keep

Off-the-shelf analytics products (BriefCam, iOmniscient, Avigilon Unusual Activity) cover the standard workloads — people counting, perimeter breach, abandoned object, loitering, plate reading. Custom engineering earns its keep in three cases.

Industry-specific class taxonomies. A packaged model knows “person, car, truck.” Your operation needs “forklift vs. pallet jack,” “hard-hat vs. helmet vs. bump-cap,” or “surgical mask vs. N95.” A custom YOLO fine-tune on 3,000–10,000 of your own labeled frames beats a generic model by 5–15 percentage points mAP on the classes that matter.

Cross-system workflows. A detection by itself does nothing. “Object detected AND door opened AND badge not scanned” is a compound event that packaged products do not express. Custom rule engines (Drools, a Go-based CEP, or a hand-rolled state machine) close the gap.

Data-sovereignty constraints. Packaged SaaS sends frames to the vendor cloud. That is a non-starter for most healthcare, defense, finance, and critical-infrastructure customers. A custom stack keeps frames on-site or in a specific region. If you are building the whole platform, our custom VMS guide picks up where this one ends.

Cost model: a 12-week custom integration

These are Fora Soft 2026 estimates with Agent Engineering, for a custom object-recognition layer built on top of an existing VMS and camera fleet. They are conservative.

Phase 1 — Model and pipeline MVP (3–4 weeks). Label collection, a YOLO or DETR fine-tune, TensorRT or HailoRT export, basic MQTT event emission, a docker-compose gateway. Budget: ~$25K–$45K.

Phase 2 — VMS and metadata integration (3–4 weeks). ONVIF Profile T emission, Milestone MIP or Genetec SDK wiring, VMS event rules, an operator UI overlay. Budget: ~$35K–$65K.

Phase 3 — Re-id, search, and privacy (4–6 weeks). Embedding pipeline, Milvus setup, face and plate anonymization, role-based access, audit log, retention jobs, DPIA artifacts. Budget: ~$55K–$95K.

Phase 4 — Hardening (2 weeks). Load testing at target camera count, failover drills, documentation, operator training. Budget: ~$25K–$45K.

Total. ~$140K–$250K for a 50–150 camera deployment; ~$200K–$350K for 150–500 cameras across multiple sites. Running costs (Milvus, Kafka, egress) typically $3K–$15K per month depending on volume.

Mini-case: human vs. vehicle recognition at the edge

On EyeBuild, a solar-powered construction-site surveillance product we built from scratch, object recognition had to run with no internet and no gateway rack. The cameras sit on poles at active building sites, connect over 4G/5G, and hold 14 days of solar battery plus a three-day reserve. We put human-versus-vehicle detection on the device itself: the model distinguishes a person from a truck, fires an automated alert, and streams 4K UHD with 360-degree pan-tilt and two night-vision modes. It runs turn-key at $249 per camera per month with same-day install — the edge-first topology from the top of this article, shipped.

When a client already runs Milestone or Genetec, the pattern we repeat is different: fine-tune YOLO on their own footage for the classes that matter, deploy on a Hailo-8 or Jetson box next to the recorder, emit ONVIF Profile T metadata for the overlay, and fire a parallel MIP-SDK event so the VMS rule engine can evaluate compound conditions. Two tracks, one detector, no rip-and-replace.

The payoff shows up in operations. On Live Eye Surveillance, an AI cloud-surveillance platform we built for retail and quick-service chains, POS-to-video matching and behavioral detection cut grocery shrink by up to 30% in the first quarter and drive-off incidents at drive-throughs by 40%, with alerts landing within 30 seconds. Want a similar assessment for your fleet? Book a 30-min camera analytics review — bring your VMS, your camera brands, and a handful of false-positive clips.

A decision framework in five questions

1. How many cameras, and where are they concentrated? Under 40, cloud-only is probably cheapest. Over 200, camera-native or gateway. 40–200 usually settles on Jetson or Hailo gateways.

2. What VMS is already in place? Milestone XProtect, Genetec Security Center, Avigilon, or custom. The VMS sets the metadata contract (ONVIF Profile T plus vendor SDK) and the operator UI path.

3. Do your classes exist in a stock model? If “person, vehicle, animal” covers it, start with a stock YOLO fine-tune or a packaged product. If you need “forklift vs. pallet jack,” budget a labeling round.

4. What regulation applies? EU, UK, or California deployments need privacy-by-design from sprint one. APAC and LATAM vary by jurisdiction; legal should weigh in before any embedding leaves the device.

5. Does the problem require compound events? If yes, plan a CEP engine or rule-based state machine from the start. Retrofitting compound logic into a single-detection pipeline always doubles the timeline.

Figure 4. Five questions that fix scope, hardware, and compliance before the first purchase order.

Five pitfalls that derail camera-analytics projects

1. Training on stock COCO and calling it done. COCO labels lean toward consumer imagery. Warehouse lighting, indoor industrial scenes, and low-light cameras sit outside that distribution. Budget a 3,000–10,000 frame custom label round; it determines the whole project's mAP.

2. Picking Jetson because it is familiar, then needing Hailo anyway. Jetson is flexible but power-hungry; Hailo is efficient but limited to two or three model graphs. A one-hour hardware workshop before the first PO saves weeks of rework.

3. Forgetting the ONVIF metadata timestamp. A 40 ms drift between video frame and metadata frame confuses the VMS tracker. We have seen deployments ship with 300 ms of drift because nobody tested the metadata pipe end-to-end in the VMS.

4. Embedding the wrong vector dimension. 128-d is too small for person re-id across months; 1024-d is wasteful and slow. 512-d OSNet or MobileFaceNet embeddings are the sweet spot for enterprise-scale Milvus indexes.

5. Treating privacy as a final-week checkbox. Anonymization, RBAC, and retention have to be designed into the data flow, not bolted on. A single CNIL fine above €100,000 has killed more custom analytics projects than any technical failure.

KPIs worth putting on the dashboard

Model KPIs. mAP on the customer's own validation set (not COCO), false-positive rate per 24 h per camera, false-negative rate on the top three operationally critical classes. Re-evaluate after every deployment round.

Pipeline KPIs. End-to-end detection latency p95 (camera to VMS event), metadata timestamp drift, stream-loss rate, inference queue depth per gateway. Page the on-call if p95 drifts past 400 ms.

Business KPIs. Alarms per operator shift, operator acknowledgment latency, incidents prevented (tied to the customer's own incident log), cost per detected event. These are what keep the project funded for year two.

When a custom build is the wrong answer

Three situations call for a packaged product instead. If your deployment is under 40 cameras and single-site, a Rekognition Video feed plus a basic VMS like Milestone Essential+ costs less over three years than a custom integration. If your use case is a solved commodity (plate reading, perimeter breach, mask detection circa 2021), the packaged analytics from Axis, BriefCam, or Avigilon are already trained on tens of thousands of hours you cannot match. And if your security team has no AI operational expertise, a managed service beats a custom pipeline running on hardware nobody on staff understands.

Custom earns its keep when the class taxonomy is proprietary, when the workflow is compound, or when data sovereignty is non-negotiable. Otherwise, buy. If you are weighing a broader surveillance build, our video surveillance development service page lays out how we engage.

FAQ

How many labeled frames do we need to fine-tune a detector?

For a single new class on a YOLO-family fine-tune, 3,000–5,000 labeled frames across varied lighting and angles usually beats a packaged model on your own footage. For 5–10 new classes or rare events, budget 10,000–30,000. Active-learning loops — label the model's low-confidence frames — are more efficient than random sampling past the first thousand.

Can we run object recognition on an existing Axis camera?

Only on cameras with an ARTPEC-7 or ARTPEC-8 chip that supports ACAP-native ML inference. A large model will not fit, but a quantized YOLOv8n or YOLO11n deploys cleanly. For older ARTPEC cameras without an ML accelerator, run inference on a Jetson or Hailo gateway next to the camera and emit ONVIF metadata back to the VMS from there.

What is the realistic stream density on a Jetson Orin?

On an Orin Nano Super, roughly eight 1080p streams running a mid-size YOLO at a 15 fps detection cadence, or sixteen at 10 fps. An Orin NX doubles that. TensorRT INT8 plus DeepStream pipeline tuning is the difference between “works” and “falls over at six streams.”

Is AWS Rekognition Video cheaper than running our own Jetson?

For a handful of cameras that only process motion events, yes. Continuous 24/7 Rekognition streaming runs about $52,560 per camera per year at $0.10 per minute; a $1,500 Jetson Orin NX covering 16 streams is about $19 per camera per year in hardware plus power. The more hours a day your cameras actually analyze, the faster self-hosting wins.

Does ONVIF Profile T work with Milestone XProtect out of the box?

Partly. XProtect consumes Profile T metadata for on-screen overlays and basic event rules, but any compound rule (“object detected AND badge not scanned”) needs the Milestone MIP SDK. We routinely ship both: ONVIF for the overlay, MIP SDK events for the rule engine. Genetec Security Center has the same split with its own SDK.

How do we handle GDPR when the cameras record a public street?

Document a legitimate-interest basis under Article 6(1)(f), complete a DPIA under Article 35, apply pixel-level face blurring before any frame leaves the gateway, stage raw footage behind role-based access with audit logging, and set retention to the minimum that serves your documented purpose (typically 30 days). The DPO must sign off; do not skip it.

What changes with the EU AI Act?

Since 2 February 2025, real-time remote biometric identification in public spaces is largely prohibited (narrow law-enforcement exceptions), and emotion recognition at work or school is banned. Most enterprise video analytics with biometric features count as high-risk, so they need risk management, data governance, logging, human oversight, and a conformity assessment — obligations that bind from 2 December 2027 under the Digital Omnibus. Plan for a formal technical file and an EU representative if you are non-EU.

How long does a realistic 50-camera rollout take?

Twelve to sixteen weeks from kickoff to production with Agent Engineering, assuming an existing VMS and accessible footage for training. Weeks one to four are model and pipeline; five to eight are VMS integration; nine to twelve are privacy, search, and hardening. The labeling round usually runs in parallel with pipeline work.

What to Read Next

Ready to ship object recognition your VMS actually understands?

Object recognition in 2026 is three decisions stacked: the detector (YOLO11, YOLO26, RT-DETR, or a custom tiny classifier), where it runs (on-camera, a Jetson or Hailo gateway, or a thin cloud layer), and how it speaks to the VMS (ONVIF Profile T plus the vendor SDK). Get those three right and the software half is mostly solved. Get any one wrong and the false-positive rate grinds your operators down until they stop acknowledging alerts.

Privacy and compliance are the invisible fourth decision. Anonymization, role-based access, retention, and audit trails are what separate a product that ships from a project legal kills. Build them in from sprint one.