Anomaly Detection in Surveillance Footage: 2026 Guide

Why Fora Soft wrote this playbook

Anomaly detection on surveillance footage fails in two boring, expensive ways. A team demos 95% accuracy in the lab, ships it, and watches the number fall apart the first rainy night. Or the product quietly turns a camera fleet into a biometric-data liability that a regulator or a plaintiff’s lawyer finds two years later. Both are preventable at week one, and both are why this guide exists.

We’re Fora Soft. We’ve built video and AI products since 2005, 250+ shipped projects, and our own video-management product, VALT, runs in 770+ organizations serving 50,000+ active users. Surveillance is the hardest corner of that work: latency is real, false alarms kill adoption, edge hardware is constrained, and the legal surface keeps widening. This is the brief our architects use at kickoff. If you want the algorithm theory instead, read our companion piece on machine-learning algorithms for surveillance anomalies; if you want a tools shootout, see our anomaly-detection models comparison. This page is the buyer-and-builder decision guide: what actually works on your footage, and what it costs.

One note on speed. Our agent-engineering practice, the AI-augmented workflow we run on every project, compresses a surveillance integration by roughly 30–40% against our 2024 baselines, because the edge-cloud orchestration, ONVIF Profile M parsers, and model-compression pipelines for Jetson and Hailo are reusable modules rather than fresh work each time.

What counts as an anomaly in footage

An anomaly is any event that deviates from what a scene normally looks like: a fight, a fall, an abandoned bag, someone loitering by a locked door at 3 a.m. In practice the phrase covers three classes, and they need different models and different evaluation. Getting this taxonomy straight up front is what stops a vendor from selling you a loitering detector when you actually need weapon detection.

1. Behavioral anomalies. Loitering, crowding, wrong-direction flow, perimeter breach, violence, falls, visible weapons, abandoned objects. These dominate retail and smart-city work.

2. Appearance anomalies. Masks where they shouldn’t be (banks), missing PPE where it should be (factories, construction), dress-code breaches in secure zones.

3. Temporal anomalies. After-hours activity, surge occupancy, unusual dwell time in a zone. Cheap to detect, but the noisiest class without scene calibration.

The winning 2026 product doesn’t just fire an alert. It answers questions. An operator types “show me everyone who entered the restricted zone without a badge after 11 p.m.” and gets clips back, because a video-language model sits on top of the detector. That shift, from alarm to answer, is the throughline of the four-pillar stack below.

The four-pillar detection stack

Every system we ship maps to four pillars. Object detection alone produces noise. Unsupervised scoring alone produces alerts nobody can explain. Reasoning on top of both is what makes an operator trust it. Drop a pillar and precision falls in production, no matter how good the headline benchmark looked.

Figure 1. The four-pillar stack, from camera-side detection to a metadata bridge into your existing VMS.

Benchmarks vs. real footage

Here’s the honest answer most vendors skip: the accuracy number in the deck is measured on a curated academic dataset, and your footage is harder. On ShanghaiTech Campus, published methods reach ~97–98% AUC. On UCF-Crime, 1,900 real, untrimmed CCTV videos across 13 crime classes from the CVPR-2018 Real-World Anomaly Detection paper, the 2025 state of the art sits near 88% AUC. That ~10-point gap is the reality gap, and your own cameras usually land a notch below UCF-Crime because your scenes never appeared in any training set.

Figure 2. Curated benchmarks flatter models; the drop to real, unseen footage is the number that decides your project.

Two rules follow. First, if a vendor can’t quote numbers on named datasets, the accuracy claim is marketing. Second, AUC hides per-class performance, so ask for the full precision-recall curve on the anomaly classes you care about, tested on a slice of your own footage. Newer research is closing the gap with training-free video-language methods and event-based spiking models (UCF-Crime-DVS, 2025), but none of that changes the buying rule: test on your cameras.

The 2026 model lineup

Four model families carry real load in 2026. Pick by deployment constraint, not by leaderboard rank.

A note on the open option: Qwen3-VL ships under Apache 2.0 with variants from 2B to 32B and a native 256K-token context, so you can run video understanding on your own hardware without per-query cloud fees. That matters when a customer’s legal team refuses to send footage off-site. Between detection and reasoning sit the glue models: ByteTrack and BoT-SORT for multi-object tracking, TransReID for cross-camera re-identification, and RTMPose for fall and fight detection without storing any face data.

Edge hardware: where inference runs

The economics tipped to the edge in 2025. Cloud inference at $0.05–0.30 per stream-hour means $438–$2,628 per camera per year for 24/7 coverage. An edge accelerator is a one-time purchase for less than a single year of that. Here’s the current menu.

How we default. Hailo-8 in the cameras for detection; Jetson Orin NX or AGX Orin at the NVR tier for tracking, re-ID, and aggregation; cloud for forensic search across months of footage. We only put AGX Thor on-site when you need video-language reasoning with zero cloud round-trip, typically rail platforms or airports where latency and data-residency both bite. For the full architecture pattern, see our edge AI surveillance architecture guide.

False positives: the metric that matters

AUC on a benchmark is table stakes. What kills a deployment is the operator who muted alerts after the tenth false fire alarm. The false-positive rate is the product. These are the techniques that move it, roughly in the order we apply them.

Figure 3. Each filtering stage cuts the false-alarm rate before a human is ever paged.

1. Temporal windowing. Require N consecutive frames above threshold before firing. Five frames at 10 fps is half a second of sustained detection. Simple and devastatingly effective.

2. Multi-model ensembling. YOLOv11 plus RT-DETR v2 plus a VLM check; vote on the box. Dropping below 2-of-3 agreement roughly halves false positives in our measurements.

3. Optical-flow filtering. Separate object motion from camera and background motion. This alone eliminates most wind, rain, and shadow triggers.

4. Scene-specific thresholds. Calibrate per camera for lighting and typical activity. One global threshold across an outdoor stadium and a windowless data center is malpractice.

5. Human-in-the-loop. For weapons and violence, a person confirms before escalation. It’s the only way under 3%, and institutional buyers won’t renew without it.

Baseline to hold a vendor to: first-gen systems run 30–60% false alarms, a tuned 2026 stack gets under 10% within 90 days, and under 3% needs a human in the loop.

VMS integration over ONVIF Profile M

The AI layer is the easy part. Making it speak fluently to the customer’s existing Milestone XProtect, Genetec Security Center, or Avigilon is what closes the deal. Three ONVIF profiles matter, and only one of them is about analytics.

Profile S / T. Streaming and advanced video (H.264/H.265/AV1, imaging control). Transport, not intelligence.

Profile M. The analytics profile. Standardized metadata: object classes, bounding boxes, confidence, geolocation, and MQTT publishing. This is the bridge that carries detections into your VMS without proprietary glue.

The standard pattern: the camera or NVR runs detection, emits Profile M metadata, the VMS applies a rule (“person plus loitering > 60s”), and MQTT bridges to a SIEM like Splunk for audit and correlation. Expensive models (VLM forensic queries) run as optional cloud escalation.

Platforms: who sells what

If you’d rather buy than build, this is the shortlist we see most in production. The right pick depends on whether you’re cloud-first, already own a VMS, or need forensic search at scale.

A caveat on non-US brands: Hikvision and Dahua ship strong edge AI at aggressive prices, but US federal procurement and several allied markets restrict them under NDAA rules. If you sell into government or critical infrastructure, that constraint decides the shortlist before features do.

Weapon detection: highest-stakes class

Weapon detection earns its own section because the failure modes are existential. Miss a real weapon and you own a liability suit. Fire too many false ones and the product gets muted right before it’s needed. The vendors worth knowing:

ZeroEyes. Live 24/7 human review by former military and law-enforcement staff. The human-in-the-loop model is its moat.

Omnilert. Multi-modal detection trained on real surveillance, focused on schools and venues.

Evolv, Scylla, Actuate. Entry screening and camera-based detection with 95%+ accuracy claims. Evolv drew FTC scrutiny in 2025 over its claims. Demand independent third-party benchmarks before you sign anything.

Our stance: deploy weapon detection only with a human verification layer and a rehearsed incident-response runbook. The alert is not the end of the pipeline. It’s the start of a procedure, and if that procedure isn’t written down and drilled, the AI is theater.

Compliance: the 2026 legal surface

Surveillance AI lives where privacy, biometric, and AI-safety law overlap. Get this wrong and the fine dwarfs the project. The 2026 snapshot, with the dates that actually apply:

The practical takeaway is the same one from the VMS section: default to metadata-only, and the bulk of this table stops applying to you. The authoritative text is short and worth reading, the EU AI Act Article 5 prohibitions run about a page.

Cost model: 100 cameras, real numbers

A concrete 100-camera deployment, mixed indoor and outdoor, 2026 pricing. The single line that decides edge vs cloud is inference: $0.10 per stream-hour times 24 hours times 365 days times 100 cameras equals $87,600 per year, every year. That’s why detection runs at the edge.

Figure 4. Year-1 total cost of ownership: edge-primary versus cloud-heavy for a 100-camera fleet.

Build vs. buy: the honest cut

Short answer: buy the platform if your anomaly classes are standard and you have fewer than ~50 cameras; build (or hire a build) when you need custom classes, on-prem data residency, or a VMS you already own. Most mid-market fleets land in a hybrid: a cloud VMS for ops, a custom edge layer for the two or three anomaly classes that actually drive your ROI.

The trap in “buy” is the per-camera subscription that looks cheap at 40 cameras and hurts at 400. The trap in “build” is underestimating the ONVIF bridge and per-scene tuning, which is where the weeks go. If you’re leaning build, our YOLO and DeepSORT surveillance build guide walks the detection-and-tracking core, and our custom surveillance solutions page shows what a bespoke engagement covers.

Mini case: 250-store retail rollout

A North American specialty retailer with 250 stores came to us with an Avigilon fleet and Milestone XProtect already in place. Organized retail crime had pushed shrinkage from 1.2% to 2.8% of revenue over 18 months. Corporate loss-prevention wanted anomaly detection across the chain in a quarter, without ripping out hardware.

We built on top of what they had. A Jetson Orin NX per store (one per 8–10 cameras) ran YOLOv11 plus ByteTrack for six anomaly classes trained on their own footage: loitering near high-value displays, reach-and-grab, multi-person exit through unmanned doors, and self-checkout non-scans. Detections rode ONVIF Profile M into an XProtect plug-in that pushed a store-manager alert with a 5-second clip. A weekly cloud batch through Twelve Labs let the corporate team run natural-language queries across the whole archive. Store managers verified before any corporate escalation.

Across a 90-day pilot on 40 stores, the false-positive rate fell from a week-one 47% to 11% by month three with active-learning retraining. Shrinkage in pilot stores dropped 0.9 points against matched controls, and weekly manager logins hit 78%. We rolled the remaining 210 stores over the next quarter. Want a similar assessment on your fleet? Book a 30-minute call.

5 pitfalls that kill these projects

1. Data bias across regions. Models trained on Western footage miss anomalies in different lighting, dress, and movement. Budget a per-market fine-tuning pass before go-live, or your Tokyo deployment misses half its events.

2. Environmental false positives. Weather, shadows, birds, flickering LEDs. Temporal windowing, optical-flow filtering, and scene calibration are the three fixes. Budget them in week one, not after launch.

3. Biometric-storage lawsuits. Even well-meaning face databases invite BIPA and EU AI Act claims. Default to metadata-only; store embeddings only when legally authorized and operationally necessary.

4. Camera placement and lighting. Garbage input, garbage output, whatever the model. Insist on a site survey, 1080p minimum, proper IR, and 5–15 fps. A bad mounting angle guarantees failure.

5. No human oversight loop. Fully autonomous alerting invites wrongful-action risk. Operator verification with an audit trail is the minimum defensible standard, and the thing institutional buyers check first.

KPIs: what to measure

Quality. False-positive rate (target under 10% within 90 days, under 3% with a human in the loop) and per-class recall on a labeled test set drawn from your footage, not the vendor’s reel.

Reliability. Mean time to alert (under 2 seconds for weapons, violence, and perimeter) and monthly model-drift checks against a held-out set, flagging any drop over 5% AUC.

Business. The outcome the procurement case was built on: shrinkage for retail, incident-response time for public safety, injury rate for manufacturing. Tie it back every quarter, or the renewal conversation gets hard.

Pick your stack in six questions

1. Which anomalies matter? Behavioral only points to YOLOv11 plus ByteTrack. Add reasoning queries and you add a VLM. Weapons or violence adds a human-in-the-loop layer, non-negotiable.

2. Edge or cloud? 24/7 monitoring is edge. Forensic and batch queries are cloud. Most deployments want both.

3. What VMS is already in place? Milestone, Genetec, or Avigilon means integrate over ONVIF Profile M. Greenfield means pick on ops preference.

4. Which jurisdiction? EU defaults to no face recognition and a conformity assessment. US is BIPA-aware, with municipal bans that matter.

5. How many cameras? Under 50, one AGX Orin covers it. 50–500, distributed Orin NX per site plus central aggregation. 500+, Hailo on cameras plus AGX Thor at regional hubs.

6. Who’s the operator? A trained SOC can take raw alerts. A store manager needs filtered, verified alerts with clips only. Design the UI for the person, not the model. If mapping this to your setup is fuzzy, that’s exactly the 30-minute call.

When NOT to build

Honesty sells, so here are the projects we turn down. We say no when a customer wants real-time public face recognition in an EU jurisdiction, because Article 5 bans it. We say no when camera resolution is below 720p or under 5 fps, because the model is capped at “bad” before software touches the stream. We say no when there’s no appetite for a 60-day pilot with threshold tuning, because it will fail on false positives in the first month.

We also decline weapon detection without a defined incident-response runbook and a verification step, and any deployment in a jurisdiction with no clear legal basis for the biometrics proposed. Shipping products that invite litigation is bad for the client and bad for us.

FAQ

What is anomaly detection in surveillance footage?

It’s using machine learning to flag events that deviate from a scene’s normal pattern, such as a fight, a fall, an abandoned bag, or loitering by a locked door. Unlike motion detection, which fires on any pixel change, anomaly detection classifies what moved and scores it against expected behavior, cutting false alarms sharply.

How accurate is anomaly detection on real footage?

On curated benchmarks, methods reach ~97–98% AUC (ShanghaiTech, 2025) but ~88% on the more realistic UCF-Crime. On your own unseen cameras, expect a notch lower. Judge a vendor on a test against a slice of your footage, not a benchmark score.

What’s the difference between motion detection and anomaly detection?

Motion detection fires on any pixel change, with false-alarm rates of 30–90%. Anomaly detection classifies the motion (person, vehicle, leaf) and scores it against expected behavior, dropping false alarms to 10–30% with modern AI and under 3% with a human in the loop.

Do I need to replace my existing cameras?

Usually not. Any 1080p ONVIF Profile S camera can feed an edge NVR running the AI pipeline. Replacement is worthwhile only if resolution is below 720p or the frame rate is under 5 fps.

Is facial recognition legal in my deployment?

It depends on jurisdiction and use. In the EU, real-time public face ID for law enforcement is banned under Article 5 (since 2 Feb 2025), with narrow exceptions. In the US, BIPA, CUBI, and CCPA/CPRA apply, and several cities ban law-enforcement use. Get legal sign-off before deployment.

Edge or cloud for anomaly detection?

Run real-time detection at the edge; cloud inference at $0.10/stream-hour is $87,600/year for 100 cameras. Keep the cloud for forensic search and cross-camera reasoning across months of footage. Most fleets run a hybrid.

How accurate is AI weapon detection really?

Vendor claims of 95%+ are common but often untested against concealment, occlusion, and low light. Reliable field performance comes only with human verification (the ZeroEyes pattern). Demand independent third-party audits before procurement.

How long does deployment take?

Our typical engagement ships a pilot on 50–100 cameras in 10–14 weeks: discovery, model selection, per-scene tuning, ONVIF Profile M bridge, and a 60-day pilot. Chain-wide rollouts add roughly a quarter per 200–300 additional sites.

What to read next

Ready to detect anomalies in your footage?

Anomaly detection on surveillance footage is a mature category in 2026: production-grade open models, cheap edge silicon, and a settling compliance regime. The winning shape is the four-pillar stack, edge detection into anomaly scoring into VLM reasoning into an ONVIF Profile M bridge, shipped through a 10–14 week integration with a 60-day pilot in the middle.

Three decisions decide success: buy on a test against your own footage, not a benchmark; run detection at the edge for cost and latency; and default to metadata-only for compliance. Get those right and the engineering is tractable. Get them wrong and the system quietly degrades into an expensive, muted alarm. If you want a partner who’s shipped this before, that’s us.

Prefer to keep reading first? Our guides on hiring computer-vision developers, choosing IP camera systems, and the video surveillance learning hub go deeper, and our AI integration service covers how we plug this into an existing product.