Video Anomaly Detection: Algorithms & Architecture 2026

Why Fora Soft wrote this playbook

Fora Soft has built software since 2005: 250+ projects, 50 in-house engineers, most of them in video, streaming, and computer vision. VALT, the intelligent video system we have been the sole development team on for over a decade, now serves 770+ US organizations and 50,000+ users under HIPAA. EyeBuild is our solar-powered, fully offline AI camera for construction sites, doing on-device human and vehicle recognition with automated alerts over 4G/5G.

That work is where this guide comes from. If you are an ML engineer, a surveillance product CTO, or a smart-building integrator scoping video anomaly detection, we wrote down which algorithm family fits which use case, the architecture that survives real-world drift, and the places these projects usually fall apart. Everything numeric is dated and sourced; where a number is modeled rather than measured, we say so.

The short version: the hard part is almost never the detector. It is the false-positive rate, the drift, and the data pipeline feeding the model. Get those three right and a mid-sized model beats a fancier one that nobody trusts.

Why generic anomaly detection fails on video

Generic anomaly detection, meaning an sklearn IsolationForest over flat tabular features, works on credit-card fraud and server metrics. It does not survive video. Three reasons keep breaking naive ports.

1. A frame is not a row. One 1080p frame is 1920 × 1080 × 3 numbers. Before any anomaly algorithm runs, you have to project that into a feature space: raw pixels, CNN embeddings, optical flow, foreground masks, or object detections. That choice of feature pipeline drives accuracy more than the scoring algorithm sitting on top.

2. Normal keeps moving. Lighting, weather, time of day, season: what counts as normal at 6 am in July is not normal at 6 pm in January. A model fit to a fixed snapshot drifts fast. Production needs rolling baselines or models conditioned on the environment.

3. Anomalies are rare and asymmetric. Real events (intrusion, a fall, fire) are a tiny fraction of frames. A 1 % false-positive rate still means 1,000+ alerts a day across a 100-camera fleet, which is operationally dead. The bar is precision, not recall. A supervised, high-stakes cousin, the AI weapon detection system, hits the same wall, which is why every flag routes through human verification.

The four algorithm families

Almost every practical detector is one of four families, or an ensemble of them. They differ by what they can see and what they cost per frame.

Figure 1. The four families side by side: inference cost against what each one can and cannot catch.

Statistical methods are cheap, interpretable, and brittle. A Z-score on a motion histogram catches a sudden change; a Mahalanobis distance on a small feature vector catches a multi-dimensional outlier. Use them as a first-pass gate before heavier inference, not as the whole system.

Distance-based methods score how far a sample sits from the training distribution, usually over CNN detection embeddings. Isolation forest is the workhorse: fast, stable, and it trains on normal-only data (scikit-learn outlier detection, 2025). One-class SVM is sharper on small sets but scales poorly past ~50k samples. Local Outlier Factor handles density variation the others miss.

Reconstruction methods learn to compress and rebuild normal frames; a high reconstruction error flags an anomaly. They train on unlabelled normal footage, which is a real advantage in surveillance where labelled anomalies barely exist. Variational and GAN variants raise quality at higher compute cost. The open-source anomalib library (Intel/OpenVINO) ships PaDiM, PatchCore, and EfficientAD implementations to start from.

Sequence-based methods capture patterns over time: loitering, abandoned objects, unusual trajectories. An LSTM or Transformer predicts the next frame, and a large prediction error becomes the anomaly score. TimesNet and masked spatio-temporal autoencoders (MAE-ST) sit at the 2025 state of the art on benchmarks like UCF-Crime, at hundreds of milliseconds per clip.

What makes video different

The four challenges below are why a model that scores 90 % on a benchmark can still flood an operator with junk in the field.

Foreground vs background. Most detection should run on moving objects, not the whole frame. Background subtraction (OpenCV MOG2) or a segmentation mask shrinks the feature space and lifts signal-to-noise before the anomaly model ever runs.

Motion is not anomaly. Every anomaly has motion; most motion is normal. A person walking is fine; a person climbing a fence at 3 am is not. The detector has to combine motion with context (object class, location, time) to tell them apart.

Lighting, weather, time of day. Training data has to cover the full deployment range. Augment hard: brightness, contrast, rain and fog overlays, IR versus visible spectrum, golden-hour light. When real coverage is thin, extend it with synthetic footage.

Camera angle and zoom. A model trained on one fixed view fails when the camera is repositioned or zoomed. Bake geometric augmentation (perspective, zoom, crop) into training, or train per camera with online adaptation.

A reference architecture that survives drift

A production system has five active layers and one retraining loop. The edge does cheap inference; each downstream layer removes false alerts; telemetry feeds a training pipeline that pushes updated models back to the fleet.

Figure 2. Edge inference, edge filter, cloud verifier, and operator triage, with a telemetry plane driving weekly retraining.

Edge inference runs a lightweight detector (YOLO plus a small autoencoder) at 30 fps on an NPU. The edge filter applies a per-camera confidence threshold and temporal smoothing: require the detection to fire on 3 of 5 consecutive frames before it counts. The cloud verifier runs a heavier Transformer or a Vision-Language Model on the handful of uploaded snapshots, plus multi-camera correlation. The operator queue ranks surviving events by confidence. The telemetry plane tracks confidence distributions per camera per time of day and feeds the training pipeline, which mines hard negatives and retrains on a cadence.

The point of the layering is that intelligence lives in the pipeline, not only the model. For the hardware tier behind the edge layer, see our edge AI for video surveillance guide; for where events go after detection, our video analytics integration write-up.

The data pipeline: labelling, augmentation, synthetic data

The model is a few weeks of work. The data pipeline is what you maintain for years, and it is where most of the accuracy actually comes from.

Labelling. Active learning beats brute force. The model proposes the frames it is least sure about, a human labels those, and the loop cuts labelling time roughly 5–10× versus random sampling. CVAT, Roboflow Annotate, and Label Studio all support it.

Augmentation. Brightness and contrast, weather overlays, perspective shifts, motion blur, sensor-noise simulation. Albumentations is the standard Python library; augmenting 5–10× the original set is normal practice.

Synthetic data. Unity, Unreal Engine, NVIDIA Omniverse, and BlenderProc can render physics-accurate surveillance footage for rare events (fights, theft, fire) where real clips are scarce. Domain randomisation lets synthetic and real footage blend without a visible seam.

Hard-negative mining. The single most effective technique on this list. Periodically sample the false positives your system produced, relabel them as normal, and retrain. That pushes the decision boundary in the right direction using the exact mistakes your cameras keep making, which is far more valuable than generic extra data.

False-positive control: the metric that decides adoption

A model at 95 % recall and 30 % false-positive rate teaches operators to ignore alerts within two weeks. Once that trust is gone, the system is dead even at 99 % recall. Three techniques compress false positives, and they stack.

1. Per-camera thresholds. Raise the confidence threshold on cameras with a high observed false-positive rate and keep sensitive cameras sensitive. Per-camera tuning routinely outperforms a single global threshold, because a parking lot and a loading dock do not share a normal.

2. Temporal smoothing (3-of-5). Require a detection on 3 of 5 consecutive frames before alerting. Single-frame artefacts (sensor noise, a brief occlusion) drop out. It adds roughly (M−1) frame intervals of latency, on the order of 100–200 ms, and removes a large share of transient false positives.

3. Cross-camera correlation. A perimeter breach should show on adjacent cameras. A lone single-camera flag with no corroboration is downgraded. This trims false positives further, at a small recall cost you tune deliberately.

A worked false-positive model

Here is the arithmetic behind the layered filter, shown so you can plug in your own retention rates. The numbers are illustrative, not measured on any one deployment; the multipliers are the design variables you tune per site.

Start with 100 raw detections per camera-night. Each layer keeps a fraction:

Per-camera threshold keeps 64 %: 100 × 0.64 = 64.
Temporal 3-of-5 keeps ~40 % of those: 64 × 0.40 ≈ 26.
Cross-camera correlation keeps ~65 %: 26 × 0.65 ≈ 17.
Cloud VLM verification keeps ~65 %: 17 × 0.65 ≈ 11.

The operator sees 11 alerts instead of 100: an 89 % reduction from filtering alone, no bigger model and no new hardware. Recall does dip at each stage, which is why the last two layers stay tunable and why cross-camera correlation is optional on sites with sparse coverage. Swap in your real retention rates and the same spreadsheet tells you where the recall budget is going.

Figure 3. Illustrative model: each filtering layer removes a share of alerts before one reaches an operator.

Drift detection and retraining cadence

A model trained on summer footage degrades in winter. New camera angles, new construction phases, new vehicle types all erode accuracy quietly. The early signal is a shift in the confidence distribution: track the mean and variance of model confidence per camera per week, and a sudden drop or widening flags drift before customers do.

Retraining cadence. Stable production: retrain monthly with a fresh hard-negative batch. A new deployment or an environment shift: retrain weekly for the first two months, then settle to monthly. The pipeline has to be automated, because manual retrain cycles fall behind real-world drift.

Canary deployment. Ship a new model to 5 % of cameras and watch false-positive rate, recall on a held-out validation set, and the confidence distribution for seven days. If it stays clean, ramp to 25 % and then 100 %. Keep the previous version on disk for instant rollback.

Benchmarks: UCF-Crime, ShanghaiTech, MVTec AD

Use public benchmarks to sanity-check an approach, never to promise a field number. Deployment conditions decide accuracy, and no public set matches your cameras.

UCF-Crime is the reference weakly-supervised set: 1,900 untrimmed videos (1,610 train, 290 test), 13 anomaly classes, about 128 hours (Sultani et al., 2018). Frame-level AUC at the 2025 state of the art reaches 90.33 % for π-VAD (CVPR 2025), with RefineVAD at 88.92 % and VadCLIP at 88.02 % (VadCLIP, AAAI 2024).

Figure 4. UCF-Crime frame-level AUC for recent weakly-supervised models (2024–2025).

ShanghaiTech Campus covers 437 videos across 13 fixed cameras, with one-class frame AUC around 97–98 % at the top end. MVTec AD is single-image industrial defect data: 15 categories, 5,354 images, where PatchCore and EfficientAD clear 99 % image AUROC. It is less about video and more a clean way to benchmark autoencoder-style detectors.

The gap between a 90 % benchmark AUC and a trustworthy field system is exactly the pipeline work in the sections above: feature choice, filtering, drift handling.

Build vs buy

Four routes, in rough order of effort. The right answer depends on how specific your anomaly is and how much your fleet costs to run.

Camera-bundled analytics (Axis, Hanwha, Avigilon). Where it wins: zero integration, decent generic person and vehicle detection. Where it breaks: vertical-specific events like PPE compliance, chain-of-custody, or construction-equipment behaviour that the vendor never trained for.

Cloud APIs (AWS Rekognition, Google Cloud Vision). Where it wins: fastest start, no ML team. Where it breaks: per-frame cost dominates at fleet scale, and you only get generic detection with no vertical specialisation.

Open-source pre-trained models (YOLO26, MMDetection, anomalib). Where it wins: strong starting points, no licence fees; YOLO26 is NMS-free and edge-optimised (Ultralytics, 2026). Where it breaks: you own the fine-tuning and the MLOps to keep it alive.

Custom build. The right call when the anomaly is domain-specific and off-the-shelf misses it, when compliance or data residency forces on-prem or your-VPC inference, or when fleet-scale cost justifies the engineering. This is usually where our AI integration and video surveillance teams come in.

How we approach it: EyeBuild on construction sites

The constraint. Construction sites have no wired power and no wired internet. EyeBuild is a solar-powered camera with a 14-day battery and a 3-day reserve, a 4K UHD sensor, 360° PTZ, dual night vision, and a 4G/5G uplink. Because bandwidth is metered and often weak, inference has to run on the device, not in the cloud.

What that forces. On-device human and vehicle recognition with automated alerts, a strict edge filter so the 4G link only carries confirmed events, and 30-day cloud retention for the clips that matter. The false-positive problem is sharp here: wildlife, blowing tarps, and passing headlights all move, and every wasted alert costs battery and bandwidth. That is why the layered filter in the worked model above matters more than a heavier detector on a device that runs on sunlight.

The same pattern reuse sits behind VALT on the regulated side, where 770+ US organizations run event-based footage search under HIPAA. Want us to sketch the edge-versus-cloud split for your own site constraints? Book a 30-min call.

A decision framework: pick your algorithm in five questions

Answer these five in order and you land on a family without a survey of the literature.

Q1. Do you have labelled anomalies? None: reconstruction (autoencoder). A few: weakly supervised. Many: a supervised classifier.

Q2. Is the anomaly temporal? Single-frame (an intruder visible at moment X): autoencoder or distance-based. Time-evolving (loitering, abandoned object): sequence-based.

Q3. What is your latency budget? Under 10 ms: statistical or distance-based. 10–100 ms: autoencoder or a shallow Transformer. Over 100 ms: a full cloud Transformer or VLM.

Q4. Edge or cloud inference? Edge NPU (Hailo-8 at 26 TOPS, Jetson Orin Nano Super at $249): autoencoder or distance-based. Cloud: a full Transformer or VLM ensemble.

Q5. What is your drift profile? Fixed camera, stable light: simpler models are fine. Outdoor, multi-season, or a camera that moves: invest in the drift-aware retraining loop before anything else.

Figure 5. The five questions as a decision path, then a latency filter that picks edge versus cloud.

Five pitfalls to avoid

1. Optimising recall before precision. Recall feels safer to chase, but precision is what keeps operators engaged. Tune for precision first, then raise sensitivity once the alerts are trusted.

2. Training on one camera, deploying on many. A model fit to a single angle fails on the rest. Include multi-angle data or train per camera with online adaptation.

3. Shipping without drift monitoring. With no confidence-distribution tracking, you find out the model degraded when a customer complains. Bake the telemetry in from day one.

4. Skipping temporal smoothing. Single-frame detections are noisy. Requiring 3-of-5 agreement removes most transient false positives for a small, tunable recall cost.

5. Treating privacy as a later problem. Anomaly detection often captures faces or profiles behaviour. Plan on-device redaction, biometric handling, and retention rules from day one, not after the first audit.

Compliance: GDPR and the EU AI Act

If your cameras capture faces, GDPR treats those images as biometric special-category data under Article 9. Large-scale monitoring needs a Data Protection Impact Assessment, plus data minimisation and retention limits. The engineering answers are concrete: blur faces on-device before any frame leaves the camera, send events rather than raw streams, and set hard retention windows.

The EU AI Act adds a second layer. Article 5 prohibitions have applied since 2 February 2025; the transparency duties in Article 50 land on 2 August 2026. Under the 2026 Digital Omnibus agreement, high-risk obligations for Annex III systems were deferred to 2 December 2027, with product-embedded systems following on 2 August 2028. Penalties reach up to €35M or 7 % of global turnover, so workplace or public-space monitoring deserves a legal read before launch, not after.

When not to build custom anomaly detection

Honest answer first: plenty of projects should not build this at all. A custom pipeline earns its keep only when the anomaly is specific, the fleet is large, or compliance forces control of the stack.

Skip a custom build when generic person and vehicle detection already solves your problem: camera-bundled analytics or a cloud API will be cheaper and faster to stand up. Skip it when your fleet is a handful of cameras, because per-frame cloud costs stay trivial and there is no scale to amortise an ML team against. Skip it when you have no path to collect and label footage from your own site, since without your data a custom model has no advantage over an off-the-shelf one.

Build custom when the off-the-shelf model keeps missing your specific event, when data residency or biometrics rules push inference on-prem, or when fleet-scale cloud bills already dwarf an engineering budget. If you are not sure which side of that line you are on, that is a 30-minute conversation, not a six-month project.

KPIs to measure

Quality KPIs. Precision (target above 70 % before trust is built, above 85 % after). Recall on critical events (target above 75 %). False positives per camera per day (target below 3). AUC on your own held-out validation set (target above 0.85).

Business KPIs. Operator engagement (share of alerts triaged inside SLA), customer-reported false-positive count, and mean time to detect a critical event. These are the numbers a buyer actually feels.

Reliability KPIs. Drift signal (a week-over-week mean-confidence change above 5 % triggers an alert), OTA model-deployment success rate (target 99.8 %), and inference latency at p99 (target below 50 ms on an edge NPU).

FAQ

What is video anomaly detection?

Video anomaly detection is the task of automatically flagging events in a video stream that differ from learned normal behaviour, such as intrusion, loitering, a fall, or fire. In surveillance it usually runs as a pipeline: a detector extracts features, an anomaly model scores them, and filtering layers cut false positives before an operator sees an alert.

Isolation forest or one-class SVM, which is better?

Isolation forest scales better and trains faster on large sets (millions of samples). One-class SVM is sharper on small sets but degrades above roughly 50k samples. For surveillance, default to isolation forest unless your training set is under about 5k.

Can a Vision-Language Model do anomaly detection?

Yes, for cloud-side verification. Send an edge-flagged frame to a current VLM (Qwen3-VL, Gemini 2.5, or a GPT-4o-class model) with a prompt like “is this person doing anything unusual?” and use the answer as a second opinion. It adds 1–3 s and a per-frame cost, so run it only on flagged events, never every frame.

How long does a custom model take to build?

Fine-tuning a pre-trained YOLO plus autoencoder on your domain: about 2–3 weeks. From scratch with custom data: 6–10 weeks. A production deployment with monitoring and a retraining loop adds another 4–6 weeks. Reusing patterns from prior builds shortens the front end.

What dataset size do I need?

For an autoencoder, 50–100 hours of normal footage covering day, night, and weather. For a supervised model, 500+ labelled examples per anomaly class. Augmentation extends both 5–10×, and synthetic data fills rare-event gaps.

Does video anomaly detection comply with GDPR?

Only if you design for it. Capturing faces or biometrics brings GDPR Article 9 into play. Mitigate with on-device face redaction, event-only transmission, retention limits, and a DPIA. If it is workplace or public-space monitoring, the EU AI Act may also classify it as high-risk.

Can edge hardware run anomaly detection at 30 fps?

Yes, on current edge NPUs. A Hailo-8 (26 TOPS) or a Jetson Orin Nano Super (up to 67 INT8 TOPS, $249) runs a YOLO-class detector plus a lightweight autoencoder at 30 fps at 1080p within a low power budget. Weaker hardware needs frame skipping or a smaller model.

How do I evaluate an anomaly detection vendor?

Run a 4-week pilot with the top two or three candidates on your real footage. Measure precision and recall on your labelled validation set, false-positive rate over two weeks of production, and drift behaviour as conditions change. A vendor benchmark on the vendor’s own data does not transfer to your cameras.

What to read next

Ready to ship anomaly detection operators trust?

Generic algorithms do not survive video. The four families (statistical, distance-based, reconstruction, sequence-based) carry different costs and blind spots, so pick by your data and latency budget, and reach for an ensemble when false positives are the enemy.

Precision comes before recall in the first 90 days. Layered filtering, shown in the worked model, does most of the heavy lifting on its own. Drift monitoring is not optional, and the retraining loop has to be automated or it falls behind the seasons. Design for GDPR and the EU AI Act from day one.