Video Surveillance Anomaly Detection: 7 ML Algorithms 2026

Anomaly detection in video surveillance is a two-layer problem, not a one-algorithm problem. Pick the wrong model and you get one of two failures that both cost money: an alert firehose that trains operators to ignore the screen, or a quiet model that misses the one moment that mattered. We have shipped surveillance and video-intelligence products since 2005 (250+ delivered), and every production pipeline we run pairs a fast unsupervised model that flags candidates with a deep model that understands what the pixels mean. This is the decision framework we actually use, with 2026 hardware, prices, and honest trade-offs.

Why Fora Soft ships surveillance anomaly detection

We are a software development company that has been building real-time video products since 2005, 250+ shipped, a 50-person team with a dedicated ML bench and deep experience in WebRTC, RTSP/ONVIF ingest, and NVIDIA Jetson edge deployment. For anomaly detection specifically, we have run production pipelines on Isolation Forest, One-Class SVM, autoencoders, YOLO-family CNNs, and LSTM/transformer temporal stacks. One of them is VALT, a research-observation recording platform now used by 770+ organizations and 50,000+ active users.

Here is what that means for your product: we don't pick an algorithm off a listicle. We profile your camera topology, latency budget, label economics, and the cost of a miss, then propose a pipeline with both a triage and a confirmation layer. If you want the commercial framing first, our video surveillance development and computer vision for surveillance pages cover it. Everything below reflects real deployments, not a literature survey.

The real-time two-layer detection pipeline

Most surveillance footage is 99% boring. Sending every frame through a deep model burns GPU budget and fires false alerts. The pattern we deploy runs three layers, and the first two do almost all the work.

Figure 1. The real-time pipeline: cheap edge triage cuts the deep model’s load 30–100×, and confirmation cuts the human’s load 10–50×.

Layer 1, triage, on the edge. Extract features (motion vectors, histograms, object boxes) and run a fast unsupervised model, Isolation Forest or GMM, tuned for high recall and moderate precision. Sub-millisecond per frame on a CPU.

Layer 2, confirmation, on a regional server. For flagged candidates only, push the 2–5 second window to a CNN plus a temporal model that returns an interpretable score and a class label.

Layer 3, human review, optional. Above a threshold, send a clip with bounding boxes to an operator queue. The human is the final arbiter on anything consequential.

What good looks like on standard CCTV: sub-second detection and under 1.5% false positives at the Layer-2 boundary. The reason isn't academic. One operator watching 100 cameras can act on maybe one or two alerts an hour before fatigue sets in; a model that fires 40 times a day per camera is just expensive noise.

Isolation Forest: sub-millisecond triage

What it is. An ensemble of random binary trees, first described by Liu, Ting and Zhou at IEEE ICDM in 2008. The fewer splits it takes to isolate a point, the more anomalous it is. It trains on “normal” data only, no labels, in linear time.

Why it matters for surveillance. Inference is sub-millisecond on a commodity CPU, it eats 50–500-dimensional feature vectors (motion, optical flow, object counts), and you can retrain it nightly on the last 24 hours of “normal” without keeping a labeled dataset. It is our default Layer-1 triage model.

Where it breaks. It does not understand pixels. Feed it raw image arrays and it loses to any deep model. Always put a featurization stage in front of it.

One-Class SVM: bounded-normal scenes

What it is. A support vector machine trained only on “normal” data, learning a boundary in feature space. Points outside the boundary are anomalies.

Why it matters for surveillance. It shines when “normal” is tight and well-defined: a secure room at night, an idle production line, a fixed vehicle loop. Kernel SVMs capture non-linear normality that Isolation Forest misses.

Where it breaks. Training scales poorly past tens of thousands of samples, and the ν/γ hyperparameters are sensitive and awkward to tune online. We rarely run it at fleet scale for that reason.

CNNs (YOLO26): frame semantics

What it is. The backbone of modern vision. In surveillance, convolutional networks do three jobs: object detection (YOLO26, RT-DETR), classification (is the unusual object present), and feature extraction for downstream models.

Why it matters for surveillance. Any semantically meaningful anomaly, “person in restricted zone,” “object left unattended,” “weapon drawn,” is best framed as a detection or classification task on a CNN. The 2026 picture: YOLO11 (Sept 2024) is the stable production baseline; YOLOv12 (Feb 2025) went attention-heavy but still needs NMS and exports awkwardly to edge; YOLO26 (Jan 2026) is end-to-end and NMS-free, built for low-power devices, which matters when the model runs on the camera.

Where it breaks. CNNs are frame-level. Anything temporal, loitering, tailgating, an abnormal trajectory, needs a sequence model on top.

LSTMs and temporal transformers: anomalies in time

What they are. Sequence models that eat a series of features, usually CNN embeddings per frame, and output a per-window anomaly score. In 2026 transformers (TimeSformer, VideoMAE) are pulling ahead of classical LSTMs on benchmarks, but LSTMs still win on CPU-only edge boxes.

Why they matter for surveillance. Most interesting anomalies are temporal: loitering, reverse motion, tailgating, dwell-time outliers, abnormal traffic flow. A CNN alone can't see them. A temporal model on CNN features is the proven recipe.

Where they break. Training data for rare temporal anomalies is scarce, so you often need synthetic generation, simulators, or generative augmentation, to get enough positive examples.

Autoencoders: catching the unknown

What it is. An encoder-decoder network trained to reconstruct “normal” frames. Reconstruction error is the anomaly score. Variants include variational autoencoders and ConvLSTM autoencoders for spatiotemporal signal.

Why it matters for surveillance. It's unsupervised. Train on hours of “normal” footage, then flag anything it can't reconstruct. That's the right tool when you don't know the anomaly vocabulary up front.

Where it breaks. It can over-generalize: reconstruct too well and it reconstructs the anomaly too. The fix that works in practice is a memory-augmented autoencoder (MemAE, Gong et al., ICCV 2019), which limits how much the decoder can improvise.

K-Means: behavioural clustering

What it is. A clustering algorithm that partitions samples into K groups. For anomaly detection, points far from any centroid, or alone in a tiny cluster, get flagged.

Why it matters for surveillance. It's cheap, interpretable, and good at trajectory clustering. Normal shoppers trace a few dozen canonical paths; a trajectory that fits no cluster is worth a look.

Where it breaks. You must choose K up front, clusters are assumed spherical, and it's a poor fit for raw pixel data.

Gaussian Mixture Models: background subtraction

What it is. A probabilistic model that represents data as a mix of Gaussian distributions. Points with very low likelihood under every component are anomalies. It's also the classic background-subtraction method.

Why it matters for surveillance. On fixed cameras it's the background-modeling workhorse, OpenCV's MOG2/KNN are GMMs, giving rock-solid pixel-level “this shouldn't be here” masks.

Where it breaks. Moving cameras, gradual lighting drift, and repetitive motion (leaves, flags) all confuse it. Pair it with motion compensation or a learned feature stack.

Algorithm comparison and use-case picks

The seven models are a toolbox, not a ranking. Here they are side by side, then mapped to the anomaly types you'll actually meet.

Figure 2. The same seven models scored green/orange per property, so you can see at a glance which need a partner layer.

Mapped to the jobs you'll meet:

• Perimeter / zone intrusion. YOLO26 detector, then Isolation Forest on trajectory features for dwell-time checks.
• Abandoned object. GMM background subtraction, then CNN classification of the candidate blob.
• Loitering. CNN detection and tracking, then an LSTM or temporal transformer on tracklet features.
• Crowd anomalies (panic, fights, direction reversal). Dense optical-flow autoencoder plus a temporal transformer.
• Industrial / conveyor line. One-Class SVM on features, autoencoder on pixels, a supervised CNN on known defects.
• Retail trajectory analysis. K-Means on person trajectories plus Isolation Forest on basket and dwell features.

Figure 3. Start from the kind of anomaly, follow the branch to a first-choice model, then add a confirmation layer.

The 2026 shift: VLMs and training-free detection

The newest research line pairs a vision-language model with an LLM. LAVAD (CVPR 2024) captions frames with a VLM, then uses an LLM to aggregate and score anomalies over time, with no task-specific training, and it is competitive on the UCF-Crime benchmark (128 hours, 1,900 real surveillance videos, 13 anomaly classes). RelVid (2025) pushes weakly-supervised VLM methods to the state of the art on the same set.

Here's the catch, and it's the honest part most vendor decks skip: VLM pipelines are heavy and slow. They're a strong fit for Layer-2 confirmation and for explainability (they emit a text reason a human can read), not for the sub-millisecond Layer-1 triage that keeps a fleet affordable. We'd run a classical triage model at the edge and reserve a VLM for the small stream of confirmed candidates, where its reasoning earns its cost. For the tool-and-vendor angle, see our companion guide on anomaly detection models for video surveillance.

Edge vs cloud and the 2026 hardware

The 2026 default is edge-first hybrid. Layer-1 triage runs on the camera or a small edge box; Layer-2 deep models run on a regional server, on-prem or in a VPC; the operator UI is hosted per your privacy posture. Before the models even see a frame, the video has to be ingested and decoded, our primer on video encoding covers that layer.

The Jetson Orin Nano Super is the story of the year: 67 TOPS at $249, up from 40 TOPS at $499, a 1.7× jump delivered partly by a software update. Edge triage also cuts stream egress hard, on VALT we measured a 78% drop versus full-cloud processing, which is a bandwidth line item and a privacy argument at the same time.

Cost math: what a pipeline really costs

Take a 50-camera retail chain going edge-first. One Orin Nano Super handles about three camera feeds for triage, so 17 boxes at $249 is $4,233. Add one shared regional confirm node, amortized at roughly $6,000, and a label budget of 8,000 events at $0.35, so $2,800. Hardware plus data lands near $13,000 one-time, with integration and tuning inside the build.

Figure 4. Illustrative, conservative numbers. Models are about 30% of the project; data, labels, and deployment are the rest.

We keep estimates conservative because we build with Agent Engineering, which makes our delivery faster and cheaper than typical agency rates, so we'd rather under-promise a number than inflate one. If a figure depends on your exact camera mix, we won't guess it in a blog post. That's a scoping-call answer.

Mini case: VALT research observation platform

The problem. VALT is a research-observation recording platform used by 770+ organizations and 50,000+ active users. Operators needed automated flags for session anomalies, an unexpected person in the room, gear misuse, odd participant behaviour, without drowning in false alerts.

The stack we built. Layer 1 was GMM motion and occupancy filtering on the edge recording boxes. Layer 2 was a YOLO-family CNN for person and gear detection plus a small LSTM on detection trajectories for dwell-time and trajectory anomalies. Layer 3 was a human operator queue in the VALT web UI, with one-click clip annotation that fed straight back into labeling.

The outcome. False positives fell from about 11% to under 1.2% after three months of the operator-feedback loop. Time to surface a session anomaly dropped from 45 minutes of manual review to about 9 seconds. Edge-box GPU stayed under 30%, leaving headroom for more analytics. Want a similar assessment for your fleet? Book a 30-minute call.

A decision framework in five questions

1. Is the anomaly semantic, temporal, or statistical? Semantic goes to a CNN, temporal to an LSTM or transformer, statistical to Isolation Forest or GMM. Most real systems are two of the three.

2. Do you have labels, or only “normal” footage? No labels points you at unsupervised models (Isolation Forest, GMM, autoencoders). Labels open up supervised CNN accuracy, at a data cost.

3. What's your latency and hardware budget? Sub-millisecond CPU triage vs GPU-backed confirmation decides how much runs on the camera vs a regional server.

4. How expensive is a miss vs a false alarm? That ratio sets your thresholds, and whether you need multi-tier alarm queues instead of one.

5. Buy, build, or hybrid? Generic use cases and a small team point to SaaS; unique data, compliance, or a product you sell point to a custom build. If you're unsure, that's exactly what a scoping call is for, we do this weekly.

When not to build custom anomaly detection

Honesty sells better than a pitch. Skip a custom build if your use case is generic (people counting, basic motion alerts) and an off-the-shelf VMS or camera app already does it, you'll spend months matching a commodity feature. Skip it if you can't commit to collecting site data and labeling, the model is the easy part and it will fail on clean benchmark data alone.

And skip real-time biometric identification of people in public spaces unless your legal team has signed off: in the EU that is a prohibited practice under the AI Act with only narrow, judicially-authorized exceptions. A custom build makes sense when the anomaly is specific to your operation, the cost of a miss is high, and you own the data and the roadmap.

Five production pitfalls we have fixed

1. Training on clean benchmark data, deploying on messy feeds. UCSD Ped2 and ShanghaiTech are great for papers; they are not your camera. Collect a week of site data before you freeze a model.

2. No feedback loop to labeling. Operators dismiss false positives all day. Capture those dismissals and feed them back into training, this alone halved VALT's false-positive rate in 90 days.

3. Ignoring concept drift. A factory that changes shifts, or a store that changes layout, breaks your “normal” model. Schedule nightly retraining on the last 7 days.

4. Single-threshold alarms. One threshold can't serve both “keep an eye on this” and “call the police.” Use multi-tier thresholds with distinct review queues.

5. Not budgeting for labels. A production CNN/LSTM pipeline needs 5–20K labeled events. Plan $0.20–1.00 per labeled event and 6–12 weeks of throughput, not an afternoon.

KPIs: what to measure

Quality KPIs. Precision, recall, and F1 per anomaly class, plus false positives per camera per day. Target under 1.5% FPR at the confirmation boundary and recall high enough that you never explain a missed event to a client.

Business KPIs. Alerts per operator per hour (keep it to one or two actionable), mean time to surface an anomaly, and the shrinkage, loss, or incident metric the system exists to move.

Reliability KPIs. End-to-end latency (ingest to alert), edge-box GPU load and thermal headroom, and model-drift signals such as a rising baseline anomaly score that means it's time to retrain.

Compliance: GDPR and the EU AI Act

GDPR, the EU AI Act, and state-level biometric laws all apply to surveillance. Two dates matter for planning: the AI Act's Article 5 prohibitions have been in force since 2 February 2025, including real-time remote biometric identification in public spaces for law enforcement (with narrow exceptions); obligations for high-risk systems, which include biometrics, apply from 2 December 2027.

Edge-first processing simplifies most of this, because raw video need not leave the premises, only alert metadata and short evidence clips. Do face and license-plate blurring at the edge when a stream must leave, keep a data-retention policy, and sign a DPA with every ML vendor. For a deeper build-side treatment, see our AI anomaly detection playbook and the AI video analytics architecture guide.

Frequently asked questions

Which algorithm is best for anomaly detection in video surveillance?

No single algorithm. The 2026 production pattern is Isolation Forest or GMM at the edge for triage, plus a YOLO-family CNN and a temporal LSTM or transformer for confirmation. Single-algorithm pipelines either flood operators or miss real events.

Do I need labels to train an anomaly detector?

Not for Layer 1. Isolation Forest, GMM, One-Class SVM, and autoencoders all train on “normal” data alone. Layer 2 semantic detection generally needs 5–20K labeled frames or clips for production accuracy, so budget labeling before committing to a CNN.

Are vision transformers replacing CNNs in 2026?

On benchmarks, yes for large models. In production surveillance, CNNs still dominate because edge inference speed and quantization support are better. Expect transformers to win at the regional-server tier first, then reach the edge as hardware catches up.

What false-positive rate should I target?

Operator tolerance sets the number. One operator on 100 cameras sustains about one or two alerts per hour before fatigue. If your model fires 40 times a day per camera, that's noise. Target under 1.5% FPR at the Layer-2 decision boundary for most surveillance cases.

Can I run all of this on the camera?

Layer 1, yes, Isolation Forest, GMM, and light CNNs fit modern SoC cameras (Ambarella CV5) and Jetson-class boxes. Layer 2 (heavy CNN plus LSTM or transformer) is happier on a regional server shared across cameras. The best cost-to-latency balance in 2026 is edge triage, regional confirmation.

How do vision-language models fit in?

VLM pipelines like LAVAD produce explainable, training-free anomaly reasoning and score well on UCF-Crime, but they are heavy. Use them for Layer-2 confirmation and human-readable explanations on a low volume of events, not for scoring every frame.

What about privacy and the EU AI Act?

Edge-first processing keeps raw video on site, which simplifies GDPR compliance. Note the AI Act: Article 5 prohibitions (including real-time remote biometric ID in public spaces) apply since 2 February 2025, and high-risk biometric obligations from 2 December 2027. Blur faces and plates at the edge, keep a retention policy, and sign a DPA with every vendor.

How long does a production system take to build?

A minimum viable pipeline on a defined scene: 6–10 weeks. A production-grade, multi-scene, multi-site system with a feedback loop and operator UI: 4–6 months. Labels and data collection are the long pole, not the ML work.

What to read next

Ready to ship noise-free surveillance alerts?

The seven algorithms are a toolbox: Isolation Forest and GMM triage, One-Class SVM locks down bounded scenes, CNNs read frame semantics, LSTMs and transformers read time, autoencoders catch the unknown, K-Means clusters behaviour. The shipping question isn't “which one” but “which two-layer pipeline,” and in 2026 the honest answer pairs a fast unsupervised triage at the edge with a deep confirmer on a regional server.

The models are the easy 30%. Data, labels, drift handling, edge deployment, and compliance are the 70% that decides whether the system earns its keep. That's the part we've done 625 times, and it's the part worth a conversation.

Sources: Liu, Ting & Zhou, Isolation Forest (IEEE ICDM 2008); Ultralytics YOLO26 / YOLO11 docs; NVIDIA Jetson Orin Nano Super; Hailo-8 / Hailo-10H product docs; EU AI Act Article 5; UCF-Crime (Sultani et al., 2018), LAVAD (CVPR 2024), RelVid (2025); Fora Soft VALT deployment data, with client permission.