ByteTrack vs DeepSORT vs OC-SORT vs BoT-SORT (2026 Guide)

Updated October 2026: Ultralytics switched its default tracker from BoT-SORT to TrackTrack (CVPR 2025) in June 2026; corrected DeepSORT’s MOT17 scores and the OC-SORT DanceTrack margin; added 2026 independent benchmarks and ByteTrack tuning defaults.

Why This Matters

If your video product needs to count something, follow something, or reason about a journey, you need a tracker. A surveillance system that counts how many people walked through a door needs to know the difference between three different people walking through and one person walking through three times. A retail analytics tool that measures dwell time needs to know that the shopper at the cereal aisle at second 12 is the same shopper at second 47. A sports broadcast that overlays player names on a basketball court needs to know that the body wearing jersey number 23 in frame 1 is still wearing number 23 in frame 1500, even when number 23 walks behind number 12 for half a second. A video-conferencing UI that highlights the active speaker with a bounding box around their face needs to keep that box on the right face when two people swap seats. All of those features are the same problem under the hood — multi-object tracking — and all of them break in the same way if the tracker is wrong. The technology has been around since the 1990s, but the model choices, the failure modes, the deployment patterns, and the evaluation metrics have all shifted since 2022, and the documentation that engineers read on the first day of a project is usually three years out of date. This article gives you the 2026 picture: which tracker to pick, why ByteTrack and OC-SORT replaced DeepSORT as the default, what HOTA measures that MOTA missed for a decade, and how to avoid the four production mistakes that sink most tracking deployments.

ByteTrack vs DeepSORT vs OC-SORT vs BoT-SORT: The Short Answer

ByteTrack is a multi-object tracker from ECCV 2022 that links detections across frames with IoU matching in two passes: high-confidence boxes first, then low-confidence boxes for the tracks left over. It uses no appearance model, scores 63.1 HOTA, 80.3 MOTA and 77.3 IDF1 on the MOT17 test set, and is the fastest sensible default for new projects.

ByteTrack vs DeepSORT: pick ByteTrack. DeepSORT (2017, arXiv:1703.07402) adds a CNN appearance feature to every box, so it costs more CPU, and on the same YOLOX-X detector it scores 61.2 HOTA on MOT17 against ByteTrack’s 63.1. DeepSORT still teaches the pattern every appearance-based tracker uses.

ByteTrack vs BoT-SORT: BoT-SORT is ByteTrack plus camera-motion compensation and optional Re-ID features. It is about two HOTA points better on MOT17 (65.0 vs 63.1); with Re-ID on a CPU it needs roughly twice the compute per stream, so choose it for moving cameras and long occlusions, and ByteTrack when many streams share one server.

OC-SORT and Deep OC-SORT: OC-SORT (CVPR 2023) fixes Kalman-filter drift after occlusion and wins on non-linear motion: 55.1 HOTA on DanceTrack against ByteTrack’s 47.7. Deep OC-SORT (ICIP 2023) adds adaptive appearance features and reaches 64.9 HOTA on MOT17.

What changed in 2026: since Ultralytics 8.4.76 (24 June 2026), model.track() runs TrackTrack, a CVPR 2025 tracker, unless you pass tracker="botsort.yaml" or tracker="bytetrack.yaml". Pin the tracker explicitly in production, so a library upgrade does not change your IDs.

What Multi-Object Tracking Actually Is

A video is a sequence of frames. An object detector — YOLO, RT-DETR, Grounding DINO, whatever you picked in the open-vocabulary detection lesson — runs on each frame independently and emits a list of bounding boxes, each with a class label and a confidence score. The detections in frame 1 and the detections in frame 2 are unrelated objects: the detector does not remember what it saw a moment ago. A tracker sits on top of the detector and stitches those per-frame detections into trajectories. Each trajectory carries a stable track ID — a small integer that says "this is the same object I saw on the previous frame". The track ID is the connective tissue that turns a sequence of detection lists into a list of trajectories that downstream code can reason about.

Why bother? Because almost no useful video feature works on per-frame detections alone. You cannot count cars crossing a line without persistent IDs — every car would be counted dozens of times as it crossed multiple frames. You cannot grade a tennis serve from a video without persistent IDs — you would not know which detection belonged to the server and which to the receiver. You cannot flag a fight in a CCTV stream without persistent IDs — the rule that fires the alert is "the same two people were within one metre for more than two seconds", and that rule requires identity persistence. Every video-AI feature that involves motion, dwell, counting, choreography, or interaction is downstream of a tracker.

The job sounds simple — match the boxes in frame t to the boxes in frame t+1 — but it gets hard fast. Objects occlude each other; the model misses a frame here or there; objects move at non-constant velocity; the camera moves; two objects look similar and trade IDs; one object splits into two detections; two detections of the same object get merged. The history of multi-object tracking is largely the history of getting more robust to those failure modes.

Figure 1. A tracker turns a sequence of independent detection lists into a small set of trajectories with stable identities. Without persistent IDs, every downstream feature breaks.

Tracking-By-Detection — The Pipeline That Won

By 2022, almost every published tracker that mattered had converged on the same skeleton — tracking-by-detection. The pipeline has four stages, and almost every modern tracker is a different choice in each of them.

The first stage is detection. A modern detector emits one bounding box per object per frame, with a class label and a confidence score. The quality of the detector dominates everything downstream — a tracker on top of a 95-percent-recall detector and a tracker on top of an 80-percent-recall detector live in different universes, and the gap between them is far larger than any clever association algorithm can close. Almost every paper since 2022 reports tracker numbers on top of a fixed, public detector (often YOLOX-X or a comparable model) precisely so that tracker comparisons measure tracker quality and not detector quality.

The second stage is motion prediction. For each existing track, the tracker predicts where the object will be in the next frame, based on its motion history. The canonical predictor is the Kalman filter, a 1960 algorithm that models the object's state — in DeepSORT the eight-dimensional vector (x, y, a, h, vx, vy, va, vh): box centre, aspect ratio, height and the velocity of each — as a Gaussian whose mean and covariance get updated each frame by a constant-velocity motion model. Modern trackers all use a Kalman filter or a close cousin; the differences are in the state vector and the noise parameters, not in the algorithm itself.

The third stage is association. Given the predicted track positions and the new detections, the tracker computes a cost matrix between every existing track and every new detection, then solves the assignment problem to match them up. The cost can be a geometric distance (intersection-over-union between the predicted box and the detection box), an appearance distance (cosine distance between deep features of the two boxes), or a weighted combination. The assignment is solved with the Hungarian algorithm, a 1955 result that finds the optimal one-to-one matching in O(n³) time and runs in microseconds at the scales tracking cares about.

The fourth stage is track lifecycle management. New detections that do not match any existing track become new tracks. Existing tracks that do not match any detection get marked lost, and after a few frames of being lost they get killed. Tracks that have just been born are usually marked tentative and have to survive a few frames of consistent detection before they become confirmed. The lifecycle parameters — birth threshold, death threshold, maximum unconfirmed frames — are tuning knobs that matter more than people realise.

Every tracker we discuss below is a different choice inside this four-stage pipeline. ByteTrack changes how the detection threshold is applied in stage 3. OC-SORT changes the Kalman filter's behaviour after a missed detection. DeepSORT adds deep appearance features to the cost matrix. BoT-SORT adds camera-motion compensation to the Kalman filter and appearance embeddings to the cost. The skeleton is the same; the muscles differ.

The Four Tracker Families That Matter In 2026

Every production deployment in 2026 picks one of four open-source tracker families. We walk through them in chronological order, because each one solved a problem that the next one inherited.

DeepSORT — the 2017 classic that introduced appearance embeddings

DeepSORT was published in 2017 by Nicolai Wojke, Alex Bewley, and Dietrich Paulus, as an extension of the 2016 SORT tracker. SORT — Simple Online and Realtime Tracking — was already a respectable baseline: Kalman filter for motion prediction, IoU distance for association, Hungarian algorithm for matching. SORT ran at hundreds of frames per second on a CPU but lost identities through occlusions and crossings because it used only geometry. DeepSORT's contribution was to add a second cost matrix on top of the IoU cost: a cosine distance between deep convolutional appearance features of the detection box and a feature gallery for each track. The appearance features were extracted by a small CNN trained for person re-identification on the MARS dataset. The result was the first tracker that could maintain identities through brief occlusions, and it dominated the literature for the next four years.

DeepSORT's score on MOT17 — the standard benchmark for pedestrian tracking — depends mostly on the detector under it. When the StrongSORT authors re-ran it in 2022 with a YOLOX-X detector and tuned settings, DeepSORT reached 61.2 HOTA, 74.5 IDF1 and 78.0 MOTA on the MOT17 test set, within two HOTA points of ByteTrack. HOTA did not exist in 2017 (it was published in 2020), so this re-evaluation is the number the field compares against. The paper itself is arXiv:1703.07402; its Kalman filter tracks the state (u, v, γ, h, u̇, v̇, γ̇, ḣ), which is the box centre, aspect ratio and height plus their velocities, often written as “x y a h vx vy va vh”.

In 2026, the practical reason to know about DeepSORT is twofold. First, it is the tracker that every tracking tutorial uses, and the one that every junior engineer reaches for first. Second, the appearance-feature pattern it introduced is the foundation that BoT-SORT, StrongSORT, and Deep OC-SORT all build on. The original DeepSORT, however, is no longer the default for new projects. The 2022-vintage trackers below beat it on HOTA and IDF1 and are equally easy to deploy.

ByteTrack — the 2022 detection-driven workhorse

ByteTrack was published at ECCV 2022 by Yifu Zhang and colleagues. It is the single most influential paper in tracking-by-detection since DeepSORT. The contribution is deceptively small: associate every detection box, not just the high-confidence ones.

Every prior tracker discarded detections below a confidence threshold (typically 0.5) before association. ByteTrack's observation was that low-confidence detections are usually still real objects — they are objects that the detector saw faintly because of partial occlusion, motion blur, or unusual pose. Discarding them throws away exactly the detections that the tracker most needs to maintain identity through occlusion. ByteTrack's procedure: first associate high-confidence detections to existing tracks using IoU distance; then take the unmatched tracks and try to associate them with the low-confidence detections, again using IoU. High-confidence detections that fail to match anything become new tracks; low-confidence detections that fail to match are discarded.

The full algorithm is fewer than fifty lines of Python. It uses no appearance features, no deep neural network beyond the detector itself, no learned components. It just changes the order in which detections are used. The result on MOT17 — 80.3 MOTA, 77.3 IDF1, 63.1 HOTA — beat every prior tracker including DeepSORT, BoT-SORT predecessors, and the entire family of trackers that used appearance embeddings. The lesson was that the detector was doing most of the work all along, and a smart cascade over detection confidence captured most of the value.

ByteTrack is the right default for any project where the detector is strong, the objects do not change appearance much, and the scene does not involve aggressive camera motion. It is roughly twice as fast as DeepSORT (since it does not run an appearance feature extractor) and needs less than half the CPU of BoT-SORT-ReID per stream (see the numeric example below). The reference implementation is at github.com/ifzhang/ByteTrack under MIT license (29.6 FPS end to end on MOT17 in the authors’ tests), and it ships as a built-in tracker in Ultralytics YOLO under bytetrack.yaml.

OC-SORT — the 2023 motion-only tracker that fixes non-linear paths

OC-SORT — Observation-Centric SORT — was published at CVPR 2023 by Jinkun Cao and colleagues. The paper identifies a specific failure mode of every Kalman-filter-based tracker, including ByteTrack: after an object is occluded for several frames, the Kalman filter's predicted position drifts because the filter keeps propagating the constant-velocity assumption while no observation arrives to correct it. When the object reappears and the tracker has to associate it, the predicted position has drifted away from the true position, and the tracker fails the match.

OC-SORT's fix is observation-centric re-update. When a previously-lost track gets re-associated with a detection after k frames of occlusion, the tracker does not just update the Kalman filter with the new observation. It also goes back, reconstructs the trajectory between the last observation and the new one using both endpoints, and re-updates the filter as if it had seen consistent observations all along. The state vector at the moment of re-association is therefore based on the actual two endpoints of the gap rather than the drifted prediction. The paper also adds an observation-centric momentum term that uses the direction of recent observations directly in the cost matrix.

The OC-SORT MOT17 numbers — 78.0 MOTA, 77.5 IDF1, 63.2 HOTA — sit right next to ByteTrack on the headline metrics, but the relative improvement is much larger on benchmarks with non-linear motion. On the DanceTrack benchmark, which features acrobatic and dance motion that violates the constant-velocity assumption, OC-SORT scores 55.1 HOTA on the DanceTrack test set against ByteTrack’s 47.7, a 7.4-point gap. The DanceTrack result is the one that matters for any product where the objects move non-linearly: dance instruction, gymnastics, parkour, drone footage, animal tracking, anything in a sports broadcast that involves changes of direction.

OC-SORT does not use appearance features either. Like ByteTrack, it is detector-driven and runs at the speed of the detector itself. The reference implementation is at github.com/noahcao/OC_SORT under MIT license. Deep OC-SORT, the 2023 extension by Maggiolino and colleagues, adds an adaptive appearance feature on top and reaches 64.9 HOTA on MOT17 and 61.3 on DanceTrack, and took first place on MOT20 with 63.9 HOTA at publication.

BoT-SORT — the 2022 tracker that was the Ultralytics default until 2026

BoT-SORT was published in 2022 by Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. It is, in its own words, a "stronger SORT" — built on the same skeleton as ByteTrack but adding two upgrades. First, camera-motion compensation: before applying the Kalman filter's motion model, the tracker estimates how the camera moved between the two frames (using sparse optical flow and the ECC image-alignment algorithm) and warps the predicted state by the camera motion. This single change is what makes BoT-SORT work on handheld and drone footage where the constant-velocity assumption fails because the world is moving relative to the camera, not because the objects are. Second, appearance embeddings: BoT-SORT-ReID, the appearance-augmented variant, attaches a re-identification feature extractor (a BoT-S50 trained on standard person Re-ID datasets) and adds an appearance cost to the association.

BoT-SORT's MOT17 numbers are 80.5 MOTA, 80.2 IDF1, 65.0 HOTA — at publication, the state of the art across all three headline metrics. The reference implementation is at github.com/NirAharon/BoT-SORT under MIT license. For production teams the bigger fact was distribution: from 2023 until June 2026 BoT-SORT was the default tracker in Ultralytics YOLO, with config at ultralytics/cfg/trackers/botsort.yaml, so most model.track() pipelines in the field run it. Ultralytics 8.4.76, released on 24 June 2026, made TrackTrack the default; BoT-SORT is still one flag away, and BoT-SORT-ReID remains the most widely deployed appearance-aware tracker for pedestrians and vehicles.

The cost is throughput. Plain BoT-SORT adds the camera-motion estimation on top of ByteTrack; BoT-SORT-ReID also runs an appearance feature extractor on every box, and on a CPU that roughly doubles the per-stream cost (32.6 vs 14.4 ms per frame in the example below). For most production deployments that overhead is acceptable — single-stream pedestrian or vehicle tracking on a modern CPU still runs at 30+ frames per second — but for high-density scenes or many simultaneous streams the cost adds up.

Figure 2. The four tracker families that dominate production in 2026. ByteTrack is the throughput pick; BoT-SORT is the accuracy pick; OC-SORT is the non-linear-motion pick; DeepSORT is the legacy reference.

What changed in 2025–2026: TrackTrack, McByte and the library shake-up

Two things moved since the four families above settled in. The first is new trackers. TrackTrack (Shim, Ko, Yang and Kim, CVPR 2025) changes association to the track’s point of view: each track takes the closest detection among all candidates, and new tracks are not started from boxes that overlap a live track. It reports state-of-the-art results on MOT17, MOT20 and DanceTrack, and in June 2026 Ultralytics made it the default (tracktrack.yaml: high-confidence threshold 0.6, second-pass threshold 0.25, 30-frame buffer, sparse-optical-flow camera-motion compensation, Re-ID off by default). McByte adds a segmentation-mask cue to a ByteTrack-style tracker. On the other end of the scale, Meta’s SAM 3, released in November 2025, detects, segments and tracks objects in video from a text prompt; it is a GPU-heavy annotation and analytics tool rather than a replacement for a tracker that must hold 30 FPS on an edge box.

The second is independent numbers. Roboflow’s open-source trackers library (Apache 2.0, v2.3.0, March 2026) runs every tracker on the same YOLOX detections, which is the closest thing to a fair comparison. With default parameters:

Two readings matter. The ranking from the papers holds — BoT-SORT above OC-SORT above ByteTrack on MOT17 — but the gaps shrink once everyone shares one detector and default settings. And licences now differ more than accuracy: the reference ByteTrack, OC-SORT and BoT-SORT repositories are MIT, Roboflow trackers is Apache 2.0, while Ultralytics and BoxMOT (v25.0.0, September 2026, with BoostTrack, StrongSORT and Hybrid-SORT) are AGPL-3.0. For a closed-source product, that choice is a legal decision before it is a technical one.

The Metrics That Actually Matter — MOTA, IDF1, And Why HOTA Replaced Both

For a decade the tracking community ranked trackers by MOTA — Multiple Object Tracking Accuracy. MOTA was introduced in 2008 by Bernardin and Stiefelhagen as a single number that combined false positives, false negatives, and identity switches. It worked well enough that it became the headline metric on MOTChallenge, the dominant benchmark for pedestrian tracking, and the number you would see at the top of every tracker paper from 2008 to 2020.

MOTA had a known flaw: it was heavily biased toward detection quality. A tracker that detected every object correctly but swapped identities at every occlusion would score nearly as high as a tracker that never swapped identities at all, because the identity-switch term in MOTA's formula is small compared to the false-positive and false-negative terms. The community responded by introducing IDF1 — an identity-focused F1 score that punishes identity switches much more aggressively. IDF1 was the right metric for any application where identity persistence mattered (multi-target re-identification, person-of-interest tracking) but it overcorrected: it underweighted detection quality in turn.

The resolution came in 2020 with HOTA — Higher Order Tracking Accuracy — proposed by Jonathon Luiten and colleagues. HOTA is an explicit, balanced combination of a detection score (DetA, how well the tracker detects objects in the first place) and an association score (AssA, how well it keeps identities consistent across detections that match the same ground-truth object). The two are combined geometrically — HOTA is the square root of DetA × AssA — so a tracker that scores high on one but low on the other gets penalised. HOTA also reports the two sub-scores separately, so a paper can show which axis a new tracker improved on.

By 2026 HOTA is the metric the field reports. MOTA still appears in every results table, but the leaderboards on MOTChallenge, DanceTrack, and SportsMOT all rank trackers by HOTA. If you read only one number from a tracker comparison, read HOTA.

Numbers from the papers and reference repositories (MOT17 test set, private detection). The DeepSORT row is the StrongSORT authors’ 2022 re-run on a YOLOX-X detector:

The numbers say what we said above: ByteTrack and OC-SORT are within a hair of each other on MOT17; BoT-SORT-ReID and Deep OC-SORT are a step above them; DeepSORT, even on a modern detector, trails by about two HOTA points. The story is different on DanceTrack — where OC-SORT pulls ahead of ByteTrack by 7.4 HOTA points (55.1 vs 47.7) — and on SportsMOT, where Roboflow's 2026 shared-detector re-run puts BoT-SORT 0.8 HOTA ahead of ByteTrack and the mask-assisted McByte another 2.7 ahead. The right tracker depends on the data, not on the leaderboard headline.

Picking The Right Tracker — A Decision Walkthrough

Here is the decision tree we use at Fora Soft when a project asks for a tracking feature. It runs in four questions.

The first question is: is the camera moving? If the camera is on a phone, drone, body-cam, dashcam, or any rig where the operator and the world both move, the constant-velocity Kalman filter at the heart of ByteTrack and OC-SORT will drift because the predictions are made in image coordinates, not world coordinates. BoT-SORT's camera-motion-compensation step is the fix. Pick BoT-SORT for any handheld, mobile, or vehicle-mounted camera.

The second question is: does the motion violate the constant-velocity assumption? Dance, gymnastics, parkour, animal movement, fast sports, and any scenario where objects suddenly change direction breaks the Kalman filter's prediction step. OC-SORT's observation-centric re-update is the fix. Pick OC-SORT for dance, sports, animal-tracking, and any scenario with sudden direction changes, when you need a motion-only tracker. If you can afford appearance or mask cues, benchmark BoT-SORT and McByte too: in Roboflow's 2026 shared-detector re-run they score 57.8 and 67.2 HOTA on DanceTrack against OC-SORT's 54.1.

The third question is: how many simultaneous streams or how dense is the scene? Appearance-based trackers (BoT-SORT-ReID, Deep OC-SORT, StrongSORT) extract a deep feature per detection per frame. On a CPU, that feature extraction is the dominant cost — running BoT-SORT-ReID on sixteen 1080p streams pegs a server. If the budget says "many streams, modest hardware", pick ByteTrack: it does no appearance work, runs at the speed of the detector, and is the right choice for high-density retail analytics and large-scale surveillance.

The fourth question is: how important is identity persistence through long occlusions? If the application can tolerate brief identity switches (vehicle counting, dwell-time estimation, crowd density), motion-only trackers (ByteTrack, OC-SORT) are fine. If identity must persist through five or more seconds of occlusion (person-of-interest tracking, sports broadcast graphics, multi-camera handoff), appearance-based trackers (BoT-SORT-ReID, Deep OC-SORT, StrongSORT) become necessary. Pick BoT-SORT-ReID for long-occlusion identity persistence.

In our experience, eighty percent of projects end up at ByteTrack (fastest, simplest, works) or BoT-SORT-ReID (slower, more accurate; plain BoT-SORT with Re-ID off was the Ultralytics default from 2023 to June 2026). OC-SORT is the special-case pick, and TrackTrack is worth a benchmark run on your own footage if you are already on Ultralytics 8.4.76 or later. DeepSORT is rarely the right answer in 2026 — every problem it was good at is now solved better by one of the three above.

Numeric example — the cost of running a tracker on a 1280 × 720 video at 30 frames per second

Here is the math for a single stream on a modern Intel server CPU, with a YOLOv8-S detector and twenty pedestrians in frame.

YOLOv8-S detection (640 × 640 input, 20 detections): about 14 milliseconds per frame. ByteTrack association (20 detections, 20 active tracks): about 0.4 milliseconds per frame. BoT-SORT-ReID appearance feature extraction (BoT-S50, 20 crops): about 18 milliseconds per frame. BoT-SORT-ReID association: about 0.6 milliseconds per frame.

Pipeline totals per frame: ByteTrack pipeline: 14 + 0.4 = 14.4 milliseconds per frame. BoT-SORT-ReID pipeline: 14 + 18 + 0.6 = 32.6 milliseconds per frame.

Per second: ByteTrack = 432 ms of CPU; BoT-SORT-ReID = 978 ms of CPU.

The ByteTrack stream uses about 43 percent of one vCPU. The BoT-SORT-ReID stream uses nearly a full vCPU all by itself. A 16-vCPU server therefore handles about thirty-five concurrent ByteTrack streams (with headroom) or sixteen concurrent BoT-SORT-ReID streams. The price difference shows up directly in the cloud bill: at AWS c7i.4xlarge pricing (16 vCPUs, $0.714/hour on demand in us-east-1, checked October 2026), ByteTrack costs $0.020 per stream-hour, BoT-SORT-ReID costs $0.044 per stream-hour. Both numbers are small in absolute terms, but the ratio matters when you scale to thousands of cameras.

If you need more streams per server, the move is not to switch to a lighter tracker; it is to drop the processing frame rate. The detector, not the tracker, is most of the bill (14 of 14.4 ms per frame above). Many tracking-driven features — counting, dwell time, fall detection — work at 10 frames per second, and running detection plus tracking on every third frame cuts the cost by about 3×. Re-check the match threshold at the lower rate, because objects move further between processed frames.

Where To Run It — Browser, Edge, Or Server

The deployment-topology decision applies to trackers exactly the same way it applied to pose models in the pose-tracking lesson. The same model can be deployed in three different places, with three very different trade-offs.

In the browser, the tracker runs inside the user's tab in WebAssembly. None of the four trackers we cover has a first-party browser implementation, but ByteTrack and OC-SORT — being motion-only and tiny — have been ported to JavaScript and ONNX Runtime Web. The deployment is appropriate for browser-based gesture and ergonomics features. BoT-SORT-ReID with appearance features is too heavy for typical browser hardware budgets in 2026.

On the edge, the tracker runs on a small box near the camera — a NVIDIA Jetson Orin Nano, an Intel NUC, or a Hailo-8L AI accelerator. All four trackers fit comfortably on Jetson Orin Nano alongside a YOLOv8-S detector. ByteTrack is the throughput pick (more streams per box). BoT-SORT-ReID is the accuracy pick (one or two streams per box, but higher identity persistence). Edge deployment is the default for AI security cameras, intelligent video analytics, retail, and industrial computer vision.

On the server, the tracker runs in your cloud account alongside the detector. RT-DETR or YOLO26 (Ultralytics, January 2026) on GPU, plus ByteTrack or BoT-SORT-ReID on CPU, is a workable pattern. The latency is dominated by the network round-trip (typically 30 to 100 milliseconds) plus the inference time. Server deployment is the default for sports analytics platforms, broadcast contribution, batch processing of recorded footage, and any feature that needs the largest detectors.

The choice between the three is rarely close. The use case picks the topology, and the topology picks the tracker.

Figure 3. The same tracker family deploys differently depending on where the model runs. The topology drives the tracker pick at least as much as the algorithm does.

Four Production Pitfalls That Sink Tracking Features

We have shipped a lot of tracking features into video products at Fora Soft. Four failure modes show up repeatedly. They are not novel — every team rediscovers them — but the cost of rediscovery is large, so it is worth naming them up front.

The first pitfall is blaming the tracker for what is really a detector problem. When a tracker drops identities or counts wrong, the instinct is to swap trackers. The fix usually is not the tracker; it is the detector. A YOLOv8-N detector trained on COCO will miss most small-pedestrian detections at 1080p resolution, and no tracker on earth can stitch identities across missed detections. Always profile the detector first — if recall on your specific data is below ninety percent, fix the detector before touching the tracker. Switching from ByteTrack to BoT-SORT-ReID buys you about two HOTA points on MOT17 (63.1 to 65.0). The detector moves more: the same 2017 DeepSORT algorithm reaches 61.2 HOTA once it sits on a YOLOX-X detector, close enough to ByteTrack that most of the gap people blame on the tracker was the detector all along.

The second pitfall is using DeepSORT in 2026. DeepSORT was the right answer in 2018. It is rarely the right answer now. Every comparable problem it was good at is now solved better by ByteTrack (faster, more accurate) or BoT-SORT-ReID (more accurate, equally easy to deploy). The reason DeepSORT survives is that the tutorials a junior engineer finds on a first project are usually three years old. Default to ByteTrack for new projects; default to BoT-SORT-ReID if you need appearance features. We have rewritten DeepSORT pipelines into ByteTrack pipelines for three clients in the past two years, and the swap was always a net upgrade on latency, throughput, and HOTA together.

The third pitfall is leaving the Kalman filter and lifecycle parameters at their defaults on data the trackers were not tuned for. Every tracker we discussed is published with hyperparameters tuned for MOT17 — mostly pedestrians, well-lit street scenes, a mix of static and slow-moving cameras. Apply that configuration to a drone-view sports broadcast or a CCTV stream of cars on a highway, and the Kalman filter's noise terms are wrong, the maximum-unconfirmed-frames knob is wrong, and the appearance threshold is wrong. The fix is to tune. Specifically: the track_buffer (how long a lost track is kept alive in case it returns) and the match_thresh (the association cost threshold) are the two knobs that move the most. For reference, Ultralytics’ 2026 bytetrack.yaml ships with track_high_thresh 0.25, track_low_thresh 0.1, new_track_thresh 0.25, track_buffer 30 frames and match_thresh 0.8. We habitually run a grid search over the buffer and the match threshold on a held-out video from the actual deployment data before shipping; it is a standard step in our computer vision development services, because the defaults were tuned on MOT17 pedestrians, not on your cameras.

# Ultralytics 8.4+: pin the tracker so an upgrade cannot change your IDs
from ultralytics import YOLO
model = YOLO("yolo26s.pt")
results = model.track("site.mp4", tracker="bytetrack.yaml", persist=True)  # or botsort.yaml / tracktrack.yaml

The fourth pitfall is trusting track IDs across camera boundaries. A track ID is local to a single camera and a single tracker run. The same person walking from camera A to camera B will be assigned different IDs by the two trackers, and no amount of clever single-camera tracking will fix that. Cross-camera identity persistence is a separate problem — person re-identification or multi-camera multi-target tracking — and it requires a dedicated Re-ID model, a shared feature gallery, and a global graph-solving step. We will cover the cross-camera case in a future lesson; for now, if your application crosses cameras, do not pretend the single-camera tracker is doing it.

Figure 4. The four production pitfalls that sink tracking features. None of them is technically new; all of them are rediscovered by every team.

Frequently Asked Questions

What is ByteTrack?

ByteTrack is a 2022 multi-object tracker (Zhang et al., ECCV 2022, arXiv:2110.06864) that associates every detection box, including low-confidence ones, in a second IoU matching pass. It scores 63.1 HOTA on MOT17, uses no appearance model and is MIT-licensed.

Is ByteTrack better than DeepSORT?

For new projects, yes. On the same YOLOX-X detector ByteTrack scores 63.1 HOTA on MOT17 against DeepSORT’s 61.2, and it skips DeepSORT’s per-box CNN feature extraction, so it needs less compute per stream.

ByteTrack vs BoT-SORT: which should I use?

Use BoT-SORT when the camera moves or identities must survive long occlusions; it scores 65.0 HOTA on MOT17 with Re-ID. Use ByteTrack when many streams share one server; in our CPU example it needs less than half the compute of BoT-SORT-ReID per stream.

What is Deep OC-SORT?

Deep OC-SORT (Maggiolino et al., ICIP 2023) is OC-SORT plus adaptive appearance features. It scores 64.9 HOTA on MOT17, 63.9 on MOT20 and 61.3 on DanceTrack.

What is the state vector in DeepSORT?

DeepSORT’s Kalman filter tracks eight values: box centre x and y, aspect ratio a, height h, and their velocities vx, vy, va and vh, under a constant-velocity model (arXiv:1703.07402).

What tracker does Ultralytics YOLO use by default in 2026?

TrackTrack, since Ultralytics 8.4.76 (24 June 2026). Before that the default was BoT-SORT. Pass tracker="botsort.yaml" or tracker="bytetrack.yaml" to choose explicitly.

Which metric should I compare trackers on?

HOTA. It is the square root of detection accuracy times association accuracy, so it punishes both missed objects and ID switches; MOTA overweights detection and IDF1 overweights identity.

For a full build walkthrough that puts a detector and DeepSORT-style tracking into a security product, see our guide to building AI video surveillance with YOLO and DeepSORT.

Where Fora Soft Fits In

We have shipped tracking features into half a dozen product categories at Fora Soft. In video conferencing, we use lightweight ByteTrack on a media-processing worker next to the SFU to keep persistent IDs on participants who change camera angles mid-call — the IDs feed downstream features like active-speaker labelling and automated meeting summaries. In retail and intelligent video analytics, we use BoT-SORT-ReID on Jetson edge boxes for dwell-time, queue-length, and out-of-stock detection — the appearance features matter because shoppers occlude each other and the appearance model resolves the resulting identity ambiguity. In OTT and broadcast contribution, we use BoT-SORT-ReID on cloud GPUs for sports overlays and player-tracking dashboards, where the longer occlusions and high-stakes identity matter. In e-learning, we use ByteTrack to track multiple students in classroom video and feed the trajectories into attention-monitoring features. In surveillance and security, we use BoT-SORT-ReID for fall, fight, and intrusion detection — the appearance features help maintain identity through long static-camera occlusions. In every project the tracker is the second decision; the detector and the topology are decisions one and zero. Our AI video recognition development team ships detector-plus-tracker pipelines on ByteTrack and Jetson, and our computer vision for video surveillance work keeps per-camera identities stable on deployments of thousands of live cameras.

The book

AI for Video: A Complete Guide to Intelligence in Video Products: From Models and Inference to Computer Vision, Speech, Search, and Generative Video

Tracking is one chapter of the computer-vision half of volume 6 of the Fora Soft Video Engineering Handbook: it takes the detector-tracker pair apart the way this lesson does, then walks the rest of the vision stack the trajectories feed — segmentation, pose, OCR — plus the deployment topologies and model artifacts that decide whether BoT-SORT-ReID runs on a Jetson at the camera or on a c7i.4xlarge in your VPC. Volume 7 carries the same models into production and prices them per stream-hour, which is where the 43-percent-of-a-vCPU arithmetic in this article turns into a monthly bill. The seven volumes together walk the whole video stack from the pixel to the production deployment, so the reader who works through them can design and build a system like the ones above rather than assemble it from tutorials. Written by Nikolay Sapunov, CEO at Fora Soft.

Read it on Amazon →Kindle & paperback · Fora Soft Video Engineering Handbook

See our case studies

References

1. Wojke, N., Bewley, A., Paulus, D. Simple Online and Realtime Tracking with a Deep Association Metric (DeepSORT). ICIP 2017. arXiv:1703.07402. The original DeepSORT paper that introduced deep appearance features into the SORT pipeline.
2. Bewley, A., Ge, Z., Ott, L., Ramos, F., Upcroft, B. Simple Online and Realtime Tracking (SORT). ICIP 2016. arXiv:1602.00763. The original SORT paper that DeepSORT extended.
3. Zhang, Y., Sun, P., Jiang, Y., Yu, D., Yuan, Z., Luo, P., Liu, W., Wang, X. ByteTrack: Multi-Object Tracking by Associating Every Detection Box. ECCV 2022. arXiv:2110.06864. The ByteTrack paper that introduced the two-stage detection-cascade association.
4. ByteTrack reference implementation. github.com/ifzhang/ByteTrack, accessed 2026-05-27. The MIT-licensed reference code that ships in Ultralytics YOLO.
5. Cao, J., Pang, J., Weng, X., Khirodkar, R., Kitani, K. Observation-Centric SORT: Rethinking SORT for Robust Multi-Object Tracking. CVPR 2023. The OC-SORT paper that introduced observation-centric re-update.
6. OC-SORT reference implementation. github.com/noahcao/OC_SORT, accessed 2026-05-27. The MIT-licensed reference code.
7. Maggiolino, G., Ahmad, A., Cao, J., Kitani, K. Deep OC-SORT: Multi-Pedestrian Tracking by Adaptive Re-Identification. ICIP 2023. arXiv:2302.11813. The appearance-augmented extension of OC-SORT.
8. Aharon, N., Orfaig, R., Bobrovsky, B.-Z. BoT-SORT: Robust Associations Multi-Pedestrian Tracking. arXiv:2206.14651, 2022. The BoT-SORT paper that introduced camera-motion compensation and Re-ID-based association upgrades.
9. BoT-SORT reference implementation. github.com/NirAharon/BoT-SORT, accessed 2026-05-27. The MIT-licensed reference code.
10. Ultralytics. Multi-Object Tracking with Ultralytics YOLO. docs.ultralytics.com/modes/track, accessed 2026-10-10. First-party documentation for the six trackers shipped with the framework (BoT-SORT, ByteTrack, OC-SORT, Deep OC-SORT, FastTracker, TrackTrack); TrackTrack is the default since v8.4.76, released 2026-06-24.
11. Shim, K., Ko, K., Yang, Y., Kim, C. Focusing on Tracks for Online Multi-Object Tracking (TrackTrack). CVPR 2025, pp. 11687–11696. Reference code: github.com/kamkyu94/TrackTrack (MIT).
12. Roboflow. trackers v2.3.0, tracker comparison. trackers.roboflow.com/latest/trackers/comparison, accessed 2026-10-10. Apache-2.0 library; HOTA, IDF1 and MOTA for SORT, ByteTrack, OC-SORT, BoT-SORT, C-BIoU and McByte on shared YOLOX detections.
13. BoxMOT v25.0.0 (2026-09-09). pypi.org/project/boxmot, accessed 2026-10-10. AGPL-3.0 tracker collection including BoostTrack, StrongSORT, Hybrid-SORT and OC-SORT.
14. Meta AI. SAM 3: Segment Anything with Concepts, released November 2025. Text-promptable detection, segmentation and tracking in images and video.
15. Luiten, J., Ošep, A., Dendorfer, P., Torr, P., Geiger, A., Leal-Taixé, L., Leibe, B. HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking. International Journal of Computer Vision, 2021. The HOTA paper that balances detection and association accuracy in a single metric.
16. Du, Y., Zhao, Z., Song, Y., Zhao, Y., Su, F., Gong, T., Meng, H. StrongSORT: Make DeepSORT Great Again. IEEE Transactions on Multimedia, 2023. The StrongSORT paper that modernises the DeepSORT pipeline with better detector, Re-ID, and post-processing.
17. Stanojevic, V., Todorovic, B. BoostTrack: Boosting the Similarity Measure and Detection Confidence for Improved Multiple Object Tracking. Machine Vision and Applications, 2024. The BoostTrack paper that combines confidence boosting and a soft IoU measure.
18. Sun, P., Cao, J., Jiang, Y., Yuan, Z., Bai, S., Kitani, K., Luo, P. DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion. CVPR 2022. The DanceTrack benchmark that exposes the constant-velocity Kalman filter's weakness on non-linear motion.
19. Cui, Y., Zeng, C., Zhao, X., Yang, Y., Wu, G., Wang, L. SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes. ICCV 2023. The SportsMOT benchmark for sports-tracking evaluation.
20. Bernardin, K., Stiefelhagen, R. Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. EURASIP JIVP, 2008. The original CLEAR-MOT paper that defined MOTA, the metric HOTA replaced.