Mode decision is the process inside an encoder of choosing how to code each block — which prediction mode (intra or inter), which block partition (one 64×64 or four 32×32 or sixteen 16×16, etc.), which reference frame, which transform, which quantization. Modern codecs offer hundreds of valid combinations per block, and the encoder has to pick one. Mode decision is where almost all of the encoder's compute time and almost all of its quality differentiation lives.

The mechanism is straightforward in concept: for each candidate mode, the encoder computes its lagrangian-cost (rate plus distortion weighted by lambda), then keeps whichever mode has the lowest cost. The combinatorial explosion is the practical problem: with hundreds of modes times millions of blocks per minute of video, exhaustive evaluation is too slow. So real encoders use cleverly designed shortcuts — try a small set of promising modes first, prune branches that look worse than the current best, skip modes that are unlikely to help for this content type. Slow presets evaluate more candidates and find better matches; fast presets evaluate fewer and produce slightly larger files.

For a product team, mode decision is the mechanism behind encoder presets. When you set x264 -preset slow versus x264 -preset ultrafast, you're really choosing how exhaustively the encoder evaluates mode decisions. The compression difference between presets is typically 20–40 %; the encoding time difference is 5–10×. Mode decision is also where commercial encoder vendors compete: their claimed efficiency gains over open-source encoders almost always come from smarter (and more secret) mode-decision heuristics, not from new coding tools.

Book cover: Video Encoding, Volume 1 of the Fora Soft Video Engineering Handbook

The book · Volume 1 of 7

Video Encoding: A Complete Guide to Video Compression — From Pixels and Color to H.264, HEVC, AV1, Quality Metrics, and AI

Mode decision picks a prediction mode and partition for every block, and it is where encoders spend most of their time. Volume 1 covers mode decision in detail: rate-distortion optimization, Lagrangian cost, fast decision shortcuts in encoder presets, and how H.264, HEVC and AV1 widen the choices. Written by Nikolay Sapunov, CEO at Fora Soft.

Read it on Amazon →Kindle & paperback · Fora Soft Video Engineering Handbook

Trying to speed up mode decision without losing compression?

Share your encoder presets and hardware: we profile mode decision settings on your content and give you a faster preset with measured quality loss. 250+ video products shipped since 2005.

Book a 30-min call →WhatsApp →Email us →