Mode decision is the process inside an encoder of choosing how to code each block — which prediction mode (intra or inter), which block partition (one 64×64 or four 32×32 or sixteen 16×16, etc.), which reference frame, which transform, which quantization. Modern codecs offer hundreds of valid combinations per block, and the encoder has to pick one. Mode decision is where almost all of the encoder's compute time and almost all of its quality differentiation lives.
The mechanism is straightforward in concept: for each candidate mode, the encoder computes its lagrangian-cost (rate plus distortion weighted by lambda), then keeps whichever mode has the lowest cost. The combinatorial explosion is the practical problem: with hundreds of modes times millions of blocks per minute of video, exhaustive evaluation is too slow. So real encoders use cleverly designed shortcuts — try a small set of promising modes first, prune branches that look worse than the current best, skip modes that are unlikely to help for this content type. Slow presets evaluate more candidates and find better matches; fast presets evaluate fewer and produce slightly larger files.
For a product team, mode decision is the mechanism behind encoder presets. When you set x264 -preset slow versus x264 -preset ultrafast, you're really choosing how exhaustively the encoder evaluates mode decisions. The compression difference between presets is typically 20–40 %; the encoding time difference is 5–10×. Mode decision is also where commercial encoder vendors compete: their claimed efficiency gains over open-source encoders almost always come from smarter (and more secret) mode-decision heuristics, not from new coding tools.
