A residual is the difference between what the encoder predicted a block would look like and what it actually looks like. After the prediction step (intra or inter), the encoder subtracts the prediction from the real pixels and gets a residual — a block of small numbers, mostly close to zero, that represents the leftover information the prediction didn't capture. The residual is what then gets transformed (dct or similar), quantized and entropy-coded for storage. Everything in the actual bitstream is either a prediction instruction (motion vector, intra mode) or a residual.

Why residuals matter: they're the mathematical reason modern video compresses so well. A perfect prediction would leave a residual of all zeros, which entropy-codes to nearly nothing. A bad prediction leaves a residual essentially as informative as the original pixels, which compresses no better than the input. The whole encoder is engineered to make residuals as small and as close to zero as possible before they hit the quantization stage.

For a product team, "residual" is mostly a piece of jargon you'll hear in encoder documentation rather than a knob you tune. But it's useful for diagnosing where compression goes wrong. Visible blockiness on flat areas? The prediction was good but quantization rounded the residual too aggressively. Visible ghosting or smearing on motion? The prediction was bad and the residual was too big to encode at the available bitrate. Sharp text losing crispness? The intra prediction couldn't capture the edge precisely, leaving a high-frequency residual that quantization mauled. Mapping each artefact back to "prediction failed" or "residual was quantized too hard" makes encoder tuning a lot more methodical.

Book cover: Video Encoding, Volume 1 of the Fora Soft Video Engineering Handbook

The book · Volume 1 of 7

Video Encoding: A Complete Guide to Video Compression — From Pixels and Color to H.264, HEVC, AV1, Quality Metrics, and AI

The residual is what actually gets coded after prediction, so a better prediction leaves less residual and fewer bits. Volume 1 covers the residual in detail: how it is formed, transformed with DCT-style transforms, quantized and entropy-coded, and why residual energy explains bitrate on hard content. Written by Nikolay Sapunov, CEO at Fora Soft.

Read it on Amazon →Kindle & paperback · Fora Soft Video Engineering Handbook

Want to understand why some of your content needs far more bitrate than the rest?

Share examples of easy and hard content: we analyse prediction and residual cost and recommend encoder settings or per-title ladders that cut the bitrate gap. 250+ video products shipped since 2005.

Book a 30-min call →WhatsApp →Email us →