A reference frame is a previously decoded frame that the codec can use to predict future frames. When a P-frame says "this block is similar to a block from 200 ms ago, shifted slightly to the left", the "frame from 200 ms ago" is the reference. Reference frames live in the decoder's memory (the dpb — Decoded Picture Buffer) for as long as future frames might need them, then get discarded to make room for new ones.
Modern codecs allow multiple reference frames simultaneously — H.264 up to 16 in High profile, HEVC similar, AV1 up to 8 named references. The encoder freely picks which one (or two, for B-frames) best matches the block it's trying to encode. More reference options mean better predictions, smaller residuals and smaller files. The trade-off: every reference frame the encoder keeps active consumes decoder memory, which is why level constraints cap the maximum number of references for each resolution tier.
For a product team, two practical insights about reference frames. First, encoder presets like slow and veryslow use more references and find better matches; that's part of why slow presets compress better. Second, codec "intelligence" lives largely in reference frame selection — picking the right reference from a pool of 16 candidates is a key part of mode decision, and it's where modern AI-driven encoders (and traditional psycho-visual heuristics) make smart choices that simple math wouldn't. If you ever see a video pipeline mysteriously producing smaller files at the same VMAF after an encoder update, reference frame selection improvements are usually responsible.
The book · Volume 1 of 7
Video Encoding: A Complete Guide to Video Compression — From Pixels and Color to H.264, HEVC, AV1, Quality Metrics, and AI
Reference frames decide how far back an encoder can look for a match, trading memory and decoder load for compression. Volume 1 explains reference frames in detail: decoded picture buffers, multiple references, long-term references, and how GOP structure and B-frames use them in H.264, HEVC and AV1. Written by Nikolay Sapunov, CEO at Fora Soft.
Tuning reference frame count and hitting decoder limits on some devices?
Send us your encoder settings and device list: we set reference frame counts and levels that save bitrate while staying within every target decoder's limits. 250+ video products shipped since 2005.
