RDO — Rate-Distortion Optimisation — is the central decision-making engine inside a modern encoder. For every block of every frame, the encoder has hundreds of possible ways to code it: which prediction mode, which block size, which transform, which quantization. RDO tries combinations and picks the one that minimises a joint score of "bits used" plus "visual error introduced". Different combinations might cost more bits but look better, or save bits but look worse — RDO finds the sweet spot.
Mathematically it computes a single cost: J = D + λ·R, where D is the visual distortion the choice introduces, R is the number of bits it costs, and λ ("lambda") is a tuning factor that translates between the two. A high λ favours fewer bits even at the cost of more distortion (low-bitrate scenarios); a low λ favours quality (high-bitrate scenarios). The encoder computes J for many candidate codings and keeps the lowest one. That sounds simple but multiplied across millions of blocks per minute of video, it's why slow encoder presets take so long.
For product teams, RDO is the source of two important facts. First, "slow presets compress better" comes directly from RDO — slow presets evaluate more candidate codings, so they find better trade-offs. Second, encoders that tune RDO for perceptual quality (using SSIM or VMAF as the distortion metric instead of raw pixel error) produce visibly better video at the same bitrate. Netflix's per-title encoding, AV1's psychovisual tuning and HEVC's advanced perceptual modes are all variations on the theme of "make RDO smarter about what humans actually see".
