Mean squared error (MSE) is the raw pixel-error measure that PSNR is built on. Read right to left, the name is the recipe: take the error at every pixel (original value minus compressed value), square it, then take the mean of all those squares across the frame. Squaring does two jobs — it makes every error positive so they do not cancel, and it punishes big errors far more than small ones. An MSE of zero means the frames are identical; a larger MSE means more error, though its scale depends on bit depth, which makes the raw number unintuitive. PSNR fixes that by rescaling MSE against the squared peak pixel value and taking a logarithm, so PSNR is, quite literally, MSE expressed in decibels. The key limitation passes to every metric built on it: MSE sums error with no model of where it landed or how the eye masks it, so two very different-looking distortions can share one MSE — the root of PSNR's weak perceptual fidelity.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
Mean squared error is the raw pixel error that PSNR rescales, and it treats every pixel difference the same whether viewers notice it or not. We wrote seven volumes covering the whole video stack, and for MSE start with Volume 1: PSNR and MSE, perceptual metrics, quantization and rate-distortion in H.264, HEVC and AV1. Written by Nikolay Sapunov, CEO at Fora Soft.
Optimizing for MSE and getting encodes that look worse than expected?
Share your encoder tuning and samples: we compare MSE-driven and perceptual tuning on your content and recommend settings. 250+ video products shipped since 2005.
