A confidence interval is the small plus-or-minus range every MOS must carry, because a MOS is a sample estimate of what the whole population would have said, not an exact measurement. People disagree, and the spread of their ratings tells you how far to trust the average. The 95% interval is the band that would contain the true MOS in 95 of 100 repeats, computed as the MOS plus or minus a multiplier times the standard error, the standard deviation of the opinion scores divided by the square root of N. ITU-R BT.500-15 uses the normal multiplier 1.96; ITU-T P.910 is more careful for small panels and uses a Student's t-value (about 2.07 for 24 subjects, 2.14 for 15), always a little wider - honesty about having estimated the spread from a handful of people. Because the standard error divides by the root of N, halving the interval takes four times the subjects. A MOS without its interval hides the information needed to judge whether two clips really differ.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
A confidence interval tells you whether two MOS or VMAF scores really differ, so skipping it turns noise into product decisions. We wrote seven volumes covering the whole video stack, and for confidence intervals start with Volume 1: quality metrics, subjective scores and how to compare encoders on rate-quality curves, with Volume 3 on QoE measurement. Written by Nikolay Sapunov, CEO at Fora Soft.
Making codec or ladder decisions from scores without a confidence interval?
Send us your test data: we compute confidence intervals, tell you which differences are real and size the panel or test set you need. 250+ video products shipped since 2005.
