The outlier ratio is a validation statistic that captures consistency, the most practical of the three validation questions: how often a metric is wrong by more than human uncertainty can explain. It is the fraction of clips where the metric's prediction error is larger than the viewers' own disagreement on that clip, specifically larger than the 95 percent confidence interval of the MOS, as defined in ITU-T P.1401. A low outlier ratio means the metric rarely surprises you, because even a model with a fine average error can blow one clip in twenty so badly that the mistake is dangerous; the outlier ratio is what catches that. It runs from 0 to 1. Its blind spot is that it depends on how tight the subjective test's confidence intervals are: looser intervals from a noisy panel make the metric look more consistent than it is. The outlier ratio sits beside SROCC, PCC, and RMSE as the four core statistics a trustworthy validation reports together, each guarding against a different way of flattering a metric.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
The outlier ratio shows how often a metric misses viewers by more than their own disagreement, which a high correlation alone can hide. We wrote seven volumes covering the whole video stack, and for outlier ratio start with Volume 1: quality metrics and how they are validated against subjective scores, with Volume 6 on evaluating models. Written by Nikolay Sapunov, CEO at Fora Soft.
Your metric correlates well with MOS — but how often is it badly wrong?
Send us metric scores and viewer ratings: we compute the outlier ratio and correlation and show the content types where the metric fails. 250+ video products shipped since 2005.
