ITU-T P.808 (06/2021) is the framework for crowdsourced subjective testing - viewers rating clips on their own devices at home, instead of in a controlled lab. Written originally for speech quality, its approach has been adopted by the video field. Crowdsourcing trades control for scale and speed: you cannot set or measure the viewer's lighting, display, or distance, so P.808 leans on layered reliability defences in place of an operator in the room - qualification screens, hidden gold-standard clips, attention traps, and time-on-task gates - followed by heavy after-the-fact filtering. Rejection rates can be very large; one 2025 study dropped about 92 percent of participants after rigorous consistency screening. Yet with strict screening, crowdsourced MOS can correlate with lab MOS at around 0.95, which is why the method is trusted for the right questions. The rule of thumb: a lab answers which encode is better under ideal viewing, a crowd answers which is better in the wild, and you must say which you ran.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
ITU-T P.808 makes crowdsourced quality tests usable by adding gold questions and attention checks that filter careless raters. We wrote seven volumes covering the whole video stack, and for P.808 start with Volume 1: subjective scores and rating scales, quality metrics and encoder comparisons, with Volume 3 on QoE. Written by Nikolay Sapunov, CEO at Fora Soft.
Thinking of crowdsourcing quality ratings under ITU-T P.808 to save lab time?
Tell us what you need to compare: we set up a P.808 crowdsourced test with reliability checks and estimate cost and turnaround. 250+ video products shipped since 2005.
