A dependency-light Python script (standard library only) that plans and screens a crowdsourced subjective video quality test. From your clip count, ratings-per-session, target valid panel (default 35 = the ITU-T P.910 10/2023 uncontrolled floor), and expected reject rate it computes: (1) how many workers to recruit so >=35 survive screening, recruit = valid / (1 - reject_rate); (2) the gold and trapping budget at ~1 of each per 10 real trials, per the open P.808 / P.910 implementations; (3) a ro
Download free PDF
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
This crowd-test planner sizes a crowdsourced subjective video test: workers to recruit for 35 valid raters under ITU-T P.910, the gold and trap budget and screening rules. Our seven-volume handbook covers the full video stack; start with Volume 1 for quality metrics and subjective testing, then Volume 3 for QoE and Volume 6 for AI quality models. Written by Nikolay Sapunov, CEO at Fora Soft.
Running a crowdsourced video quality test and worried about noisy ratings?
Share your clips and goals, and we will plan panel size, traps and screening so the results hold up. 250+ video products shipped since 2005.
You get a realistic estimate, technical recommendations and next steps. Free, no obligations.
