One page: how to build a trustworthy eval rig for a video AI feature. The five parts of the bench (golden set, feature under test, judge, scorecard, gate); why word-overlap metrics (BLEU/ROUGE) and text-only judges fail on video; when to use a VLM-as-judge that reads sampled frames to check grounding; the three scoring-mode decisions (pointwise vs pairwise, reference-based vs reference-free, coarse vs fine scale); the three bias controls (swap-and-average for position bias, length cap for verbosity bias, cross-family judge for self-preference bias); and the judge-the-judge calibration step (measure agreement with humans via Cohen's kappa before trusting any automated score). Maps to the NIST AI RMF MEASURE function and ISO/IEC 42001.
Download free PDF
The book · Volume 6 of 7
AI for Video: A Complete Guide to Intelligence in Video Products: From Models and Inference to Computer Vision, Speech, Search, and Generative Video
This checklist builds an eval rig for a video AI feature: golden set, judge, scorecard and gate, VLM-as-judge on sampled frames, scoring modes, bias controls and judge calibration. Volume 6 covers the AI features such a rig measures: the lifecycle of an AI feature, model artifacts and the ML basics for decisions, computer vision, speech and search. Written by Nikolay Sapunov, CEO at Fora Soft.
Need numbers you can trust before shipping a video AI feature?
Tell us the feature and what good output looks like: we will build the golden set and judge, calibrate it against human ratings and set a release gate. 250+ video products shipped since 2005.
You get a realistic estimate, technical recommendations and next steps. Free, no obligations.
