A/B testing splits live users into groups that each see a different variant - of an algorithm, layout, artwork, or flow - and measures which moves a target metric like watch time, conversion, or retention. It replaces opinion with evidence and is the engine of data-driven product development at streaming services.
Streaming experimentation is famously deep: recommendations, artwork, onboarding, and even encoding choices are tuned through controlled experiments. Doing it well requires a real experimentation platform - proper randomization, sample-size and significance discipline, guardrail metrics, and awareness of long-term effects - so that short-term wins do not mask harm to retention or QoE.
The handbook · 7 volumes
Fora Soft Video Engineering Handbook
A/B testing on a streaming service compares player, ABR, or UI variants on real viewers, and QoE metrics decide which one wins. The seven volumes cover the video stack; for A/B testing start with Volume 3 on ABR algorithms, players, and QoE monitoring, with encoding ladders in Volume 1 and AI models in Volume 6. Written by Nikolay Sapunov, CEO at Fora Soft.
Planning A/B testing on your streaming player without hurting viewer experience?
We will design the experiment setup and QoE metrics for your service and send a plan with sample sizes. 250+ video products shipped since 2005.
