Metric Variability & Detectability
The variability of your metric directly determines how long your experiment needs to run. Both proportion metrics (e.g. conversion rate) and continuous metrics (e.g. average revenue, scores) have different variability profiles, and choosing the right metric type can significantly reduce experiment runtime.
Key Insight
If your continuous metric has a Coefficient of Variation (CoV) above 0.5 (i.e. your standard deviation is above 50% of the value of the mean), consider whether you can reframe it as a proportion metric. This results in a substantially lower MDE and faster experiment runtime - at the cost of a slight change in metric definition.
MDE by Proportion Baseline and Continuous Metric CoV
Both axes use the same n = 1,000 per group. Proportion MDE is expressed as an absolute percentage-point uplift; continuous MDE is expressed as a relative % increase in the metric value. Hover for runtime comparison.
The proportion curve peaks at 50 % baseline and falls toward 0 % and 100 %. The continuous curve grows linearly with CoV. Where the green line sits above the blue, switching to a proportion metric means a lower MDE and a faster experiment.
Switching Metric Type: The Trade-off
Converting a continuous metric to a proportion often means slightly redefining what you measure. The benefit is a large reduction in variability and experiment runtime.
Example
Continuous: Average application quality score per user(e.g. mean dotmatch score: high CoV, noisy)
Proportion equivalent: % of users who submitted at least one high-fit application(binary: did they hit the threshold? Much lower variance)
What changes
- Substantially lower MDE, so it is detectable with fewer samples
- Faster experiment runtime
- Metric definition shifts slightly: you're now measuring a threshold, not an average
- You lose granularity: a user scoring 0.95 vs 0.61 looks the same
Whether this trade-off is acceptable depends on your hypothesis. If the direction of the effect matters more than the magnitude, a proportion metric is often the right call.
Understanding Coefficient of Variation (CoV)
For continuous metrics, the Coefficient of Variation (CoV) captures how noisy the metric is relative to its mean:
- CoV < 0.3: extremely low variability, fast experiments, small MDE achievable
- CoV ≈ 0.5: similar variability to a 50% proportion metric. This is the crossover point
- CoV > 0.5: high variability, slow experiments. Consider a proportion equivalent
- CoV > 1: extremely high variability, quite standard in user metrics. Use proportion if business accepts the metric redefinition
Revenue and score-based metrics often have CoV between 1 and 3 due to extreme outliers, making them inherently harder to detect changes in, even after winsorizing.
CUPED can reduce variance for continuous metrics, often a faster alternative to switching metric type.
Go to CUPED Variance Reduction →Winsorizing can also reduce the impact of outliers on high-CoV continuous metrics.
Go to Winsorizing →