Skip to content
AdvancedAB

CUPED Variance Reduction

CUPED reduces variance in A/B tests using pre-experiment data. Its real-world impact depends on both correlation strength and how many users actually have historical data.

Parameters

R² = 0.70

How predictive pre-experiment data is for in-experiment behavior.

80%

Fraction of users eligible for CUPED adjustment.

Effective Variance Reduction

56.0%

R² × Coverage

Variance Reduction

CUPED narrows the distribution by removing predictable variance.

Minimum Detectable Effect

Lower MDE means you can detect smaller effects with the same sample size.

Standard A/B4.38%
With CUPED2.91%
33.7% smaller MDE

How Coverage Dampens CUPED

Even with strong correlation, CUPED’s real-world impact shrinks if historical data is missing for many users.

Raw R²0.70Effective R²0.56

How CUPED Works

1. Collect Pre-Experiment Data

Collect the same metric from a historical period before running your A/B test.

2. Calculate Correlation

Measure how well the pre-experiment metric predicts the experiment metric (R²).

3. Adjust for Covariates

CUPED adjusts each user's metric by subtracting their expected value based on history.

4. Reduced Variance

This reduces predictable variance by correcting individual metrics: users below their historical average are adjusted upward, and users above their average are adjusted downward, wherever pre-experiment data is available.

CUPED Formula

CUPED adjusts each user's metric to reduce predictable variance while preserving the overall mean:

YiCUPED = Yi - θ × (Xi - X̄)

  • Yi = observed metric for user i
  • Xi = pre-experiment metric for user i
  • X̄ = mean of the pre-experiment metric across all users
  • θ = Cov(X,Y) / Var(X) = coefficient capturing how much pre-experiment behavior predicts the metric

This adjustment reduces variance while keeping the mean intact, allowing the test results to remain interpretable in real business terms.

Key Insight

Std Dev: 250 → 166 (33.7% reduction)
Sample Savings: 56.0% fewer samples needed
MDE Improvement: 4.38% → 2.91%

Why Use CUPED?

CUPED's effectiveness depends on how well your pre-experiment data correlates with in-experiment behavior. When R² is high (strong correlation) and coverage is high, CUPED delivers substantial benefits:

  • Faster Test Results: Reduced variance allows faster detection of significant effects.
  • Smaller Sample Requirements: Achieve the same power with fewer users.
  • Detect Smaller Effects: Variance reduction makes tests more sensitive.
  • Fair Comparison Across Users: Reduces bias when some flights are dominated by high- or low-value users, making results more reliable.
  • Maintain Real-World Meaning: Adjusts for variance while keeping the true mean intact, ensuring business impact and uplift remain understandable.
  • Better ROI: Run more experiments with the same traffic.

Multiple flights increase the risk of false positives

Learn to adjust for multiple flights →