Sample Size Calculator

The Core Formula

Sample size calculations use the standard statistical formula that balances Type I and Type II error rates:

Where n is the sample size per group, z_α and z_β are critical values from the normal distribution, and variance depends on your metric type.

Key Components

• Significance Level (α): The probability of incorrectly rejecting a true null hypothesis (Type I error). Typically 0.05 means a 5% chance of a false positive.
• Power (1 - β): The probability of correctly detecting a true effect. Higher power (e.g., 0.80 or 80%) requires larger sample sizes but gives you more confidence in detecting real effects.
• Effect Size: The difference you want to detect between variants. Smaller effects require larger samples. We express this as Cohen's d for continuous metrics or absolute/relative uplift for binary metrics.
• Variance: For continuous metrics, this is 2 × (standard deviation)². For binary metrics, this is 2 × p × (1-p), where p is the baseline success rate.

Multiple Comparison Correction

When testing multiple flights (variants), we apply Bonferroni correction to control for Family-Wise Error Rate (FWER):

This more conservative threshold increases the required sample size. For example, with 3 flights and pairwise comparisons (3 comparisons total), the adjusted α would be 0.05 / 3 ≈ 0.0167 instead of 0.05.

What the Results Mean

• Per Flight Group: The minimum sample size needed for each variant to achieve your desired power and significance level.
• Total Samples: The complete sample size across all flights. Multiply per-group size by the number of flights.
• Example: 5,000 samples per flight × 3 flights = 15,000 total samples needed for your experiment.

Practical Implications

• Larger effects require smaller samples
• Higher power requirements increase sample size
• Multiple comparisons significantly inflate sample size needs
• Binary metrics (like click rates) typically need more samples than continuous metrics (like revenue) for the same effect size