Family-Wise Error Rate Simulator
Explore how multiple comparison corrections (Bonferroni, Holm, Tukey, Dunnett) affect false positive and false negative error rates when testing multiple hypotheses simultaneously.
The Multiple Comparisons Problem
Why This Matters
When testing multiple flights (variants) simultaneously, each individual test might use α = 0.05, giving a 5% chance of a false positive. But when you run multiple tests, the probability that at least one produces a false positive increases dramatically.
Understanding Pairwise Comparisons
With 3 flights (including control), pairwise comparisons test every possible combination:
Example with 3 flights:
- A vs B
- A vs C
- B vs C
Total comparisons: n × (n-1) / 2 = 3 × 2 / 2 = 3
This grows quickly: 3 flights = 3 comparisons, 4 flights = 6 comparisons, 5 flights = 10 comparisons, 6 flights = 15 comparisons. The number of comparisons grows quadratically with the number of flights!
The Math Behind It
With 3 independent comparisons, the probability of making at least one false positive (Family-Wise Error Rate) without correction is:
FWER = 1 - (1 - α)ⁿ = 1 - (1 - 0.05)^3 = 14.3%
That's 2.9x higher than your nominal α = 0.05 threshold!
Real-World Scenario
In A/B/n testing, you might test multiple variations of a product feature. Without proper correction:
- You'll falsely declare "winners" far too often in A/A tests
- Your published results will be misleading and unreliable
- Downstream teams will waste resources implementing changes that don't actually work
FWER correction methods adjust your significance thresholds to maintain the overall error rate at your desired level, preventing false discoveries while balancing the trade-off with statistical power.
Error Rates by Number of Comparisons
This chart shows how Type I (false positive) and Type II (false negative) error rates change as the number of comparisons increases for different correction methods.
Type I Error Rate (False Positives)
Without correction, Type I error increases dramatically with more comparisons. ANOVA prior provides additional protection.
Type II Error Rate (False Negatives)
Bonferroni increases Type II error (reduces power) as comparisons grow. ANOVA prior has a small additional cost.
Key Insights:
- • Type I Error (Left): Risk of false positive. No correction escalates rapidly with comparisons.
- • Type II Error (Right): Risk of missing true effects. Increases with stricter corrections.
- • ANOVA Prior: Reduces Type I error with minimal impact on Type II error.
- • Trade-off: As you add more comparisons, you must choose between controlling false positives vs maintaining power.
Simulation Parameters
Choose test scenario
Number of groups to compare (2-10)
Type of comparisons to perform
Average metric value
Measure of data spread
Users per flight group
Type I error rate threshold
Real effect size on one flight only
Iterations for stable estimates
Outliers can inflate variance and reduce detecting effects
Go to Winsorizing to learn how to handle outliers →