Effect Detection Calculator
Calculate the minimum detectable effect for your experiment setup
Total: 20,000
Control: 10,000
Treatment: 10,000
MDE comparison: 10,000 vs 10,000
Minimum Detectable Effects
Current Configuration
Detectable lift
+0.79%
Absolute lift
+0.7924
Baseline
100.0000
Target needed to detect
100.7924
Total population: 20,000
Control group: 10,000
Treatment group: 10,000
Comparison used for MDE: 10,000 vs 10,000
Comparisons: 1
Adjusted α per comparison: 0.0500
Allocation: Equal split across 2 groups
Half Sample Size
Detectable lift
+1.12%
Absolute lift
+1.1206
Baseline
100.0000
Target needed to detect
101.1206
Total population: 10,000
Control group: 5,000
Treatment group: 5,000
Comparison used for MDE: 5,000 vs 5,000
Comparisons: 1
Adjusted α per comparison: 0.0500
Allocation: Equal split across 2 groups
Double Sample Size
Detectable lift
+0.56%
Absolute lift
+0.5603
Baseline
100.0000
Target needed to detect
100.5603
Total population: 40,000
Control group: 20,000
Treatment group: 20,000
Comparison used for MDE: 20,000 vs 20,000
Comparisons: 1
Adjusted α per comparison: 0.0500
Allocation: Equal split across 2 groups
Configuration
- Metric: continuous
- Test: two-sided
- α = 0.050, Power = 0.80
- Flights: 2
- Sample input mode: Per group
- Allocation mode: equal
- Current total population: 20,000
- Current control group: 10,000
- Current treatment group size: 10,000
How the Calculation Works
Minimum Detectable Effect (MDE)
The MDE is the smallest difference between variants that your test is powered to detect at the chosen significance level and power.
For equal-sized two-group tests, this simplifies to the familiar form with √(2 / n). For unequal allocation, the calculator uses the group sizes from the selected allocation.
Unequal allocation
When unequal allocation is selected, the control gets the chosen share of traffic and the remaining traffic is split equally across the treatment groups.
The MDE gets worse when the comparison groups are more imbalanced because the standard error depends on both group sizes.
Multiple flights and FWER
If you test multiple flights, you may want to control the family-wise error rate using Bonferroni correction.
This makes each comparison harder to declare significant, which increases the MDE. This calculator keeps the sample sizes fixed and adjusts the significance threshold.
How to read the result
- Detectable lift: the smallest relative uplift the test is powered to detect.
- Absolute lift: the corresponding absolute change in the metric.
- Target needed to detect: the approximate treatment value needed to detect that lift.
- Comparison used for MDE: the group sizes used in the variance calculation.
Sample size impact
- Doubling sample size reduces MDE by about 29%
- Halving sample size increases MDE by about 41%
- To detect half the effect, you need about 4x the sample size
Ready to analyse the results of your experiment?
Open Test Results Calculator →