Test Duration Calculator

Sample Size Calculation

First, we calculate the required sample size per group using the power analysis formula:

Where z_α and z_β are the critical values for your significance level and power. Variance depends on metric type (continuous uses 2, binary uses 2p(1-p)). The effect size is calculated from your MDE and metric parameters.

Duration Formula

Test duration is determined by how long it takes to collect the required sample size:

The total sample size is calculated from the per-group requirement multiplied by the number of variants (flights) in your test. Then we divide by your daily traffic per flight to estimate how many days the test must run.

Traffic Distribution Options

• Across All Flights: Daily traffic value is the total split evenly among all variants. For example, 10,000 daily users with 3 flights means ~3,333 per flight per day.
• Per Flight Only: Daily traffic value applies to each flight individually. Use this when you have dedicated traffic pools for each variant. 10,000 per flight with 3 flights = 30,000 total daily users.

Key Factors Affecting Duration

• Smaller MDE: Requires larger sample size → longer test duration
• Lower Traffic: Fewer daily conversions → longer test duration
• More Variants: More comparisons needed → larger sample size → longer duration
• Higher Power: More stringent requirements → larger sample size → longer duration

Multiple Comparison Impact

With multiple flights, Bonferroni correction requires a stricter significance threshold. This increases the sample size needed by a factor roughly proportional to the number of comparisons.

Example: Testing 3 variants with pairwise comparisons (3 comparisons total) roughly triples your duration compared to a 2-variant test.

Practical Considerations

• Longer tests have lower risk of being affected by external events or temporal patterns
• Shorter tests reduce time-to-insight but have less data to work with
• Consider business constraints: Can you wait 60 days? Can you spare the traffic? Are seasonal factors a concern?
• High traffic or large expected effects can dramatically reduce test duration