Test Results Calculator

Statistical Testing Overview

This calculator performs hypothesis testing to determine if your observed difference between variants is statistically significant. The process:

1. Calculate the observed difference between variants
2. Compute a test statistic (t-test for continuous, z-test for binary)
3. Calculate the p-value: probability of observing this result if there's no real difference
4. Compare p-value to significance level (α) to determine significance

Understanding Your Results

• Difference: The observed change between control and treatment. Positive means treatment is better; negative means it's worse.
• P-value: The probability of seeing this difference (or larger) if there's actually no real effect. Smaller p-values provide stronger evidence of a real effect.
• Confidence Interval: A range of plausible values for the true effect. Narrower intervals indicate more precise estimates. If the interval doesn't cross zero, the effect is significant.
• Significant: p-value is less than α, meaning we reject the null hypothesis and conclude there's a real difference.

Continuous Metrics (T-test)

For metrics like revenue, time on page, or other numerical measures. We compare means and account for variability:

Where SE (standard error) is calculated from your sample sizes and standard deviation.

Binary Metrics (Z-test)

For metrics like click rate, conversion rate, or other proportions. We compare success proportions:

Where SE accounts for both proportions and sample sizes.

Multiple Comparison Correction

When testing multiple variants, we apply Bonferroni correction to reduce false positives:

This makes it stricter to achieve significance. For example, with 3 pairwise comparisons and α=0.05, the adjusted threshold is 0.05/3 ≈ 0.0167.

Important Notes

• A non-significant result doesn't mean there's no effect; it may mean your sample size was too small
• Confidence intervals are often more informative than p-values alone
• Always check practical significance: Is a 0.1% improvement worth implementing?
• Be cautious of p-hacking or multiple testing without proper corrections