Peeking Simulator
Simulate A/B tests with multiple peeks to understand false positive rates when checking results before test completion.
Peeking Simulation Parameters
Choose whether there's a real difference
Average value for control group
Variability in the data
How long the test runs
Number of observations in each group
How many test runs to simulate
95% means p-value threshold of 0.05
5.01%
Smallest effect you can reliably detect with current sample size and test duration
Will check 28 times over 28 days
Results
Run a simulation to see results
Understanding the Visualisation
The visualization demonstrates how frequent checking (peeking) inflates the false positive rate beyond your chosen significance level. Each time you check your test results, you're essentially running a new hypothesis test, increasing the probability of finding a significant result by chance alone.
In the timeline chart, you can see how p-values fluctuate naturally throughout the test duration. When peeking frequently, you're more likely to catch a moment when the p-value dips below your significance threshold (e.g., 0.05) even when there's no true effect (A/A test).
Key insight: The more often you check your results, the higher your actual false positive rate becomes. For example, if you check daily over 28 days with α = 0.05, your true false positive rate can exceed 20-30%, far above the intended 5%. This is why it's critical to either:
- Wait until your predetermined sample size is reached before checking
- Use sequential testing methods (e.g., always valid p-values, spending functions)
- Apply multiple testing corrections (e.g., Bonferroni) to your significance threshold
Learn how peeking and frequent checks relate to experimental guardrails.
Learn About Setting Statistical Guardrails →