Guardrails Simulator
Simulate A/B tests with guardrails to detect when metrics fall below acceptable thresholds and stop tests early to prevent damage.
Guardrails Simulation Parameters
Choose whether there's a real difference
Average value for control group
Variability in the data
How long the test runs
Number of observations in each group
How many test runs to simulate
Two-sided CI: 99% is more conservative than 95%
5.01%
Smallest effect you can reliably detect with current sample size and test duration
Will check 28 times over 28 days
Results
Run a simulation to see results
Understanding Guardrails
Manual Guardrails
Manual guardrails use fixed thresholds (e.g., stop if metric drops below -5%) to protect against severe negative impacts. While simple and intuitive, they have a significant drawback: high variance early in the test can easily trigger false alarms.
With limited early data, natural statistical fluctuations can cause metrics to temporarily breach your manual threshold even when no real harm is occurring. This leads to prematurely stopping potentially neutral or positive experiments.
Statistical Guardrails
Statistical guardrails account for variance by requiring statistical significance of the negative effect before triggering. They use confidence intervals (e.g., 99% confidence) to determine if the observed drop is likely real or just noise.
This approach is far more robust: it's extremely unlikely for a metric to show a statistically significant -99% effect and then recover to become significantly positive. Statistical guardrails adapt to sample size - requiring stronger evidence when data is limited - making them ideal for continuous monitoring throughout your test.
Recommendation: Use statistical guardrails for ongoing monitoring, as they balance safety with statistical rigor. Reserve manual guardrails only for absolute red-line thresholds where any breach - regardless of statistical significance - requires immediate action.
Peeking frequency: Weekly guardrail checks reduce false early stops compared to daily monitoring. Daily peeks inflate volatility at low sample sizes, increasing noise-driven guardrail breaches without improving detection of genuinely harmful effects.
Imbalanced exposure can distort guardrail metrics in A/B tests
Go to Imbalanced Experiments →