Understanding Null Hypothesis Significance Testing
An interactive visualization of statistical power, Type I and Type II errors, and effect sizes.
This visualization shows the sampling distributions under the null hypothesis (H₀) and alternative hypothesis (H₁) for a one-sample Z-test. Adjust the parameters below to see how they affect statistical power, Type I error (α), and Type II error (β).
- Type I error (α): Rejecting H₀ when it's true (false positive)
- Type II error (β): Failing to reject H₀ when it's false (false negative)
- Power (1-β): Probability of correctly rejecting a false H₀
Sampling Distributions
Understanding the Visualization
The Distributions
The blue curve (H₀) represents the sampling distribution under the null hypothesis, centered at μ₀ = 0. The red curve (H₁) represents the sampling distribution under the alternative hypothesis. Solid vertical lines mark each mean.
Key insight: Changing alpha (α) moves the critical value, while adjusting power changes the effect size required.
One-Sided vs Two-Sided Tests
A one-sided test only rejects H₀ in one tail. A two-sided test splits alpha across both tails, requiring slightly larger effect sizes for the same power.
Type I Error (α)
The red shaded area under H₀ represents Type I error (false positive). Lower α moves the critical value outward, making the test more conservative.
Type II Error (β) and Power
The orange shaded area under H₁ (left of the critical value) is Type II error (false negative).
The green shaded area represents power (1-β). Higher desired power increases the required effect size or sample size.
How the Parameters Interact
Power (1-β): Higher power requires larger effect sizes or larger samples.
Sample Size (n): Larger samples reduce standard error, allowing smaller effects to be detected at the same power.
Alpha (α): Lower α moves critical values outward, requiring larger effects to achieve the same power.
Test Type: Two-sided tests split alpha, requiring slightly larger effects than one-sided tests for the same power.
See how frequent peeking can inflate false positives in A/B tests
Explore False Positives from Peeking →