Statistical Glossary

The significance level of a test, representing the probability of making a Type I error (false positive). Commonly set at 0.05, meaning a 5% chance of rejecting a true null hypothesis.

The probability of making a Type II error (false negative), which is failing to reject a false null hypothesis. Statistical power is calculated as 1 - β.

The probability of correctly rejecting a false null hypothesis, calculated as 1 - β. Higher power means a better chance of detecting a true effect when it exists.

The probability of obtaining test results at least as extreme as the observed results, assuming the null hypothesis is true. A smaller p-value provides stronger evidence against the null hypothesis.

A quantitative measure of the magnitude of a phenomenon. Common measures include Cohen's d, which represents the standardized difference between two means.

The default assumption that there is no effect or no difference between groups. Statistical tests aim to determine whether to reject this hypothesis.

The hypothesis that contradicts the null hypothesis, suggesting that there is an effect or a difference between groups.

A false positive error that occurs when the null hypothesis is incorrectly rejected when it is actually true. The probability of this error is equal to the significance level (α).

A false negative error that occurs when the null hypothesis is incorrectly accepted when it is actually false. The probability of this error is denoted as β.

The practice of repeatedly checking test results before the planned end of an experiment. This inflates the false positive rate and can lead to incorrect conclusions.

The probability of making at least one Type I error when conducting multiple hypothesis tests. Methods like Bonferroni correction control for this inflated error rate.

A conservative method to control the family-wise error rate by dividing the significance level by the number of tests performed (α/n).

Controlled-experiment Using Pre-Experiment Data. A variance reduction technique that uses pre-experiment data to reduce noise and improve the sensitivity of A/B tests.

The number of observations or data points in a study. Larger sample sizes generally provide more precise estimates and greater statistical power.

A range of values that likely contains the true population parameter with a specified level of confidence (typically 95%). Wider intervals indicate more uncertainty.

A hypothesis test where the alternative hypothesis specifies a direction (greater than or less than). All of the significance level is placed in one tail of the distribution.

A hypothesis test where the alternative hypothesis does not specify a direction (simply not equal to). The significance level is split between both tails of the distribution.

The threshold value that the test statistic must exceed to reject the null hypothesis. It is determined by the significance level and the sampling distribution.

A measure of the variability of a sample statistic, typically the standard deviation of the sampling distribution. It decreases as sample size increases.

The smallest effect size that can be detected with a given level of statistical power, sample size, and significance level.

Pre-specified thresholds for key metrics that, if crossed, trigger an early stop of the experiment to prevent significant harm.

Statistical methods that allow for valid interim analyses during an experiment without inflating the Type I error rate, unlike traditional peeking.

Allocating different proportions of subjects to treatment and control groups (e.g., 80/20 split). This affects statistical power and sample size requirements.

A standardized measure of effect size calculated as the difference between two means divided by the pooled standard deviation. Values of 0.2, 0.5, and 0.8 are considered small, medium, and large effects.