Metric Normalisation
When running experiments across regions with different price levels, imbalanced user allocation can create spurious effects. Normalisation reduces variance by putting all regions on the same scale, leading to tighter confidence intervals.
The Problem
Consider an experiment measuring order value across regions with different price levels. If randomisation happens to assign more users from a high-priced region to treatment, that group will show a higher average even if the treatment has no real effect.
The raw data has high variance because values from different regions are on completely different scales. This inflates confidence intervals and makes it harder to detect true effects.
Moreover, when treatment effects apply proportionally, a +$5 absolute uplift might represent a meaningful 20% gain in a low-price region but only a negligible 2.5% increase in a high-price region. In raw aggregation, the signal gets dominated by the high-volume, high-price market where the percentage effect appears smallest, potentially masking significant relative improvements in smaller markets.
The Solution
Ideally, you would run separate experiments within each region and randomise within region to ensure balanced allocation. However, this isn't always practical - you may lack sufficient sample size per region, or operational constraints may prevent stratified randomisation.
When you must pool across regions, Z-score normalisation transforms each region's data to have mean 0 and standard deviation 1. This:
- Removes the scale differences between regions
- Dramatically reduces overall variance in the pooled data
- Produces tighter confidence intervals for the same sample size
- Prevents regional imbalances from creating false positives
Normalisation Methods
| Method | Formula | When to Use | Notes |
|---|---|---|---|
| Z-score (Standardisation) | x' = (x - mean) / std | Aggregation across regions | Widely used; centers and scales distribution |
| Min-Max Scaling | x' = (x - min) / (max - min) | Fixed, bounded range | Fast and intuitive; sensitive to outliers |
| Divide by Max | x' = x / max | Quick comparison, values greater than or equal to 0 | Simpler; depends on max stability |
| Percent-of-Mean | x' = x / mean | Relative effect matters more | More interpretable in business settings |
This simulator uses Z-score standardisation, highlighted above.
Simulation Parameters
Each region has a different baseline distribution due to local price levels and market conditions
The same relative effect is applied to all regions
Regional Imbalance Impact
Shows the spurious lift created when treatment has more users from one region vs control (with no true treatment effect)
If treatment randomly gets 10% more users from North America, it appears to win by ~15.6% even with zero true effect.
With baseline means of $25 vs $200, small allocation imbalances create large spurious effects that can mask or mimic real treatment impact.
When to Use Normalisation
Good Use Cases
- Multi-region experiments with different price levels
- Cross-market analysis with currency differences
- Geographic regions with varying cost of living
- Markets at different maturity stages
Caution Needed
- When absolute revenue impact matters more than relative
- If regions have very different sample sizes
- When region-level effects need separate interpretation
- If normalisation parameters can be influenced by treatment
Understand guardrails to see how monitoring choices impact experiment safety
Explore Guardrails →