Skip to content
AdvancedAB
Lessons

What testing taught us

Real lessons from real experiments. Each one is a reason not to trust your gut alone.

Same prompt, three places, three different results

Same sign-up prompt on three pagesVisitors move from the homepage to search results to a product page. The same create-an-account prompt was ignored on search results, raised sign-ups on the product page, and unexpectedly also worked on the homepage, where returning customers who already trusted the brand signed up.HomepageSearch resultsProduct pageSign-ups from the promptRegulars signed upalready trust the brandIgnoredbusy searchingSign-ups roseclose to their goal

We tested asking visitors to create an account at three points: the homepage, the search results page, and the product page.

On search results, almost everyone ignored it. They were busy looking for the thing they came for, and anything in the way was noise. On the product page, sign-ups rose: visitors were close to their goal and saw a reason to commit. The homepage also performed well, which nobody predicted. When we dug in, most of those sign-ups were returning customers who already trusted the brand. For them, an account was an easy yes.

That’s the point. Even experienced teams can’t reliably predict how people will behave. The only way to know is to test.

Get a free testing plan →

The change that looked like a disaster

Visitors followed for four weeks after sign-in became mandatoryAfter sign-in was made mandatory, the number of visitors leaving immediately jumped. Over the following weeks many of them came back and signed in, and completed purchases stayed flat the whole time.Sign-in made mandatoryLeft immediatelysharp jumpCame back and signed inmany returnedCompleted purchasesno changeBeforeWeek 1Week 2Week 3Week 4

Making sign-in mandatory caused a sharp jump in visitors leaving immediately. On that number alone, most teams would have scrapped it.

But we followed visitors over the following weeks. Many of those who left came back and signed in. The ones who never returned turned out to be low-value visitors, and completed purchases weren’t affected at all.

The lesson: judge a change on the outcome that matters to the business, over a long enough window, not on the first scary metric.

Get a free testing plan →

When a big percentage is a small deal

Support emails up 25%, orders up 4%: the site is about $3,500 better off a monthAfter switching to a single-page checkout, customer support emails rose 25% and orders rose 4%. On a site turning over $100,000 a month, the extra orders are worth about $4,000 and the extra emails cost about $500 in staff time, so the business is about $3,500 better off each month.Inside the testpercentage change+25%+4%Support emailsOrdersAcross the businessdollars per month, $100k site−$500+$4,000+$3,500SupportOrdersNet

Sometimes a change helps one number and hurts another, and you have to weigh one against the other. Inside an experiment, those shifts can look dramatic.

Say you replace a three-step checkout with a single page. Orders rise 4%, but customer support emails jump 25%, mostly from people unsure whether their order went through. The 25% sounds alarming, and top-line revenue can’t tell you whether it matters, because support costs staff time, not sales.

On a site turning over $100,000 a month from 1,000 orders, 4% more orders is about $4,000 a month. If you get 300 support emails a month, 75 more at about 10 minutes each is roughly $500 of staff time. That’s about $3,500 better off every month, so the change is worth rolling out. And the emails point to the next fix: a clearer confirmation screen keeps the lift and wins back most of that cost.

We weigh every trade-off in real business terms: what is it worth across the whole business? We also check that both shifts are real, not just noise. Often the scary number turns out to be minor. Sometimes it’s the other way round.

Get a free testing plan →

What would your visitors choose?

Get your free testing plan. You’ll see exactly what we’d test first, and whether it’s worth it for your site.

Get a free testing plan