The framework
Resource-first A/B testing
A go-no-go framework for teams without Google-scale traffic. Fix the duration first, work out what the test can actually detect, then decide if it’s worth the people it takes. Play with the numbers as you read.
01
Why not test everything
The loudest voices in experimentation are Google, Amazon, Booking.com, Microsoft, Netflix and Airbnb. They had money, infrastructure, statisticians and millions of users before they ran their first online test. Their rules are great rules, just not for every company. Tap a card to see why.
Running experiments that are not sensitive enough may lead to fatigue and demotivation among members of the team involved in experimentation.
02
Resource-first, not experiment-first
In the usual flow the experiment comes first and the resources and duration follow, whatever they turn out to be. With limited users, people and time, turn it around.
- 1Someone has an idea
- 2Estimate the resources to build it
- 3Design and build the change
- 4Set MDE, α, β and the test type
- 5Calculate the sample size and duration
- 6Launch, however long it takes
- 1Fix the maximum duration for this kind of test
- 2Count the users you can get in that time
- 3Calculate the effect the test can detect (its sensitivity)
- 4Weigh that against the resources it needs
- 5Go, go as a guardrail, or ship without a test
- 6Only now design and build
Ask “can we detect anything?” before anyone designs or builds a thing.
03
Set your limits
Every kind of test gets a maximum duration. Cookie-based tests on the website go stale quickly; tests on a stable user ID can run longer, and revenue metrics, with their higher variance, get the longest. These are the limits that work for us. Pick the kind of test you’re planning.
04
Measure sensitivity
Sensitivity is the smallest uplift the test can reliably detect (its minimum detectable effect, MDE) with the users you can get in the time you allowed. Two-sided test, α = 0.05, power = 0.80.
- Users per variant
- 29,250
- Your duration limit
- 45 days
Days needed to reach each band
05
Name the source of change
Not every test starts from the same place. Some changes would ship anyway; others exist only because someone wants to try a risky idea. The difference decides which resources count.
How much will it take to design, build and set up the test?
06
Make the call
Go, carefully
Launch it, but watch the cost.
6.9% is good sensitivity. Be careful about how many resources you channel into this one.
- Keep design and development scope tight
- Run it for the full 45 days and don’t stop early
Thresholds and limits are mine; adjust them to your business. The logic is what matters: sensitivity against resources, with the source of change deciding which resources count.
07
Report a null result honestly
“There is no difference between the variants” is the sentence that kills experimentation culture. Absence of evidence is not evidence of absence. Say what the test could and couldn’t see instead. Here’s the sentence for the test you set up above:
There is no significant difference between the variants.
We failed to detect a difference between the variants after 45 days. It may still exist: we would have detected an uplift of 10.6% or more with 99% probability, and one of 6.9% or more with 80% probability.
08
What tends to be sensitive enough
Over time a pattern shows up in which tests clear the bar. If yours lands in the red, these are the levers to pull before giving up on it.
A change across the whole site or a group of pages collects users far faster than one landing page.
Everyone sees the first screen. Only a few scroll to the bottom, so changes down there starve for sample.
Every new user passes through it, so the number of users is maxed out at the start of the journey.
Testing on new users avoids having to estimate primacy and novelty effects in people used to the old version.
The full argument, with the sources behind it, is in the original post.
Read “Not A/B Testing Everything is Fine” →