A/B Testing Go-no-Go Part 2: Impact

We should always ask ourselves what portion of our users will be exposed to the experiment before making a go-no-go decision. If our experiment will be shown only to a small subset of users (say, 10 percent of all new users we get weekly), even a decidedly big uplift of 15–25 percent of the metric will be diluted in the entire user population.

So, if there’s no reason to suspect that only a small subset of users will be impacted by the change, we should try to extend experiment to as many users as possible, given that there’s an expectation that we can influence those users with our change.

If there’s no way to formulate the experiment in such a way that it impacts as many users as possible, maybe we should lay it off for the time being, if we want our effort to bring a noticeable change.

Example

Let’s imagine we are a part of a relatively new startup with 10,000 weekly new users. 85% of those (8,500 users) complete their account setup in less than 3 days and those who don’t (1,500 users), receive an automated email on 3rd day encouraging them to finish the account setup. Around 30% of those users respond positively to the call-to-action and finish the setup. That’s the basic figures we have:

New weekly users: 10 000
Users who complete the setup in less than 3 days: 8500 (85%)
Users who don't complete the setup in 3 days and receive the email: 1500 (15%)
Users who receive the email and complete the setup in 7 days: 450 (30%)
10th day onboarding rate: 8500 + 450 / 10000 = 0.895

One of the team members suggests we make a small change to the automated email that is supposed to improve the onboarding rate of those who received it by 20%, so we figure out the required sample size and start the experiment.

Sample size for the email experiment

Baseline conversion rate

30%

Minimum detectable effect

20% relative

Significance level α

5%

Statistical power 1−β

80%

30%

24% – 36%

Conversion rates inside the band can't be told apart from the baseline.

929visitors per variation

It would take us around two weeks to detect the change and when the time is up, that’s what we find:

Email experiment result

SignificantA: 1,000 visitors, 300 conversionsB: 1,000 visitors, 360 conversions

Conversion rate A

30.00%

Conversion rate B

36.00%

Relative uplift

20.00%

z-score

2.8591

p-value (one-sided)

0.0021

Looks like our team member deserves a bonus and things are going to get better from now on, now that our overall onboarding rate improved by… less than 1 percent?

New weekly users: 10 000
Users who complete the setup in less than 3 days: 8500 (85%)
Users who don't complete the setup in 3 days and receive the email: 1500 (15%)
Users who receive the email and complete the setup in 7 days: 540 (36%)
10th day onboarding rate: 8500 + 540 / 10000 = 0.904

Onboarding before we implemented the change: 89.5%
Onboarding after we implemented the change: 90.4%

It sure doesn’t look like much. But let’s imagine the exact same startup in a parallel universe with somewhat different base metrics:

New weekly users: 10000
Users who complete the setup in less than 3 days: 3000 (20%)
Users who don't complete the setup in 3 days and receive the email: 8000 (80%)
Users who receive the email and complete the setup in 7 days: 2400 (30%)
10th day onboarding rate: 3000 + 2400 / 10000 = 0.54

As we can see, this one fares less well with the onboarding rate stuck around boring 54%, whereas only 20% of the users finish the setup by the 3rd day. But our excellent growth team member suggests to introduce the same change to the email with the same expected uplift of 20%. We run the experiment, see the desired uplift and make the change to the email for all users. After a while we check the metrics and that’s what we find:

New weekly users: 10 000
Users who complete the setup in less than 3 days: 3000 (20%)
Users who don't complete the setup in 3 days and receive the email: 8000 (80%)
Users who receive the email and complete the setup in 7 days: 2880 (36%)
10th day onboarding rate: 3000 + 2880 / 10000 = 0.588

This time around we got around 5% improvement in the general onboarding rate after releasing the change to the email, which is much better than 1% we saw in the first example (from 54% to 58%).

This makes us think about things that actually may be quite obvious:

• the smaller the user group that’s going to be the part of the experiment, the smaller impact it’s going to have on the overall picture.
• if the resources are tight and the possibilities are in abundance (when we have a relatively new startup with tons of things that can be improved), it’s better to focus on experiments that are going to have bigger impact.

You can still opt in for the lesser-impact experiments in case:

• you have a nice polished app where tons of things have been already improved and it’s time to move onto delicate and small improvements.
• you have tons of resources allocated to your growth team and it’s easy for you to run 50 small-impact experiments at a time that are going to have a big overall impact.
• the knowledge of the cause and effect you gain for your app, what things move the users and what don’t, is more important for you than the uplift itself. After all, the more knowledge you gain, the more successful all your subsequent experiments are going to be thanks to the meta analysis of the existing body of experiment results.