Let’s imagine we have an online service which utilizes a subscription-based business model. We offer email marketing features and chatbot builder functionality, both requiring a separate subscription.
- When users register, they are presented with a simple form asking them what features they are going to use — chatbots or emails, so we can easily distinguish users and put them into groups.
- For the sake of simplicity, let’s assume that both user groups have the same conversion rate into paying customers which equals 15%.
- Only 20% of new weekly users register in our service for chatbot functionality and the rest of them — 80%, register for email marketing features.
Someone on the team comes up with an idea that by making a small improvement in the chatbot builder interface, we can increase the conversion rate of chatbot users by as much as 13%, bringing the conversion rate up to 16.95%.
- The base conversion rate of chatbot users is 15%
- Expected conversion rate after suggested improvement: 16.95% (13% uplift from the base conversion rate)
We then calculate the sample size required for detecting the desired change and it comes down to:
How many users do we need?
- Baseline conversion rate
- 15%
- Minimum detectable effect
- 13% relative
- Significance level α
- 5%
- Statistical power 1−β
- 80%
We need 5,345 new users or more participating in each variation of the experiment to detect the desired change, 10,690 in total.
We get 26,725 new users every week, 5,345 of which (20%) register for chatbot builder functionality. Since we need at least 10,690 users to see the desired uplift, we will have to run the experiment for two weeks or more.
Sometimes it’s tempting to run an experiment on all users, regardless of whether they had a chance to be exposed to it, since this kind of experiment setup can be easier to implement.
But good experiment design usually requires to implement some sort of triggering mechanism that puts users into an experiment just at the right time when they actually have a chance to be exposed to the change.
In our case those are users who:
- Showed interest in the chatbot builder features and saw the improvements in the interface (Treatment Group)
- Showed interest in the chatbot builder, saw the chatbot builder interface but were bucketed into the control group, so were exposed to the original version.
There’s a reason for that and many refer to it as “eliminating noise”. What’s the noise exactly in the experiment context? In our particular case, those are all the users who didn’t have a chance to be exposed to the changes since they didn’t have intention to use the chatbot functionality.
Why do we have to eliminate them from the experiment? Because it decreases the experiment sensitivity, making it harder to detect the difference in the variations, thus increasing the chance of false negative (Type II Error), especially given the same sample sizes.
To better demonstrate an example of how noise can potentially fail our experiment, let’s simulate two different scenarios of experimentation:
- Scenario 1: Including only users who are interested in chatbot functionality.
- Scenario 2: Including all new users into the experiment, regardless of whether they came for emails or chatbots.
Scenario 1: We include only those who had a chance to see the page with the discount offer
We ran experiment for two weeks to gather the required sample sizes and then checked the results. That’s what we found:
Control Group: 5,345 users, 802 conversions. Conversion rate is: 15%
Treatment Group: 5,345 users, 906 conversions. Conversion rate is: 16.95%
Turns out we were right in our assumptions and our expectations turned out to be true (in real life, it’s highly unlikely our expectations match so well with the reality).
Scenario 1: chatbot users only
- Conversion rate A
- 15.00%
- Conversion rate B
- 16.95%
- Relative uplift
- 12.97%
- z-score
- 2.7463
- p-value (one-sided)
- 0.0030
Scenario 2: We include all new users regardless of whether they showed interest in the chatbot functionality
Let’s run the same experiment for the same amount of time — two weeks, but this time we are going to include all new registered users regardless of whether they showed interest in the chatbot functionality.
We established earlier that we get around 26,725 new registered users a week, so we are going to have 53,450 participants.
Control Group: 26,725 users
Treatment Group: 26,725 users
We run the experiment for two weeks, crunch some numbers and decide to plug the figures into the same statistical significance calculator we used first time around. That’s what we find:
Control Group: 26,725 users, 4,009 conversions. Conversion rate is: 15%
Treatment Group: 26,725 users, 4,113 conversions. Conversion rate is: 15.39%
Scenario 2: all new users
- Conversion rate A
- 15.00%
- Conversion rate B
- 15.39%
- Relative uplift
- 2.59%
- z-score
- 1.2531
- p-value (one-sided)
- 0.1051
Well, we can no longer be 95% certain that our changes were an improvement over the original and we lost our statistical significance.
Let’s break down the numbers to see what happened exactly.
We already knew that both email and chatbot users had the same conversion rate into paying customers around 15%. We also know that only 20% of all users are chatbot users.
Since we included all new users into the experiment, we can break them down into groups like this:
Control Group: 26,725 users: 5,345 users (20%) who showed interest in chatbots and 21,380 users (80%) who showed interest in emails.
Treatment Group: 26,725 users: 5,345 users (20%) who showed interest in chatbots and 21,380 users (80%) who showed interest in emails.
The composition of Control and Treatment
| Email users | Chatbot users | Total participants | |
|---|---|---|---|
| Control | 26,725 × 0.8 = 21,380 | 26,725 × 0.2 = 5,345 | 26,725 |
| Treatment | 26,725 × 0.8 = 21,380 | 26,725 × 0.2 = 5,345 | 26,725 |
We already know from Scenario 1 that chatbot users who were exposed to the alternative version of the user interface showed an improvement in conversion rate which was around 16.95%. But since this time our treatment group includes both email and chatbot users, the conversion rate will be different:
The impact of email users on the treatment conversion rate
| Email users | Chatbot users | Participants | Conversions | Conversion rate | |
|---|---|---|---|---|---|
| Control | 21,380 × 0.15 = 3,207 | 5,345 × 0.15 = 802 | 26,725 | 4,009 | 0.15 |
| Treatment | 21,380 × 0.15 = 3,207 | 5,345 × 0.1695 = 906 | 26,725 | 4,113 | 0.1539 |
Since email users weren’t exposed to the new interface but were assigned to the treatment upon registration anyways, their conversion rate of 15%, which we couldn’t influence, dragged down the conversion rate of the entire Treatment group, regardless of the fact that chatbot users in the treatment showed a significant improvement in conversion rate, which increased to 16.95%.
The more noise there is (users, who were put into the experiment but didn’t have a chance to be exposed to the new variation) the less chance there is for us to see a statistically significant difference between variations. In our case, only 20% of all users were qualified for the experiment, but if their share in the overall user body was even smaller, say 10%, it would be even more difficult to see the impact of our changes.
Now, if we segmented the data and explored the chatbot users who saw the improved interface, we would still arrive at 16.95%, but that would require prior experiment configuration that would allow us to slice the data by user interest later, which would still require an additional layer of setup.
First published on Medium on Nov 8, 2022.