Caveats of Testing Elements below the Fold

Below the fold refers to the section of a web page that is only visible after scrolling down. The digital marketing conventional wisdom is that above the fold content gets more attention, and is therefore more valuable than below the fold content.

We’ve been running experiments on our website on an on-and-off basis for some time, and when it comes down to calculating the required sample size, the logic is seemingly simple. Say, we wanted to test a certain change on a landing page.

• Get the base conversion rate for the page
• Figure out the expected uplift (which we will set as our MDE — minimum detectable effect)
• Specify the statistical power (usually at 0.80) and error probability (usually at 0.05)
• Get the required sample size
• Divide it by the average monthly visitors we get to the landing page

This would allow us to get the required sample size and come up with some sort of projections for the required testing time.

However, this is a naive approach and it might be a bit more complicated than that, especially when we test something that is not immediately seen by visitors. I am talking about changes below the fold.

As we recently discovered at SendPulse, only 8% of visitors to our landing pages across the website scroll to the bottom of it. So if we wanted to test, say, changes to “Testimonials” or “Our Customers” blocks, which we have somewhere at the bottom of landing pages, sample size calculations and experiment runtime prognosis should probably take this into account.

Let’s say we want to test the aforementioned “Testimonials” block on our ourbrand.com/features/product page by tweaking it a little — adding a photo of the person, their position and their company, as well as a link to their LinkedIn profile. This should hypothetically improve trust to the testimonials and bump up the conversion rate.

The historical base conversion rate of the landing page is at 10% with 100,000 monthly visitors on average.

If we set the MDE (minimum detectable effect) at 5% with the statistical power of 80%, the required sample size would be something around 91,000 visitors (45,500 visitors per variant). We choose this MDE as we believe that changes like that are unlikely to have drastic impact on the overall conversion rate and 5% is something we should probably aim for.

Required sample size for a 5% MDE

Input

Tails

One

Proportion p1

0.10

Proportion p2

0.105

α

0.05

Target power

0.80

Sample size per group

45,500

Output

Critical z

1.6449

Noncentrality δ

2.4865

Power (1−β)

0.8000

These naive calculations would be OK for testing any elements and changes that are almost guaranteed to be seen by visitors (something on the first screen or above the fold), and so, the visitors will be correctly assigned to the experiment.

However, since we know that only 8% of the visitors are actually going to see the “Testimonials” block, change or no change, things get drastically different. Instead of monthly 100,000 visitors who we expected to participate in the experiment, the real number is somewhere around 8,000 — these are visitors who will scroll the page and REALLY be exposed to the change.

The rationale behind this is exactly the same as testing changes to pop-up elements to changes to different pages altogether:

• If we test changes to certain pop-up, we base our prognosis on visitors who actually triggered and saw the pop-up, and not simply those who visited the page where pop-up could be triggered.
• If we test a change to a single page, we don’t include visitors to the other pages in our experiment, nor do we include them in our prognosis and sample size calculations.

Thinking in terms of tested landing page visitors has its shortcomings, but thinking in terms of “number of visitors to whom the element can be exposed to” may be desirable.

And seeing the potential uplift of 5% with the sample size like that in a span of 30 days or so is very unlikely.

With 4,000 visitors per variant, our statistical power would be somewhere around 0.18 at the 5% MDE, which makes it very unlikely for us to detect the positive change in a month.

Achieved power with 4,000 visitors per variant

Input

Tails

One

Proportion p1

0.10

Proportion p2

0.105

α

0.05

Sample size per group

4,000

Output

Critical z

1.6449

Noncentrality δ

0.7373

Power (1−β)

0.1820

And if we wanted the statistical power to be fixed at 0.80, we would need to run the experiment for almost a year:

91 000 / 8 000 = 11.37 (months)

Where

91 000 — required sample size
8 000 — monthly average number of visitors exposed to the element being tested

But, given the knowledge we have, there’s one question that begs asking:

“Should we spend our time on testing something that visitors rarely see?”

And I personally believe that even if we should not completely discard experiments of this kind, we should at least lower their priority and concentrate on changes that visitors are more likely to see (the rationale is the same as testing things that impact more visitors as opposed to a limited segment). Moreover, if we believe that the “Testimonials” block is capable of increasing our conversion rate, why not move it and place it somewhere near the top of the page where visitors are more likely to see it?