Today, it is relatively simple to experiment with different versions of the same website. There are many technologies and tools that can help e-commerce businesses build and run randomised controlled trials (otherwise known as A/B tests). The amount of data available to large e-commerce sites means that businesses can measure the effect of changing design, messaging and merchandising. Over the last three years, Qubit has been helping these businesses explore which changes are associated with an increase in revenue.
In previous work , Qubit showed that many of the practices used in the A/B testing industry at the time were fundamentally flawed. Since its release we have seen a change in both the statistical models used in the industry, and a shift to more robust experimental procedures. In this paper, we would like to move the industry forward again, and answer the question – what kind of changes do our clients make, and how do they impact revenue?
We will present the results of a meta-analysis, conducted in 2017, on Qubit’s large database of experiments. We will describe the effects of 29 treatment types and estimate the cumulative impact of these experiments on site wide revenue. The methodology used in this paper was independently assured by PricewaterhouseCoopers UK LLP (PwC)1. To our knowledge, this is the first published, independently assured quantitative analysis of its type. We hope it will be used to improve the quality of A/B testing, to reset expectations, and to prioritise optimisations to websites.
We have decided to separate this work into three sections to answer three slightly different questions, keeping methodologies and results together where possible. In section 2 we divide our experiments into different treatment categories, and estimate the overall impact of each of them. In section 3 we estimate the overall distribution of all experiment impacts used in this work. In section 4 we look at how A/B testing impacts overall site-wide revenue across sets of web-domains. There are a number of appendices expanding on the results of these sections.