AB Testing Basics

Introduction

Online retailers spend a lot of money on getting their website up and running. A major portion of their investment is directed towards increasing the traffic of their websites or the conversion rate. A/B testing or split testing is one tool by which these retailers can pin point which elements  are to be displayed in their webpage which have a positive impact on number of customers visiting the website or increasing the conversion.

A/B Testing is basically the comparison between two versions of a single webpage to find out which version offers better result in terms of traffic or conversion.

Why A/B testing

A/B testing is a very handy cost effective way of determining the performance of any web page. Even the slightest change for eg: change in location of a Button can lead to a significant change in traffic or conversion rate.

What can you Test

There are loads of elements on a particular webpage which can be tested for better results. Some of these elements are listed below

1)      Headlines

2)      Content displayed on the webpage

3)      Location of header, footer

4)      Location of buttons

5)      News articles

6)      Different layout of images

Process

The figure below captures the entire A/B testing process

 

Determine Objective:  This is the most important step of A/B testing planning. Decide what do you want out of this testing? Is it more traffic or is it better conversion or is it for improving search rankings? Your objective will ultimately decide which pages to target and which elements of a web page should be changed.

Plan Accordingly:  Based on the objective, decide the budget and timelines for this project. Remember to remove all the test element s once this testing is over. Also be careful about SEO. Ensure that you display same content to the customers and also to the googlebot, otherwise google can remove your page from the search rankings. Also plan to use 302 redirects (temporary) rather than 301 (permanent) . This tells the search engine that the link is temporary.  There are also a couple of other SEO things which are to be considered for A/B testing. Failure to do so will lead to sandboxing by Google.

Conduct Research: Find out different stats about your website like how many users visit the landing page. What is the click through rate across targeted elements? What is the bounce rate?

Conduct the experiment & analyze data : Your objective will define your sample. The sample could be a random sample (every customer has equal chance of being selected for this testing) or a stratified sample (a specific customer segment can be selected for this testing by choosing a specific time frame or an occasion). Also choose an appropriate sample size .You can use Google Analytics or any other tool for data analysis.

Create a Hypothesis: You will have to start by assuming certain results. It’s called Hypothesizing.  Statisticians use hypothesis to prove or reject a statement based on analysis of data.

For eg

H(0): Null Hypothesis : The color of button ‘Add to cart’ has no impact on the conversion rate.

H(1): Alternate Hypothesis: The  color of button ‘Add to cart’ increases the conversion rate

Test the Hypothesis:  Based on the data collected during the testing, you can use a variety of probability distributions like Normal distribution or t distribution to find out whether the Null Hypothesis is true and hence reach a conclusion.

Track the results and repeat the experiment: If   your analysis says that your change did not give you the correct result, then you need to make some more changes and repeat the entire process.

Please find below some important and useful links for AB Testing

1)  https://www.optimizely.com/ab-testing

2) http://www.maxymiser.com/resources/ab-testing

3) http://kb.mailchimp.com/article/how-a-b-split-testing-works