Before we go further into the steps required for A/B testing, let’s rewind back to the concept of hypothesis testing. Let’s consider the two-population proportion example where we want to test if the proportion of the desired characteristic is statistically the same in the two populations.
When testing the difference between two proportions, i.e, the proportion of desired characteristics in the two populations, the null hypothesis is usually that.
You would also remember that we generally follow the six-step process for carrying out any hypothesis test:
Now, in the case of the two-population proportion, the test statistic takes the form:
.
Let’s recall these with the help of an example that you already saw in the module on hypothesis testing.
Remember the last example regarding the survey about 'Achche Din'? You tried to find out if the proportion of people optimistic about 'Achche Din' is different in the BJP-ruled states and the opposition-ruled states.
So you carried out another survey and tabulated the result.
S. No. | Ruling Party | Sample Size | No. of Optimistic Respondents | No. of Pessimistic Respondents |
---|---|---|---|---|
1 | BJP | 100 | 57 | 43 |
2 | Others | 100 | 39 | 61 |
Now, let represent the proportion of sample optimistic about 'Achche Din' in the BJP-ruled states and
represent the proportion of sample optimistic about 'Achche Din' in the non-BJP ruled states. So, now formulating the null and alternative hypotheses, we get:
,
,
where represents the proportion of population optimistic about 'Achche Din' in the BJP-ruled states and
represents the proportion of population optimistic about 'Achche Din' in the non-BJP ruled states. Now, if you look at the formula for the z-statistic,
,
you will notice that we need a pooled proportion , which is given by the total number of people optimistic about 'Achche Din' divided by the total number of people surveyed across the two categories of states.
Thus, = (57 +39)/ (100 +100) = 96/200 = 0.48.
Now, for a significance level of 0.05, the critical z-values are -1.96 and +1.96. So, depending on the value of the test statistic, you need to either reject or fail to reject the null hypothesis.
Having refreshed your knowledge about the two-population proportion test, you can now go back to understanding how to formulate an A/B test.
You can use this link to find the sample size calculator from Optimizely.
You saw the different steps involved in setting up an A/B test. These are:
Additional Reading
You can read more about the mathematics behind A/B testing and the calculation of sample size using the links provided.