Before we go further into the steps required for A/B testing, let’s rewind back to the concept of hypothesis testing. Let’s consider the two-population proportion example where we want to test if the proportion of the desired characteristic is statistically the same in the two populations.

When testing the difference between two proportions, i.e, the proportion of desired characteristics in the two populations, the null hypothesis is usually that.

$$/$$

You would also remember that we generally follow the six-step process for carrying out any hypothesis test:

Set up the null and alternative hypotheses, and decide the significance level.
Calculate the sample statistic.
Formulate the test statistic.
Make the decision criteria.
Calculate the test statistic.
Take decision.

Now, in the case of the two-population proportion, the test statistic takes the form:

$z = \frac{(\hat{p_1} - \hat{p_2}) - 0}{\sqrt{\bar{p}*(1 - \bar{p})*(\frac{1}{n_1} + \frac{1}{n_2})}}$ .

Let’s recall these with the help of an example that you already saw in the module on hypothesis testing.

Remember the last example regarding the survey about 'Achche Din'? You tried to find out if the proportion of people optimistic about 'Achche Din' is different in the BJP-ruled states and the opposition-ruled states.

So you carried out another survey and tabulated the result.

Table 1: Survey on 'Achche Din'
S. No.	Ruling Party	Sample Size	No. of Optimistic Respondents	No. of Pessimistic Respondents
1	BJP	100	57	43
2	Others	100	39	61

Now, let $\hat{p_1}$ represent the proportion of sample optimistic about 'Achche Din' in the BJP-ruled states and $\hat{p_2}$ represent the proportion of sample optimistic about 'Achche Din' in the non-BJP ruled states. So, now formulating the null and alternative hypotheses, we get:

$H_0: P_1 = P_2$ ,

$H_1: P_1 \neq P_2$ ,

where $P_1$ represents the proportion of population optimistic about 'Achche Din' in the BJP-ruled states and $P_2$ represents the proportion of population optimistic about 'Achche Din' in the non-BJP ruled states. Now, if you look at the formula for the z-statistic,

$z = \frac{(\hat{p_1} - \hat{p_2}) - 0}{\sqrt{\bar{p}*(1 - \bar{p})*(\frac{1}{n_1} + \frac{1}{n_2})}}$ ,

you will notice that we need a pooled proportion $\bar{p}$ , which is given by the total number of people optimistic about 'Achche Din' divided by the total number of people surveyed across the two categories of states.

Thus, $\bar{p}$ = (57 +39)/ (100 +100) = 96/200 = 0.48.

$$/$$

Now, for a significance level of 0.05, the critical z-values are -1.96 and +1.96. So, depending on the value of the test statistic, you need to either reject or fail to reject the null hypothesis.

$$/$$

Having refreshed your knowledge about the two-population proportion test, you can now go back to understanding how to formulate an A/B test.

$$/$$

You can use this link to find the sample size calculator from Optimizely.

You saw the different steps involved in setting up an A/B test. These are:

Analysing the data.
Forming a hypothesis.
Constructing the experiment.
Interpreting the result.

Additional Reading

You can read more about the mathematics behind A/B testing and the calculation of sample size using the links provided.