Bayes Theorem Explained With Example – Complete Guide

Introduction

What is Bayes Theorem?

Bayes’s theorem is used for the calculation of a conditional probability where intuition often fails. Although widely used in probability, the theorem is being applied in the machine learning field too. Its use in machine learning includes the fitting of a model to a training dataset and developing classification models. 

What is conditional probability?

A conditional probability is usually defined as the probability of one event given the occurrence of another event. 

• If A and B are two events, then the conditional probability me be designated as P(A given B) or P(A|B). 
• Conditional probability can be calculated from the joint probability(A | B) = P(A, B) / P(B)
• The conditional probability is not symmetrical; For example P(A | B) != P(B | A)

Other ways of calculating conditional probability includes using the other conditional probability, i.e.

P(A|B) = P(B|A) * P(A) / P(B)

Reverse is also used

P(B|A) = P(A|B) * P(B) / P(A)

This way of calculation is useful when it is challenging to calculate the joint probability. Else, when the reverse conditional probability is available, calculation through this becomes easy.

This alternate calculation of conditional probability is referred to as the Bayes Rule or Bayes Theorem. It is named under the person who first described it, “Reverend Thomas Bayes”.

The Formula of Bayes theorem

Bayes theorem is a way of calculating conditional probability when the joint probability is not available. Sometimes, the denominator can’t be directly accessed. In such cases, the alternative way of calculating is as:

 P(B) = P(B|A) * P(A) + P(B|not A) * P(not A) 

This is the formulation of the Bayes theorem which shows an alternate calculation of P(B).

P(A|B) = P(B|A) * P(A) / P(B|A) * P(A) + P(B|not A) * P(not A)

The above formula can be described with brackets around the denominator

P(A|B) = P(B|A) * P(A) / (P(B|A) * P(A) + P(B|not A) * P(not A))

Also, if we have P(A), then the P(not A) can be calculated as

P(not A) = 1 – P(A)

Similarly, if we have P(not B|not A),then P(B|not A) can be calculated as

P(B|not A) = 1 – P(not B|not A)

Bayes Theorem of Conditional Probability

Bayes Theorem consists of several terms whose names are given based on the context of its application in the equation.

Posterior probability refers to the result of P(A|B), and prior probability refers to P(A).

• P(A|B): Posterior probability.
• P(A): Prior probability.

Similarly, P(B|A) and P(B)  is referred to as the likelihood and evidence.

• P(B|A): Likelihood.
• P(B): Evidence.

Therefore, the Bayes theorem of conditional probability can be restated as:

Posterior = Likelihood * Prior / Evidence

If we have to calculate the probability that there is fire given that there is smoke, then the following equation will be used:

P(Fire|Smoke) = P(Smoke|Fire) * P(Fire) / P(Smoke)

Where, P(Fire) is the Prior, P(Smoke|Fire) is the Likelihood, and P(Smoke) is the evidence.

An Illustration of Bayes theorem

A Bayes theorem example is described to illustrate the use of Bayes theorem in a problem.

Problem

Three boxes labeled as A, B, and C, are present. Details of the boxes are:

• Box A contains 2 red and 3 black balls
• Box B contains 3 red and 1 black ball
• And box C contains 1 red ball and 4 black balls

All the three boxes are identical having an equal probability to be picked up. Therefore, what is the probability that the red ball was picked up from box A?

Solution

Let E denote the event that a red ball is picked up and A, B and C denote that the ball is picked up from their respective boxes. Therefore the conditional probability would be P(A|E) which needs to be calculated.

The existing probabilities P(A) = P(B) = P (C) = 1 / 3, since all boxes have equal probability of getting picked.

P(E|A) = Number of red balls in box A / Total number of balls in box A = 2 / 5

Similarly, P(E|B) = 3 / 4 and P(E|C) = 1 / 5

Then evidence P(E) = P(E|A)*P(A) + P(E|B)*P(B) + P(E|C)*P(C) 

                   = (2/5) * (1/3) + (3/4) * (1/3) + (1/5) * (1/3) = 0.45

Therefore, P(A|E) = P(E|A) * P(A) / P(E) = (2/5) * (1/3) / 0.45 = 0.296

Example of Bayes Theorem

Bayes theorem gives the probability of an “event” with the given information on “tests”.

• There is a difference between “events” and “tests”. For example there is a test for liver disease, which is different from actually having the liver disease, i.e. an event.
• Rare events might be having a higher false positive rate.

Example 1

What is the probability of a patient having liver disease if they are alcoholic? 

Here, “being an alcoholic” is the “test” (type of litmus test) for liver disease.

• A is the event i.e. “patient has liver disease””.

As per earlier records of the clinic, it states that 10% of the patient’s entering the clinic are suffering from liver disease. 

Therefore, P(A)=0.10

• B is the litmus test that “Patient is an alcoholic”.

Earlier records of the clinic showed that 5% of the patients entering the clinic are alcoholic.

Therefore, P(B)=0.05

• Also,  7% out of the he patient’s that are diagnosed with liver disease, are alcoholics. This defines the B|A: probability of a patient being alcoholic, given that they have a liver disease is 7%.

As, per Bayes theorem formula, 

P(A|B) = (0.07 * 0.1)/0.05 = 0.14

Therefore, for a patient being alcoholic, the chances of having a liver disease are 0.14 (14%).

Example2

• Dangerous fires are rare (1%)
• But smoke is fairly common (10%) due to barbecues,
• And 90% of dangerous fires make smoke

What is the probability of dangerous Fire when there is Smoke?

Calculation

P(Fire|Smoke) =P(Fire) P(Smoke|Fire)/P(Smoke)

= 1% x 90%/10% 

= 9%

Example 3

What is the chance of rain during the day? Where, Rain means rain during the day, and Cloud means cloudy morning.

The chance of Rain given Cloud is written P(Rain|Cloud)

P(Rain|Cloud) =  P(Rain) P(Cloud|Rain)/P(Cloud) 

P(Rain) is Probability of Rain = 10%

P(Cloud|Rain) is Probability of Cloud, given that Rain happens = 50%

P(Cloud) is Probability of Cloud = 40%

P(Rain|Cloud) =  0.1 x 0.5/0.4   = .125

Therefore, a 12.5% chance of rain. 

Applications

Several applications of Bayes theorem exist in the real world. The few main applications of the theorem are:

1. Modelling Hypotheses

The Bayes theorem finds wide application in the applied machine learning and establishes a relationship between the data and a model. Applied machine learning uses the process of testing and analysis of different hypotheses on a given dataset.

To describe the relationship between the data and the model, the Bayes theorem provides a probabilistic model.

P(h|D) = P(D|h) * P(h) / P(D)

Where, 

P(h|D): Posterior probability of the hypothesis 

P(h): Prior probability of the hypothesis.

An increase in P(D) decreases the P(h|D). Conversely, if P(h) and the probability of observing the data given hypothesis increases, then the probability of P(h|D) increases.

2. Bayes Theorem for Classification

The method of classification involves the labelling of a given data. It can be defined as the calculation of the conditional probability of a class label given a data sample.

P(class|data) = (P(data|class) * P(class)) / P(data)

Where P(class|data) is the probability of class given the provided data.

The calculation can be carried out for each class. The class having the largest probability can be assigned to the input data.

Calculation of the conditional probability is not feasible under conditions of a small number of examples. Therefore, the direct application of the Bayes theorem is not feasible. A solution to the classification model lies in the simplified calculation.

• Naive Bayes Classifier

Bayes theorem considers that input variables are dependent on other variables which cause the complexity of calculation. Therefore, the assumption is removed and every input variable is considered as an independent variable. As a result the model changes from dependent to independent conditional probability model. It ultimately reduces the complexity.

This simplification of the Bayes theorem is referred to as the Naïve Bayes. It is widely used for classification and predicting models.

• Bayes Optimal Classifier

This is a type of probabilistic model that involves the prediction of a new example given the training dataset. One example of the Bayes Optimal Classifier is “What is the most probable classification of the new instance given the training data?”

Calculation of the conditional probability of a new instance given the training data can be done through the following equation

P(vj | D) = sum {h in H} P(vj | hi) * P(hi | D)

Where vj is a new instance to be classified,

 H is the set of hypotheses for classifying the instance,

 hi is a given hypothesis, 

P(vj | hi) is the posterior probability for vi given hypothesis hi, and 

P(hi | D) is the posterior probability of the hypothesis hi given the data D.

3. Uses of Bayes theorem in Machine learning

The most common application of the Bayes theorem in machine learning is the development of classification problems. Other applications rather than the classification include optimization and casual models.

• Bayesian optimization

It is always a challengeable task to find an input that results in the minimum or maximum cost of a given objective function. Bayesian optimization is based on the Bayes theorem and provides an aspect for the search of a global optimization problem. The method includes the building of a probabilistic model (surrogate function), searching through an acquisition function, and the selection of candidate samples for evaluating the real objective function.

In applied machine learning, Bayesian optimization is used to tune the hyperparameters of a well-performing model.

• Bayesian Belief networks

Relationships between the variables may be defined through the use of probabilistic models. They are also used for the calculation of probabilities.  A fully conditional probability model might not be able to calculate the probabilities due to the large volume of data. Naïve Bayes has simplified the approach for the calculation. Yet another method exists where a model is developed based on the known conditional dependence between random variables and conditional independence in other cases. The Bayesian network displays this dependence and independence through the probabilistic graph model with directed edges. The known conditional dependence is displayed as directed edges and the missing connections represent the conditional independencies in the model.

4. Bayesian Spam filtering

Spam filtering is another application of Bayes theorem. Two events are present:

• Event A: The message is spam.
• Test X: The message contains certain words (X)