What is Linear Discriminant Analysis for Machine Learning?

Linear Discriminant Analysis or LDA in machine learning is a dimensionality reduction technique. It is used as a pre-processing step in Machine Learning and applications of pattern classification. The goal of LDA is to project the features in higher dimensional space onto a lower-dimensional space in order to avoid the curse of dimensionality and also reduce resources and dimensional costs.

LDA is a powerful supervised classification technique, playing a very integral role in crafting competitive machine learning models. Its versatility spans across various domains, from image recognition to predictive analysis in marketing. 

Learn more about machine learning and data science through our detailed Data Science Course. Now, let's dive in to understand LDA in machine learning, how it works, its limitations, and more through our article. 

What is Linear Discriminant Analysis (LDA)?  

Linear Discriminant Analysis for Machine Learning (LDA) is a common statistical method for machine learning and pattern identification. It is used mostly for dimensionality reduction and classification tasks. The primary principle behind LDA is to identify a linear combination of attributes that best distinguishes two or more classes of objects or events.

LDA assumes that separate classes create data with Gaussian distributions and seeks to maximize the ratio of between-class variance to within-class variance in any given dataset, resulting in maximum separability. The combination of attributes produces the greatest "discriminant" or distinguishing information about the classes. 

How to Have a Practical Approach to an LDA Model?

Let's consider a situation where you have plotted the relationship between two variables where each color represents a different class. One is shown with a red color and the other with blue.

If you are willing to reduce the number of dimensions to 1, you can just project everything to the x-axis as shown below:

This approach neglects any helpful information provided by the second feature. However, you can use LDA to plot it. The advantage of LDA is that it uses information from both the features to create a new axis which in turn minimizes the variance and maximizes the class distance of the two variables.

Read : 25 Machine Learning Interview Questions & Answers

How Does LDA Work in Machine Learning?

LDA focuses primarily on projecting the features in higher dimension space to lower dimensions. You can achieve this in three steps:

• Firstly, you need to calculate the separability between classes which is the distance between the mean of different classes. This is called the between-class variance.

• Secondly, calculate the distance between the mean and sample of each class. It is also called the within-class variance.

• Finally, construct the lower-dimensional space which maximizes the between-class variance and minimizes the within-class variance. P is considered as the lower-dimensional space projection, also called Fisher’s criterion.

How are LDA Models Represented?

The representation of LDA is pretty straight-forward. The model consists of the statistical properties of your data that has been calculated for each class. The same properties are calculated over the multivariate Gaussian in the case of multiple variables. The multivariates are means and covariate matrix.

Predictions are made by providing the statistical properties into the Linear Discriminant Analysis for Machine Learning equation. The properties are estimated from your data. Finally, the model values are saved to file to create the LDA model.

How Do LDA in Machine Learning Models Learn?

The assumptions made by an LDA model about your data:

• Each variable in the data is shaped in the form of a bell curve when plotted,i.e. Gaussian.
• The values of each variable vary around the mean by the same amount on the average,i.e. each attribute has the same variance.

The LDA model is able to estimate the mean and variance from your data for each class with the help of these assumptions.

The mean value of each input for each of the classes can be calculated by dividing the sum of values by the total number of values:

Mean =Sum(x)/Nk

where Mean = mean value of x for class
           N = number of
           k = number of
           Sum(x) = sum of values of each input x.

The variance is computed across all the classes as the average of the square of the difference of each value from the mean:

Σ²=Sum((x - M)²)/(N - k)

where  Σ² = Variance across all inputs x.
            N = number of instances.
            k = number of classes.
            Sum((x - M)²) = Sum of values of all (x - M)².
            M = mean for input x.

Enrol for the Machine Learning Course from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

How Does an LDA Model Make Predictions?

LDA, i.e. linear discriminant analysis in machine learning, models uses Bayes’ Theorem to estimate probabilities. They make predictions based upon the probability that a new input dataset belongs to each class. The class which has the highest probability is considered the output class and then the LDA makes a prediction.  

The prediction is made simply by the use of Bayes’ Theorem which estimates the probability of the output class given the input. They also make use of the probability of each class and the probability of the data belonging to each class:

P(Y=x|X=x)  = [(Plk * fk(x))] / [(sum(PlI * fl(x))]

Where x = input.
            k = output class.
            Plk = Nk/n or base probability of each class observed in the training data. It is also called prior probability in Bayes’ Theorem.
            fk(x) = estimated probability of x belonging to class k.

The f(x) is plotted using a Gaussian Distribution function and then it is plugged into the equation above and the result we get is the equation as follows:

Dk(x) = x∗(mean/Σ²) – (mean²/(2*Σ²)) + ln(PIk)

The Dk(x) is called the discriminant function for class k given input x, mean,  Σ² and Plk are all estimated from the data and the class is calculated as having the largest value, will be considered in the output classification.

How to Prepare Data from LDA?

Some suggestions you should keep in mind while preparing your data to build your LDA model:

• LDA is mainly used in classification problems where you have a categorical output variable. It allows both binary classification and multi-class classification.
• The standard LDA model makes use of the Gaussian Distribution of the input variables. You should check the univariate distributions of each attribute and transform them into a more Gaussian-looking distribution. For example, for the exponential distribution, use log and root function and for skewed distributions use BoxCox.
• Outliers can skew the primitive statistics used to separate classes in LDA, so it is preferable to remove them.
• Since LDA assumes that each input variable has the same variance, it is always better to standardize your data before using an LDA model. Keep the mean to be 0 and the standard deviation to be 1.

How to Implement an LDA Model from scratch?

You can implement a Linear Discriminant Analysis model from scratch using Python. Let’s start by importing the libraries that are required for the model:

from sklearn.datasets import load_wine
import pandas as pd
import numpy as np
np.set_printoptions(precision=4)
from matplotlib import pyplot as plt
import seaborn as sns
sns.set()
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix

Since we will work with the wine dataset, you can obtain it from the UCI machine learning repository. The scikit-learn library in Python provides a wrapper function for downloading it:

wine_info = load_wine()
X = pd.DataFrame(wine_info.data, columns=wine_info.feature_names)
y = pd.Categorical.from_codes(wine_info.target, wine_info.target_names)

The wine dataset comprises of 178 rows of 13 columns each:

X.shape

(178, 13)

The attributes of the wine dataset comprise of various characteristics such as alcohol content of the wine, magnesium content, color intensity, hue and many more:

X.head()

The wine dataset contains three different kinds of wine:

wine_info.target_names 
array(['class_0', 'class_1', 'class_2'], dtype='<U7')

Now we create a DataFrame which will contain both the features and the content of the dataset:

df = X.join(pd.Series(y, name='class'))

We can divide the process of Linear Discriminant Analysis into 5 steps as follows:

Step 1 - Computing the within-class and between-class scatter matrices.
Step 2 - Computing the eigenvectors and their corresponding eigenvalues for the scatter matrices.
Step 3 - Sorting the eigenvalues and selecting the top k.
Step 4 - Creating a new matrix that will contain the eigenvectors mapped to the k eigenvalues.
Step 5 - Obtaining new features by taking the dot product of the data and the matrix from Step 4.

Within-class scatter matrix

To calculate the within-class scatter matrix, you can use the following mathematical expression:

where, c = total number of distinct classes and

where, x = a sample (i.e. a row).
            n = total number of samples within a given class.

Now we create a vector with the mean values of each feature:

feature_means1 = pd.DataFrame(columns=wine_info.target_names)
for c, rows in df.groupby('class'):
feature_means1[c] = rows.mean()
feature_means1

The mean vectors (mi ) are now plugged into the above equations to obtain the within-class scatter matrix:

withinclass_scatter_matrix = np.zeros((13,13))
for c, rows in df.groupby('class'):
rows = rows.drop(['class'], axis=1)

s = np.zeros((13,13))
for index, row in rows.iterrows():
x, mc = row.values.reshape(13,1),
feature_means1[c].values.reshape(13,1)

s += (x - mc).dot((x - mc).T)

withinclass_scatter_matrix += s

Between-class scatter matrix

We can calculate the between-class scatter matrix using the following mathematical expression:

where,

and

feature_means2 = df.mean()
betweenclass_scatter_matrix = np.zeros((13,13))
for c in feature_means1:    
   n = len(df.loc[df['class'] == c].index)
   mc, m = feature_means1[c].values.reshape(13,1), 
feature_means2.values.reshape(13,1)
betweenclass_scatter_matrix += n * (mc - m).dot((mc - m).T)

Now we will solve the generalized eigenvalue problem to obtain the linear discriminants for:

eigen_values, eigen_vectors = 
np.linalg.eig(np.linalg.inv(withinclass_scatter_matrix).dot(betweenclass_scatter_matrix))

We will sort the eigenvalues from the highest to the lowest since the eigenvalues with the highest values carry the most information about the distribution of data is done. Next, we will first k eigenvectors. Finally, we will place the eigenvalues in a temporary array to make sure the eigenvalues map to the same eigenvectors after the sorting is done:

eigen_pairs = [(np.abs(eigen_values[i]), eigen_vectors[:,i]) for i in range(len(eigen_values))]
eigen_pairs = sorted(eigen_pairs, key=lambda x: x[0], reverse=True)
for pair in eigen_pairs:
print(pair[0])

237.46123198302251
46.98285938758684
1.4317197551638386e-14
1.2141209883217706e-14
1.2141209883217706e-14
8.279823065850476e-15
7.105427357601002e-15
6.0293733655173466e-15
6.0293733655173466e-15
4.737608877108813e-15
4.737608877108813e-15
2.4737196789039026e-15
9.84629525010022e-16

Now we will transform the values into percentage since it is difficult to understand how much of the variance is explained by each component.

sum_of_eigen_values = sum(eigen_values)
print('Explained Variance')
for i, pair in enumerate(eigen_pairs):
   print('Eigenvector {}: {}'.format(i, (pair[0]/sum_of_eigen_values).real))

Explained Variance
Eigenvector 0: 0.8348256799387275
Eigenvector 1: 0.1651743200612724
Eigenvector 2: 5.033396012077518e-17
Eigenvector 3: 4.268399397827047e-17
Eigenvector 4: 4.268399397827047e-17
Eigenvector 5: 2.9108789097898625e-17
Eigenvector 6: 2.498004906118145e-17
Eigenvector 7: 2.119704204950956e-17
Eigenvector 8: 2.119704204950956e-17
Eigenvector 9: 1.665567688286435e-17
Eigenvector 10: 1.665567688286435e-17
Eigenvector 11: 8.696681541121664e-18
Eigenvector 12: 3.4615924706522496e-18

First, we will create a new matrix W using the first two eigenvectors:

W_matrix = np.hstack((eigen_pairs[0][1].reshape(13,1), eigen_pairs[1][1].reshape(13,1))).real

Next, we will save the dot product of X and W into a new matrix Y:

Y = X∗W

where, X = n x d matrix with n sample and d dimensions.
            Y = n x k matrix with n sample and k dimensions.

In simple terms, Y is the new matrix or the new feature space.

X_lda = np.array(X.dot(W_matrix))

Our next work is to encode every class a member in order to incorporate the class labels into our plot. This is done because matplotlib cannot handle categorical variables directly.

Finally, we plot the data as a function of the 2 LDA components using different color for each class:

plt.xlabel('LDA1')
plt.ylabel('LDA2')
plt.scatter(
X_lda[:,0],
X_lda[:,1],
c=y,
cmap='rainbow',
alpha=0.7,
edgecolors='b'
)

<matplotlib.collections.PathCollection at 0x7fd08a20e908>

What Are the Extensions to LDA?

LDA is considered to be a very simple and effective method, especially for classification techniques. Since it is simple and well understood, so it has a lot of extensions and variations:

• Quadratic Discriminant Analysis(QDA) – 

  When there are multiple input variables, each of the class uses its own estimate of variance and covariance.

• Flexible Discriminant Analysis(FDA) – 

  This technique is performed when a non-linear combination of inputs is used as splines.

• Regularized Discriminant Analysis(RDA) – 

  It moderates the influence of various variables in LDA by regularizing the estimate of the covariance.

Also read: 15 Interesting Machine Learning Project Ideas For Beginners

Linear Discriminant Analysis vs PCA

Below are the differences between LDA and PCA in machine learning:

• PCA ignores class labels and focuses on finding the principal components that maximizes the variance in a given data. Thus it is an unsupervised algorithm. On the other hand, LDA is a supervised algorithm that intends to find the linear discriminants that represents those axes which maximize separation between different classes.
• LDA performs better multi-class classification tasks than PCA. However, PCA performs better when the sample size is comparatively small. An example would be comparisons between classification accuracies that are used in image classification.
• Both LDA and PCA are used in case of dimensionality reduction. PCA is first followed by LDA.

Let us create and fit an instance of the PCA class:

from sklearn.decomposition import PCA
pca_class = PCA(n_components=2)
X_pca = pca.fit_transform(X, y)

Again, to view the values in percentage for a better understanding, we will access the explained_variance_ratio_ property:

pca.explained_variance_ratio_

array([0.9981, 0.0017])

Clearly, PCA selected the components which will be able to retain the most information and ignores the ones which maximize the separation between classes.

plt.xlabel('PCA1')
plt.ylabel('PCA2')
plt.scatter(
   X_pca[:,0],
   X_pca[:,1],
   c=y,
   cmap='rainbow',
   alpha=0.7,
   edgecolors='b

Now to create a classification model using the LDA components as features, we will divide the data into training datasets and testing datasets:

X_train, X_test, y_train, y_test = train_test_split(X_lda, y, random_state=1)

The next thing we will do is create a Decision Tree. Then, we will predict the category of each sample test and create a confusion matrix to evaluate the LDA model’s performance:

data = DecisionTreeClassifier()
data.fit(X_train, y_train)
y_pred = data.predict(X_test)
confusion_matrix(y_test, y_pred)

array([[18,  0,  0], 
       [ 0, 17,  0], 
       [ 0,  0, 10]])

So it is clear that the Decision Tree Classifier has correctly classified everything in the test dataset.

Advantages & Disadvantages of using LDA  in Machine Learning

Learn about: Python Project Ideas & Topics

Real-Life Applications of LDA

Some of the practical applications of LDA are listed below:

• Face Recognition – 

  LDA is used in face recognition to reduce the number of attributes to a more manageable number before the actual classification. The dimensions that are generated are a linear combination of pixels that forms a template. These are called Fisher’s faces.

• Medical – 

  You can use LDA to classify the patient disease as mild, moderate or severe. The classification is done upon the various parameters of the patient and his medical trajectory. 

• Customer Identification –

  You can obtain the features of customers by performing a simple question and answer survey. LDA helps in identifying and selecting which describes the properties of a group of customers who are most likely to buy a particular item in a shopping mall.

Conclusion

The Linear Discriminant Analysis in Python or LDA in machine learning to be more precise is a very simple and well-understood approach of classification in machine learning. Though there are other dimensionality reduction techniques like Logistic Regression or PCA, but LDA is preferred in many special classification cases. 

If you want to be an expert in machine learning, knowledge of Linear Discriminant Analysis would lead you to that position effortlessly and Data Science and Machine Learning Courses are the best means to do so.

If you’re interested to learn more about machine learning, check out IIIT-B & upGrad’s PG Diploma in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.

Explore our top AI and ML blogs and free courses, packed with insights and hands-on skills to fuel your journey in artificial intelligence and machine learning.

Master trending machine learning skills with our expert-led courses, designed to keep you ahead in the fast-paced world of AI and data science.

Unlock the world of Machine Learning and AI with our top online courses, designed to empower you with high-impact skills for tomorrow’s tech careers.