Cross Validation in Machine Learning: 4 Types of Cross Validation

Introduction

Model Development is a crucial step in a Data Science Project Life Cycle where we will try to train our data set with different types of Machine Learning models either of Supervised or Unsupervised Algorithms based on the Business Problem.

As we are aware that we have a lot of models that can be used to solve a business problem we need to assure that whatever model we select at the end of this phase should be performing well on the unseen data. So, we cannot just go with the evaluation metrics in order to select our best performing model.

We need something more apart from the metric which can help us to decide on our final Machine Learning model which we can deploy to production.

The process of determining whether the mathematical results calculating relationships between variables are acceptable as descriptions of the data is known as Validation. Usually, an error estimation for the model is made after training the model on the train data set, better known as the evaluation of residuals.

In this process, we measure the Training Error by calculating the difference between predicted response and original response. But this metric cannot be trusted because it works well only with the training data. It’s possible that the model is Underfitting or Overfitting the data.

Enrol for the Machine Learning Course from the World’s top Universities. Earn Masters, Executive PGP, or Advanced Certificate Programs to fast-track your career.

So, the problem with this evaluation technique or any other evaluation metric is that it does not give an indication of how well the model will perform to an unseen data set. The technique that helps to know this about our model is known as Cross-Validation.

FYI: Free nlp online course!

In this article, we will get to know more about the different types of cross-validation techniques, pros, and cons of each technique. Let’s start with the definition of Cross-Validation.

Cross-Validation Techniques in Machine Learning

Cross-Validation is a resampling technique that helps to make our model sure about its efficiency and accuracy on the unseen data. It is a method for evaluating Machine Learning models by training several other Machine learning models on subsets of the available input data set and evaluating them on the subset of the data set.

We have different types of Cross-Validation techniques but let’s see the basic functionality of Cross-Validation: The first step is to divide the cleaned data set into K partitions of equal size.

1. Then we need to treat the Fold-1 as a test fold while the other K-1 as train folds and compute the score of the test-fold.
2. We need to repeat step 2 for all folds taking another fold as a test while remaining as a train.
3. Last step would be to take the average of scores of all the folds.

Read: Machine Learning Projects For Beginners

Types of Cross-Validation techniques in Machine Learning

1. Holdout Method

This technique works on removing a part of the training data set and sending that to a model that was trained on the rest of the data set to get the predictions. We then calculate the error estimation which tells how our model is doing on unseen data sets. This is known as the Holdout Method.

Pros

1. This Method is Fully independent of data.
2. This Method only needs to be run once so has lower computational costs.

Cons

1. The Performance is subject to higher variance given the smaller size of the data.

2. K-Fold Cross-Validation

In a Data-Driven World, there is never enough data to train your model, on top of that removing a part of it for validation poses a greater problem of Underfitting and we risk losing important patterns and trends in our data set, which in turn increases Bias. So ideally, we require a method that provides ample amounts of data for training the model and leaves ample amounts of data for validation sets.

In K-Fold cross-validation, the data is divided into k subsets or we can take it as a holdout method repeated k times, such that each time, one of the k subsets is used as the validation set and the other k-1 subsets as the training set. The error is averaged over all k trials to get the total efficiency of our model.

We can see that each data point will be in a validation set exactly once and will be in a training set k-1 time. This helps us reduce bias as we are using most of the data for fitting and reduces variance as most of the data is also being used in the validation set.

Pros

1. This will help to overcome the problem of computational power.
2. Models may not be affected much if an outlier is present in data.
3. It helps us overcome the problem of variability.

Cons

1. Imbalanced data sets will impact our model.

3. Stratified K-Fold Cross-Validation

K Fold Cross Validation technique will not work as expected for an Imbalanced Data set. When we have an imbalanced data set, we need a slight change to the K Fold cross validation technique, such that each fold contains approximately the same strata of samples of each output class as the complete. This variation of using a stratum in K Fold Cross Validation is known as Stratified K Fold Cross Validation.

Pros

1. It can improve different models using hyper-parameter tuning.
2. Helps us compare models.
3. It helps in reducing both Bias and Variance.

Also Read: Career in Machine Learning

 4. Leave-P-Out Cross-Validation

In this approach we leave p data points out of training data out of a total n data points, then n-p samples are used to train the model and p points are used as the validation set. This is repeated for all combinations, and then the error is averaged.

Pros

1. It has Zero randomness
2. The Bias will be lower

Cons

1. This method is exhaustive and computationally infeasible.

What are different cross validation techniques used for regression problems?

Curious about What are different cross validation techniques used for regression problems? 

Cross Validation is an essential tool for assessing your model’s classification accuracy. Logistic Regression, Random Forest, and SVM carry their own advantages and disadvantages to the models. Here is where cross validation comes in.

When you cross validate, you take a variety of test and training data, average it, and then assign a score to your model. In practice, we typically fit Logistic Regression, Random Forest, and SVM to each model, but what if the scores for each model are similar? A cross validation technique in machine learning can notify you which among your models is the most accurate. There are various cross validation methods, but we will focus on three today: Validation, the “leave one out” method, and K-Fold. To comprehend the importance of cross validation and when you should use which method, we must first understand the models and their respective drawbacks.

Logistic regression

This is a binary classification model. It is commonly used when there are only two classes, such as in medical identification. You either have cancer, or you don’t, and you either have or don’t have Congestive Heart Failure. This model has the advantage of being a simple regression that works well for what it does, which is again a binomial classifier. This model, however, is susceptible to overfitting. Overfitting occurs when a model is trained and adapts to all of its data points, including outliers. This means that the model has received training so well that the anticipated output does not match the model’s goals.

Random Forest

Random Forest’s major benefit is its ability to handle massive amounts of data with thousands of features. It resamples the data during its training model. This is called bootstrapping. To recap, Bootstrapping is when you utilize data for sampling and then return the data to the original data set for reuse. Also, keep in mind that only one-third of the data in Random Forest is used for testing. These are known as Out of Bag samples (OOB), and you should base your model on the Out of Bag Error. Random Forest, like Logistic Regression, is susceptible to Overfitting if your set of data is disruptive.

Support Vector Machines (SVM)

These are useful when you need to fit a reasonably precise model to the data. It performs effectively when the number of features outnumbers the number of actual samples (a data set with 1000 columns and 800 rows). This model is also useful when there are a lot of features (here, “features” are called “dimensions”). It also works well when the class separation is accurate and leaves no room for error. It is also less susceptible to overfitting. Its disadvantage is that it does not work well with large data sets. This takes a lot of time to train. The final model is also difficult to comprehend and interpret.

Below are the three main classifiers. But bear in mind that once you fit and train all three models, you must check with a cross validation technique in machine learning.

Simple Validation

You will test on 50% of the data and train the rest 50% of the model.

Leave One Out Cross Validation

This method identifies one data point from the set of data and trains the remaining data. It then loops through the entire data set, omitting one data point each time. The average is then used to calculate the score. This method will consume a lot of time and a large number of computer resources.

K-Fold

Finally, there is K-Fold. K-Fold cross validation requires a number, k, that indicates how and where to split your data. Let’s start with k = 5. As a result, the data is divided into five categories. The first 20% would be regarded as test data, while the remaining 80% would be regarded as train data. It then tends to take the next 20%, with the remaining 80% being test data and so on. It then computes a score by using the average of all five data sets.

These are the common cross validation techniques in regression models.

Importance of Cross Validation in Machine Learning

When your original validation partition doesn’t really represent the whole population, you get a model with a high degree of accuracy.

However, because it can only work with a specific set of data, it will be of little use in practice. The machine cannot recognize data that is outside of its scope, resulting in poor accuracy.

Cross validation in machine learning is used to test the accuracy of your model on multiple and diverse subsets of data. As a result, you must ensure that it extends effectively to the data. It improves the accuracy of the model.

Limitations of Cross Validation techniques

We learned about cross validation in machine learning and realized how important it is. It is an important element of machine learning, but it is not without limitations.

• In an ideal world, cross validation would produce precise and meaningful results. Nevertheless, the world is not without flaws. You never know what data the model will come across in the future.
• In predictive modeling, the framework under study generally changes over time. As a result, there may be discrepancies between both the training and validation sets. Assume you have such a model that forecasts stock prices. You used data from the previous five years to train the model. Is it reasonable to expect precise forecasts over the next five years?
• And here is another case where the limitations of cross validation processes are highlighted. You create a model that predicts an individual’s risk of developing a specific disease. However, you train the model with data from a study that included a subset of the population. When the model is applied to the general public, the results may vary significantly.

Application of cross validation techniques

• Cross validation can be used to compare the results of a set of predictive methods.
• It is extremely useful in medical research. Consider how we predict whether a cancer patient would then respond to a particular drug based on the expression levels of a specified number of proteins, let’s say 15. The best approach is to figure out which subset of the 15 features produces the best predictive model. You can use cross validation to identify the specific subset that produces the best results.
• In the area of medical statistics, data analysts have recently used cross validation. These procedures are beneficial in meta-analysis.

Cross Validation in Python

Python supports cross validation. It can help prevent overfitting and underfitting. Now let’s look at some cross validation applications in Python.

Overfitting occurs once the model is trained “too well.” It occurs when there is a complex model with a large number of variables in contrast to the number of observations.

In such cases, the model will perform admirably in training mode but may be inaccurate when applied to new data. It is because of the fact that it is not a simplistic model.

Underfitting, as opposed to overfitting, takes place when the model doesn’t really fit the training data. As a result, it cannot be generalized to new data.

It’s because you’re staring at a basic model with insufficient independent variables. It can also occur when fitting a linear model to non-linear data.

In data analysis, overfitting and underfitting are both undesirable. Always strive for a sensible approach or a model that is “just right.” Cross-validation can assist you in avoiding overfitting and underfitting.

These problems can be solved using the K-Fold Cross Validation and LOOCV processes. We’ve all seen how these procedures operate. The K-Fold CV method is depicted graphically below.

You have ten different outcomes. To determine the final accuracy figure, take the average of the ten results.

Machine learning necessitates extensive data analysis. Cross validation techniques are an excellent method for preparing the machine for real-world situations.

As a result, the machine is prepared to integrate new data and generalize it in order to make accurate predictions. Join the Machine Learning with Python Course to learn about machine learning and Python fundamentals.

Conclusion

In this article, we have learned about the importance of Validation of a Machine Learning Model in the Data Science Project Life Cycle, got to know what is validation and cross-validation, explored the different types of Cross-Validation techniques, got to know some advantages and disadvantages of those techniques.

If you’re interested to learn more about machine learning, check out IIIT-B & upGrad’s PG Diploma in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.