Python is one of the most used programming languages among developers globally. Its capabilities of data automation and algorithms make it ideal for building and training programs, machines, and computer-based systems and making predictions. Therefore, candidates with Python skills are increasingly preferred for lucrative career paths, such as Machine Learning and Data Science.

If you are a beginner, finding the right tools on your own may seem daunting. Free software like Scikit-learn can empower you to pick up relevant skills with little effort. The machine learning library has several classifications, regression, and clustering algorithms for Python programmers. 

This blog will focus specifically on classification problems and sklearn metrics to guide you in your learning journey. You will learn about the application of evaluation metrics and also understand the mathematics behind them. 

Classification Metrics in Scikit-Learn

Classification is an integral part of predictive modelling. You use it to identify the class to which a particular sample from a population belongs. Suppose you want to predict whether a patient will be hospitalised again. The two possible types here are – Positive (Hospitalised) and Negative (Not Hospitalised). The classification model would predict the bucket where the sample should be placed, Predicted Positive or Predicted Negative. You will discover the accuracy of predictions once you train the model. 

Most data scientists and machine learning engineers use the Scikit-Learn package for analysing the performance of predictive models. The sklearn metrics module gives you access to many built-in functionalities. Let’s uncover the process of writing functions from scratch with these metrics. 

Join the Machine Learning Course online from the World’s top Universities – Masters, Executive Post Graduate Programs, and Advanced Certificate Program in ML & AI to fast-track your career.

Sklearn Metrics Explained

Sklearn metrics lets you implement scores, losses, and utility functions for evaluating classification performance. 

Here are the key steps involved:

• Load data;
• Split it into train set and test set;
• Build the training model;
• Make predictions or forecasts on the test data;
• Evaluate the machine learning model with a particular method.

To proceed, you will need to load a sample data set and prediction capabilities for two models, Random Forest and Linear Regression. Let’s call them Model_RF and Model_LR. 

Run this code to import the Pandas module and read the data file and inspect its elements. 

import pandas as ps

df = ps.read_csv(‘data.csv’)

df.head()

In most projects, you define a threshold and label the prediction probabilities as predicted positive and predicted negative. This would add two more columns to your table. 

thresh = 0.5

df[‘forecasted_RF’] = (df.model_RF >= 0.5).astype(‘int’)

df[‘forecasted_LR’] = (df.model_LR >= 0.5).astype(‘int’)

df.head()

Now that we have actual and forecasted labels, we can divide our samples into four different buckets.

With confusion_matrix, we can get a 2X2 array with the labels bifurcated into the following buckets:

• True Positive (TP)
• False Positive (FP)
• False Negative (FN)
• True Negative (TN)

After importing the confusion_matrix from sklearn metrics and passing the actual and forecasted labels, you can define your functions to verify it. 

You can also check if your results match manually using Python’s assert function and NumPy’s array_equal function.

We can calculate many other performance metrics using the four buckets of TP, FP, TN, and FN. These are:

1. accuracy_score

It takes the actual and forecasted labels as inputs and produces the fraction of samples predicted correctly.

2. recall_score

It gives the fraction of positive events predicted correctly. The recall is also known as sensitivity.

3. precision_score

It shows the fraction of predicted positive events that are positive.

After calculating all these metrics, suppose you find the RF model better at recall and precision. The choice here would be easy. But what if the LR model was better at recall and the RF model was better at precision? In this case, you would need another method called the F1 score. 

4. f1_score

It is the harmonic mean of recall and precision. The model with the higher score is considered the better option.

The above metrics have been calculated with a defined threshold of 0.5. One may wonder if a change in this threshold would change the performance metrics as well. The answer? Yes, it will. 

We have another way of assessing a model without picking a threshold, i.e. Receiver Operating Characteristic (ROC) curves. Scikit-learn also has built-in functions for analysing them.

The roc_curve and roc_auc_score functions take the actual labels and forecasted probabilities as inputs. 

• roc_curve

It returns three lists, namely thresholds (unique forecasted probabilities in descending order), FPR (the false-positive rates), and TPR (the true positive rates).

• roc_auc_curve

It finds the areas under the curve for both RF and LR models.

You can determine the better performance metric once you plot the ROC curve and add the AUC to the legends. 

Summing Up

In predictive analytics, you can choose from a variety of metrics. Accuracy, recall, precision, f1, and AUC are some of the popular scores. 

Some may prefer defining a threshold and using performance metrics like accuracy, recall, precision, and f1 scores. Others may like to use AUC to analyse a model’s performance as it does not require threshold selection. In the end, you should go for the metric that best suits the business problem at hand. 

With this, we have given you an overview of sklearn metrics. You can use this information to clarify the basics of python programming and keep learning with online courses. You can also undertake project work to practice and refine your skills. Programmes like upGrad’s Master of Science in Machine Learning & Artificial Intelligence can help with both.

The curriculum familiarises you with the complete data science toolkit and covers practical aspects of Scikit-Learn and other software. Additionally, credentials from reputed institutes like the Liverpool John Moores University and IIIT Bangalore set you apart from the competition in job applications and placement interviews.