Data Cleaning and Preparation - II in Logistic Regression

You’ve merged your dataframes and handled the categorical variables present in them. But you still need to check the data for any outliers or missing values and treat them accordingly. Let's get this done as well.

You saw that one of the columns, i.e. 'TotalCharges' had 11 missing values. Since this isn't a big number compared to the number of rows present in a dataset, we decided to drop them since we won't lose much data.

Now that you have completely prepared your data, you can start with the preprocessing steps. As you might remember from the previous module, you first need to split the data into train and test sets and then rescale the features. So let’s start with that.

Recall that, for continuous variables, Rahim scaled the variables to standardise the three continuous variables — tenure, monthly charges and total charges. Recall that scaling basically reduces the values in a column to within a certain range — in this case, we have converted the values to the Z-scores.

For example, let’s say that, for a particular customer, tenure = 72. After standardising, the value of scaled tenure becomes:

$\frac{72-32.4}{24.6}=1.61$

because for the variable tenure, mean($\mu$) = 32.4 and standard deviation($\sigma$) = 24.6.

The variables had these ranges before standardisation:

• Tenure = 1 to 72
• Monthly charges = 18.25 to 118.80
• Total charges = 18.8 to 8685

After standardisation, the ranges of the variables changed to:

• Tenure = -1.28 to +1.61
• Monthly charges = -1.55 to +1.79
• Total charges = -0.99 to 2.83

Clearly, none of the variables will have a disproportionate effect on the model’s results now.

Churn Rate and Class Imbalance

Another thing to note here was the Churn Rate which Rahim talked about at the end of the video. You saw that the data has almost 27% churn rate. Checking the churn rate is important since you usually want your data to have a balance between the 0s and 1s (in this case churn and not-churn). 

The reason for having a balance is simple. Let’s do a simple thought experiment - if you had a data with, say, 95% not-churn (0) and just 5% churn (1), then even if you predict everything as 0, you would still get a model which is 95% accurate (though it is, of course, a bad model). This problem is called class-imbalance and you'll learn to solve such cases later.

Fortunately, in this case, we have about 27% churn rate. This is neither exactly 'balanced' (which a 50-50 ratio would be called) nor heavily imbalanced. So we'll not have to do any special treatment for this dataset.