Model Validation and Importance of Stability in Logistic Regression

So, you've seen how a model is evaluated, using various parameters such as accuracy, sensitivity, KS statistic, gini coefficient, etc. However, so far you've only tested models on data that was formed after splitting one data set into training data and testing data. Is that enough? Let's see.

Let's again look at the telecom churn example from before. The data used to build the model was from 2014.

You split the original data into two parts, i.e. training and testing data. However, these two parts were both built with data from 2014.

This is called in-sample validation. Testing your model on this test data may not be enough though, as the test data here is too similar to training data.

So, it makes sense to actually test the model on data that is from some other time, like 2016. This is called out-of-time validation.

Another way to do the same thing is to use K-fold cross validation. Basically, the evaluation of the sample is done for k-iterations. E.g. here's a representation of how 3-fold cross validation works:

Basically, there are 3 iterations in which evaluation is done. In the first iteration, 1/3rd of the data is selected as training data and the remaining 2/3rd of it is selected as testing data. In the next iteration, a different 1/3rd of the data is selected as the training data set and then the model is built and evaluated. Similarly, the third iteration is completed.

Such an approach is necessary if the data you have for model building is very small, i.e. has very few data points.

If these three methods of validation are still unclear to you, you need not worry as of now. They will be covered at length in the module on Model Selection.

Note for 0:43 to 2:07: While explaining how WOE is linked to stability and PSI, Hindol used IV values mistakenly. He meant to use WOE values.

Obviously, a good model will be stable. A model is considered stable if it has:

1. Performance Stability: Results of in-sample validation approximately match those of out-of-time validation
2. Variable Stability: The sample used for model building hasn't changed too much and has the same general characteristics