How to Choose a Feature Selection Method for Machine Learning

Feature Selection Introduction

Lots of features are used by a machine learning model of which only a few of them are important. There is a reduced accuracy of the model if unnecessary features are used to train a data model. Further, there is an increase in the complexity of the model and a decrease in the Generalization capability resulting in a biased model. The saying “sometimes less is better” goes well with the concept of machine learning. The problem has been faced by a lot of users where they find it difficult to identify the set of relevant features from their data and ignore all the irrelevant sets of features. The less important features are termed so as they don’t contribute to the target variable.

Therefore, one of the important processes is feature selection in machine learning. The goal is to select the best possible set of features for the development of a machine learning model. There is a huge impact on the performance of the model by the feature selection. Along with data cleaning, feature selection should be the first step in a model design. 

Feature selection in Machine Learning may be summarized as

• Automatic or manual selection of those features that are contributing most to the prediction variable or the output.
• The presence of irrelevant features might lead to a decreased accuracy of the model as it will learn from irrelevant features.

Benefits of Feature Selection

• Reduces overfitting of data: a less number of data leads to lesser redundancy. Therefore there are fewer chances of making decisions on noise.
• Improves accuracy of the model: with lesser chance of misleading data, the accuracy of the model is increased.
• Training time is reduced: removal of irrelevant features reduces the algorithm complexity as only fewer data points are present. Therefore, the algorithms train faster.
• The complexity of the model is reduced with better interpretation of the data.

Supervised and Unsupervised methods of feature selection 

The main objective of the feature selection algorithms is to select out a set of best features for the development of the model. Feature selection methods in machine learning can be classified into supervised and unsupervised methods.

1. Supervised method: the supervised method is used for the selection of features from labeled data and also used for the classification of the relevant features. Hence, there is increased efficiency of the models that are built up.
2. Unsupervised method: this method of feature selection is used for the unlabeled data.

List of Methods Under Supervised Methods

Supervised methods of feature selection in machine learning can be classified into

1. Wrapper Methods

This type of feature selection algorithm evaluates the process of performance of the features based on the results of the algorithm. Also known as the greedy algorithm, it trains the algorithm using a subset of features iteratively. Stopping criteria are usually defined by the person training the algorithm. The addition and removal of features in the model take place based on the prior training of the model. Any type of learning algorithm can be applied in this search strategy. The models are more accurate compared to the filter methods. 

Techniques used in Wrapper methods are:

1. Forward selection: The forward selection process is an iterative process where new features that improve the model are added after each iteration. It starts with an empty set of features. The iteration continues and stops until a feature is added that doesn’t further improve the performance of the model.
2. Backward selection/elimination: The process is an iterative process that starts with all the features. After each iteration, the features with the least significance are removed from the set of initial features.  The stopping criterion for the iteration is when the performance of the model doesn’t improve further with the removal of the feature. These algorithms are implemented in the mlxtend package.
3. Bi-directional elimination: Both methods of forward selection and backward elimination technique are applied simultaneously in the Bi-directional elimination method to reach one unique solution.
4. Exhaustive feature selection: It is also known as the brute force approach for the evaluation of feature subsets. A set of possible subsets are created and a learning algorithm is built for each subset. That subset is chosen whose model gives the best performance.
5. Recursive Feature elimination (RFE): The method is termed to be greedy as it selects features by recursively considering the smaller and smaller set of features. An initial set of features are used for training the estimator and their importance is obtained using feature_importance_attribute. It is then followed through the removal of the least important features leaving behind only the required number of features. The algorithms are implemented in the scikit-learn package.

Figure 4: An example of code showing the recursive feature elimination technique

2. Embedded methods

The embedded feature selection methods in machine learning have a certain advantage over the filter and wrapper methods by including feature interaction and also maintaining a reasonable computational cost. Techniques used in embedded methods are:

1. Regularization: Overfitting of data is avoided by the model by adding a penalty to the parameters of the model. Coefficients are added with the penalty resulting in some coefficients to be zero. Therefore those features that have a zero coefficient are removed from the set of features. The approach of feature selection uses Lasso (L1 regularization) and Elastic nets (L1 and L2 regularization).
2. SMLR (Sparse Multinomial Logistic Regression): The algorithm implements a sparse regularization by ARD prior (Automatic relevance determination) for the classical multinational logistic regression. This regularization estimates the importance of each feature and prunes the dimensions which are not useful for the prediction. Implementation of the algorithm is done in SMLR.
3. ARD (Automatic Relevance Determination Regression): The algorithm will shift the coefficient weights towards zero and is based on a Bayesian Ridge Regression. The algorithm can be implemented in scikit-learn.
4. Random Forest Importance: This feature selection algorithm is an aggregation of a specified number of trees. Tree-based strategies in this algorithm rank on the basis of increasing the impurity of a node or decreasing the impurity (Gini impurity).  The end of the trees consists of the nodes with the least decrease in impurity and the start of the trees consists of nodes with the greatest decrease in impurity. Therefore, important features can be selected out through pruning of the tree below a particular node.

3. Filter methods

The methods are applied during the pre-processing steps. The methods are quite fast and inexpensive and work best in the removal of duplicated, correlated, and redundant features. Instead of applying any supervised learning methods, the importance of features is evaluated based on their inherent characteristics. The computational cost of the algorithm is lesser compared to the wrapper methods of feature selection. However, if enough data is not present to derive the statistical correlation between the features, the results might be worse than the wrapper methods. Therefore, the algorithms are used over high dimensional data, which would lead to a higher computational cost if wrapper methods are to be applied. 

Techniques used in the Filter methods are:

1. Information Gain:  Information gain refers to how much information is gained from the features to identify the target value. It then measures the reduction in the entropy values. Information gain of each attribute is calculated considering the target values for feature selection.
2. Chi-square test: The Chi-square method (X2) is generally used to test the relationship between two categorical variables. The test is used to identify if there is a significant difference between the observed values from different attributes of the dataset to its expected value. A null hypothesis states that there is no association between two variables.

Source

The formula for Chi-square test 

Implementation of Chi-Squared algorithm: sklearn, scipy

 An example of code for Chi-square test

Source

4. CFS (Correlation-based feature selection): The method follows “Features are relevant if their values vary systematically with category membership.” Implementation of CFS (Correlation-based feature selection): scikit-feature

Join the AI & ML Courses online from the World’s top Universities – Masters, Executive Post Graduate Programs, and Advanced Certificate Program in ML & AI to fast-track your career.

5. FCBF (Fast correlation-based filter): Compared to the above-mentioned methods of Relief and CFS, the FCBF method is faster and more efficient. Initially, the computation of Symmetrical Uncertainty is carried out for all features. Using these criteria, the features are then sorted out and redundant features are removed.

Symmetrical Uncertainty= the information gain of x | y divided by the sum of their entropies. Implementation of FCBF: skfeature

6. Fischer score: Fischer ration (FIR) is defined as the distance between the sample means for each class per feature divided by their variances. Each feature is independently selected according to their scores under the Fisher criterion. This leads to a suboptimal set of features. A larger Fisher’s score denotes a better-selected feature.

Source

The formula for Fischer score

Implementation of Fisher score: scikit-feature

The output of the code showing Fisher score technique

Source

Pearson’s Correlation Coefficient: It is a measure of quantifying the association between the two continuous variables. The values of the correlation coefficient range from -1 to 1 which defines the direction of relationship between the variables.

7. Variance Threshold: The features whose variance doesn’t meet the specific threshold are removed. Features having zero variance are removed through this method. The assumption considered is that higher variance features are likely to contain more information.

Figure 15: An example of code showing the implementation of Variance threshold

8. Mean Absolute Difference (MAD): The method calculates the mean absolute

difference from the mean value.

An example of code and its output showing the implementation of Mean Absolute Difference (MAD) 

Source

9. Dispersion Ratio: Dispersion ratio is defined as the ratio of the Arithmetic mean (AM) to that of Geometric mean (GM) for a given feature. Its value ranges from +1 to ∞ as AM ≥ GM for a given feature. 

A higher dispersion ratio implies a higher value of Ri and therefore a more relevant feature. Conversely, when Ri is close to 1, it indicates a low relevance feature.

1. Mutual Dependence: The method is used to measure the mutual dependence between two variables. Information obtained from one variable may be used to obtain information for the other variable.
2. Laplacian Score: Data from the same class are often close to each other. The importance of a feature can be evaluated by its power of locality preservation. Laplacian Score for each feature is calculated. The smallest values determine important dimensions. Implementation of Laplacian score: scikit-feature.

Conclusion

Feature selection in the machine learning process can be summarized as one of the important steps towards the development of any machine learning model. The process of the feature selection algorithm leads to the reduction in the dimensionality of the data with the removal of features that are not relevant or important to the model under consideration. Relevant features could speed up the training time of the models resulting in high performance.

If you’re interested to learn more about machine learning, check out IIIT-B & upGrad’s Executive PG Program in Machine Learning & AI which is designed for working professionals and offers 450+ hours of rigorous training, 30+ case studies & assignments, IIIT-B Alumni status, 5+ practical hands-on capstone projects & job assistance with top firms.