Feature Engineering for Machine Learning: Process, Techniques, and Examples

Feature engineering involves transforming raw, often unstructured data, such as text or images, into structured features that improve model accuracy. Without proper feature engineering, your model might not perform well or even fail to learn effectively. 

This article will walk you through key techniques and share examples of feature engineering in machine learning. By the end, you'll have a clear understanding of key feature engineering techniques and how to apply them to improve model performance.

What is Feature Engineering for Machine Learning? An Overview

Feature engineering for machine learning is the process of transforming raw data into useful features that help your model better understand and make predictions. 

At its core, feature engineering is about selecting, modifying, or creating new variables (features) from your dataset to improve the learning process. These features are the input your model uses to make predictions. 

For example, in a customer dataset, features like 'age,' 'income,' and 'purchase history' help the model detect buying trends, predict future purchases, and segment customers for targeted marketing campaigns. 

But it’s not just about having features; it’s about choosing the right ones and transforming them into a useful format.

Feature Engineering vs. Data Preprocessing

While they may sound similar, feature engineering and data preprocessing serve different purposes:

• Data Preprocessing focuses on preparing raw data for modeling. This includes cleaning, handling missing values, and normalizing data.
• Feature engineering involves creating new features or modifying existing ones to make them more useful for the model. It requires creativity and domain knowledge to craft features that directly influence a model's ability to identify meaningful patterns.

Understanding the importance of feature engineering is key to improving your model’s performance. Let’s break it down.

Importance of Feature Engineering in Machine Learning

Raw data often comes with a variety of challenges that can hinder your model's performance. These issues, such as noise, missing values, and irrelevant data, can confuse machine learning algorithms and make it harder for them to identify meaningful patterns. 

Here's why transforming raw data into well-engineered features is crucial:

• Noise: Raw data is often noisy, meaning it contains random, irrelevant information. Feature engineering helps reduce this noise by selecting and creating features that focus on what matters most for the model.
• Missing Values: Many datasets have missing values, which can lead to inaccurate results if not handled properly. Through feature engineering, you can impute or create new features to fill in gaps and keep your model’s predictions reliable.
• Irrelevant Data: Raw data might contain irrelevant features that don’t contribute to the model’s prediction. Feature engineering allows you to remove these features or transform them into something more useful.

By transforming raw data into usable features, you enable the model to focus on important patterns, which enhances its learning process. This leads to:

• Improved Accuracy: With the right features, models can more easily identify patterns, improving prediction accuracy.
• Better Generalization: Well-engineered features help your model generalize better to new, unseen data, making it more robust and reliable in real-world scenarios.

Now that you see why it matters, let’s break down the key steps in making feature engineering work for your model.

How Does Feature Engineering in ML Work? Key Steps

To make the most of your data, there are key steps you should follow, each contributing to improved model accuracy and efficiency. Let’s walk through each step in detail:

Step 1: Data Collection and Cleaning

The first step is collecting the data and cleaning it to ensure it’s accurate and useful. Raw data often contains missing values, duplicates, or errors.

• Example: Imagine you’re working with a dataset on house prices. Some rows might have missing values for square footage or number of rooms. You'll need to fill in these missing values or remove entries with too much missing data.
• Tip: Before filling in missing values, explore the dataset to understand the missing data pattern. Sometimes, dropping rows with too many missing values can help prevent bias.

Also Read: Data Cleaning Techniques: Learn Simple & Effective Ways To Clean Data

Step 2: Feature Selection

Once your data is cleaned, it's time to select the features that matter most. Feature selection helps you focus on the variables that directly affect your model’s output.

• Example: When predicting house prices, "neighborhood" and "square footage" are key, while "postal code" or "homeowner's political preference" may not be helpful.
• Tip: Use correlation analysis to identify features that are highly correlated with your target variable. This can help you narrow down to the most relevant features, which speeds up the model and reduces complexity.

Step 3: Feature Transformation

This step involves transforming your features to make them more useful for the model. Common transformations include normalization, encoding, and converting features to a different scale or format.

• Example: If your dataset includes “date of purchase,” you can create new features like “day of the week” or “month” to capture trends or patterns.
• Tip: Consider using techniques like one-hot encoding or label encoding when working with categorical data. But be mindful of high cardinality—too many categories can increase complexity.

Step 4: Feature Creation

Feature creation involves making new features from the existing ones. By doing this, you can introduce relationships that the model might not capture on its own.

• Example: In a customer dataset, you could create a new feature called "spending per visit," which is calculated by dividing "total spending" by "number of visits."
• Tip: Be strategic in feature creation. Creating too many features is easy, but remember that more features don’t always lead to better performance. Focus on features that truly add value.

Step 5: Feature Scaling

Many machine learning algorithms perform better when features are on a similar scale. Feature scaling involves normalizing or standardizing the features so they contribute equally to the model.

• Example: If “square footage” ranges from 100 to 10,000 and “year built” ranges from 1800 to 2022, scaling these features helps the model treat them equally.
• Tip: Use StandardScaler (for standardization) or MinMaxScaler (for normalization) when scaling features, but test both methods. The choice of scaler can influence model performance depending on the algorithm you're using.

Also Read: The Data Science Process: Key Steps to Build Data-Driven Solutions

Let’s look at how feature engineering is critical throughout the machine learning lifecycle.

Role of Feature Engineering in the Machine Learning Lifecycle

Feature engineering for machine learning isn’t just a one-time task. It’s a continuous process that evolves throughout the entire machine learning lifecycle. As you gather more data or retrain your model, feature engineering in ML adapts to ensure your model keeps improving.

Here’s how feature engineering fits into the lifecycle:

Adapting Features as Data Changes

Suggestion: "As your data evolves, the features used by your model should also adapt to maintain or improve accuracy. The relationships in the data may shift, and new features might be more relevant to the model’s performance.

• Example: In the housing market, seasonal trends or local economic shifts can affect housing prices. As these trends change, so should the features you use to model prices, like adding "seasonal demand" or adjusting for "economic inflation."
• Tip: Regularly monitor the model’s performance. If accuracy drops after changes in the data, consider revisiting your features and updating them based on the new patterns you observe.

Keeping feature engineering a dynamic, ongoing part of your machine learning workflow ensures that your model stays relevant and accurate over time. 

Also Read: A Comprehensive Guide to the Data Science Life Cycle: Key Phases, Challenges, and Future Insights

Next, let’s look at some of the most effective techniques and tools used for feature engineering in ML.

Popular Techniques and Tools for Feature Engineering in ML

Several common techniques in feature engineering for machine learning help you prepare your data for optimal model training. Each technique uniquely addresses specific issues like missing data, outliers, and skewed distributions. Here’s an overview of some key techniques:

1. Imputation

Imputation is used to handle missing values in your dataset. Instead of discarding rows with missing data, imputation fills in these gaps using various strategies, such as replacing missing values with the mean, median, or mode of the column.

• Example: If you have a dataset with missing values in the "age" column, you can replace those missing values with the median age of the other records.
• Tip: Imputation can significantly improve model performance by ensuring the dataset is complete and usable without losing important data.

2. Handling Outliers

Outliers are values that differ significantly from other observations in your data, and they can skew your model’s results. Identifying and handling outliers is important to prevent them from negatively impacting model accuracy.

• Example: If you're predicting salaries based on years of experience, a salary of INR 10LPA might be an outlier that needs to be handled.
• Tip: You can either remove outliers, cap them at a certain threshold, or transform them to reduce their influence on the model.

3. Log Transform

A log transformation is useful when your data is skewed, meaning it has extreme values or a long tail. Applying a log transformation helps to make the distribution more normal, improving the model’s ability to learn from the data.

• Example: If you have a feature like "income" with large values, applying a log transform will compress the range and make the data more manageable for the model.
• Tip: Log transformations work well when the feature’s distribution is highly skewed to the right (positively skewed).

4. Binning

Binning groups continuous data into discrete categories or "bins." This technique simplifies the data by reducing the number of distinct values, making it easier for the model to interpret.

• Example: If you have continuous data for "age," you can bin it into age groups like "0-18," "19-35," "36-50," etc.
• Tip: Binning can be useful when the relationship between the feature and target variable is non-linear.

5. Feature Split

Feature split involves breaking a single feature into multiple components. This is particularly useful when you have combined features that contain multiple pieces of information.

• Example: If you have a "full address" feature, you can split it into "street," "city," and "zip code."
• Tip: Feature split can reveal hidden patterns in the data and help the model learn more specific relationships.

6. One-Hot Encoding

One-hot encoding is used for categorical data, converting each category into a separate binary feature (0 or 1). This helps the model understand the different levels of categorical variables without assigning any arbitrary order.

• Example: For a "color" feature with values like "red," "blue," and "green," one-hot encoding would create three new columns: "is_red," "is_blue," and "is_green."
• Tip: One-hot encoding is ideal for nominal categorical variables, but be careful when dealing with high-cardinality features as it can lead to an explosion in the number of features.

Let’s now turn to the tools that can make feature engineering in ML a whole lot easier.

Essential Tools for Feature Engineering in Machine Learning

Feature engineering can be complex, but with the right tools, it becomes much easier to manage. Below are some of the most popular libraries that automate and streamline the feature engineering process, making your tasks more efficient: 

Understanding and using the right libraries is crucial for effective feature engineering in machine learning. 

Now, let’s see how these feature engineering techniques are applied in real-world scenarios to solve actual challenges.

Real-World Examples of Feature Engineering in Machine Learning

Feature engineering refines raw data into features that boost model performance. For example, creating new variables like "price per square foot" or converting categorical data like "furnishing status" into numerical values helps the model better interpret key relationships. 

These techniques allow the model to make more accurate predictions and generalize better, ultimately improving its overall performance.

Let’s dive into detailed examples of how feature engineering works in this context.

Example 1: Feature Engineering in a Regression Problem

In this example, we will apply feature engineering to a dataset used for predicting house prices. The raw dataset contains information like the square footage of the house, the number of rooms, and the location. Our goal is to improve model performance by applying different feature engineering techniques.

Dataset: Predicting House Prices

The dataset includes the following columns:

• price: The price of the house (target variable).
• area: The area (in square feet) of the house.
• bedrooms: The number of bedrooms in the house.
• bathrooms: The number of bathrooms in the house.
• stories: The number of stories the house has.
• mainroad: Whether the house is located on a main road ("yes" or "no").
• guestroom: Whether the house has a guestroom ("yes" or "no").
• basement: Whether the house has a basement ("yes" or "no").
• hotwaterheating: Whether the house has hot water heating ("yes" or "no").
• airconditioning: Whether the house has air conditioning ("yes" or "no").
• parking: The number of parking spaces available.
• prefarea: Whether the house is located in a preferred area ("yes" or "no").
• furnishingstatus: The furnishing status of the house (e.g., "furnished", "semi-furnished").

Let's walk through the feature engineering steps applied to this dataset:

1. Creating a New Feature (Price per Square Foot): The area of the house is an important feature, but we can improve it by calculating the price per square foot. This new feature will help the model better understand the relationship between the house's price and size.
2. Encoding Categorical Variables: Many of the features are categorical (e.g., "mainroad", "guestroom", "furnishingstatus"). We need to convert these into numerical values that the machine learning model can use.
3. Handling Missing Values: In real-world datasets, some values might be missing. We need to handle missing data appropriately to ensure the model can work with the complete dataset.
4. Applying Feature Scaling: Features like "area" and "price" are on very different scales. Scaling these features ensures that the model treats all features equally and doesn't overemphasize features with larger values.

Let’s start by performing these steps using Python and some common libraries like Pandas and Scikit-learn. 

# Step 1: Create a new feature 'price_per_sqft'
housing_data['price_per_sqft'] = housing_data['price'] / housing_data['area']

# Step 2: Encode categorical variables
# Convert "yes"/"no" to 1/0 for binary features
binary_columns = ['mainroad', 'guestroom', 'basement', 'hotwaterheating', 'airconditioning', 'prefarea']
housing_data[binary_columns] = housing_data[binary_columns].applymap(lambda x: 1 if x == 'yes' else 0)

# Convert 'furnishingstatus' into numerical values (e.g., 'furnished' = 2, 'semi-furnished' = 1, 'unfurnished' = 0)
housing_data['furnishingstatus'] = housing_data['furnishingstatus'].map({'furnished': 2, 'semi-furnished': 1, 'unfurnished': 0})

# Step 3: Handle missing values
# In this case, let's fill missing values in 'parking' with the median value of the column
housing_data['parking'].fillna(housing_data['parking'].median(), inplace=True)

# Step 4: Apply feature scaling to 'area' and 'price'
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
housing_data[['area', 'price']] = scaler.fit_transform(housing_data[['area', 'price']])

# View the updated dataset
housing_data.head()

Code Explanation: 

Here’s what each part of the code does:

1. Creating the "price_per_sqft" Feature:

The line 

housing_data['price_per_sqft'] = housing_data['price'] / housing_data['area'] 

calculates the price per square foot for each house by dividing the price by the area. This new feature helps the model understand the relationship between the house’s price and its size.

2. Encoding Categorical Variables:

The lines that encode "yes"/"no" values into 1/0 (for features like mainroad, guestroom, basement, etc.) convert categorical data into numerical data. This transformation is essential for machine learning models since they require numerical input. We used applymap(lambda x: 1 if x == 'yes' else 0) to perform this task.

3. Converting "furnishingstatus":

The line 

housing_data['furnishingstatus'] = housing_data['furnishingstatus'].map({'furnished': 2, 'semi-furnished': 1, 'unfurnished': 0}) 

maps the "furnishingstatus" categories into numerical values: "furnished" becomes 2, "semi-furnished" becomes 1, and "unfurnished" becomes 0. This is another encoding step for categorical data.

4. Handling Missing Values in "Parking":

The line housing_data['parking'].fillna(housing_data['parking'].median(), inplace=True) 

fills any missing values in the "parking" column with the median value of the existing "parking" data. This ensures that we don’t lose valuable data or rows due to missing values.

5. Feature Scaling for "Area" and "Price":

The line scaler = StandardScaler() initializes a standard scaler, and 

housing_data[['area', 'price']] = scaler.fit_transform(housing_data[['area', 'price']]) 

standardizes the "area" and "price" columns by transforming them to have a mean of 0 and a standard deviation of 1. This is important because machine learning models, especially those that rely on distances (like KNN or SVM), perform better when features are on a similar scale.

Output: 

After applying the feature engineering steps, here’s a preview of the transformed dataset:

Output Explanation: 

Here’s a breakdown of what each step does:

• Price per Square Foot: We create a new feature by dividing the price of the house by the area to calculate the price per square foot. This gives us a better understanding of price relative to size.
• Encoding Categorical Variables: We convert the categorical variables like "yes"/"no" into binary values (1/0) so that the machine learning model can process them. For the "furnishingstatus" column, we assign numerical values to the categories, where "furnished" gets a value of 2, "semi-furnished" gets 1, and "unfurnished" gets 0.
• Handling Missing Values: We fill any missing values in the "parking" column with the median value, ensuring that the dataset remains intact without dropping rows or columns.
• Feature Scaling: We scale the "area" and "price" features to have a mean of 0 and a standard deviation of 1. This ensures that features with larger ranges (like price) don't dominate the learning process.

Let’s look at another example to see how feature engineering applies in a classification problem.

Example 2: Feature Engineering in a Classification Problem

In this example, we'll apply feature engineering to a spam email detection dataset. The raw data includes email content, sender details, and timestamps. Our goal is to preprocess and create features that improve model performance for classifying emails as "spam" or "ham" (non-spam).

Dataset: Spam Email Detection

The dataset includes the following columns:

• label: Whether the email is "spam" or "ham".
• text: The actual content of the email.
• label_num: A numerical representation of the label (0 for "ham", 1 for "spam").
• Unnamed: 0: An unnecessary index column that we’ll discard. 

Here’s a breakdown of the feature engineering steps we'll apply:

1. Text Processing:
   • Emails need to be cleaned and preprocessed. This includes removing stopwords, stemming words, and tokenizing the text into individual words.
2. Creating New Features:
   • We will create features like "email length" and the "frequency of specific words" (e.g., "free", "win") to capture relevant patterns associated with spam emails. 

Let's perform the necessary feature engineering steps using Python: 

# Step 1: Text Processing
import re
from sklearn.feature_extraction.text import CountVectorizer
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer

# Removing unwanted characters and stopwords
stop_words = set(stopwords.words('english'))
stemmer = PorterStemmer()

def preprocess_text(text):
    # Remove special characters and digits
    text = re.sub(r'[^a-zA-Z\s]', '', text)
    # Tokenization and stopword removal
    tokens = text.lower().split()
    tokens = [stemmer.stem(word) for word in tokens if word not in stop_words]
    return " ".join(tokens)

# Apply text preprocessing to the email content
spam_data['cleaned_text'] = spam_data['text'].apply(preprocess_text)

# Step 2: Creating New Features
# Create a feature for email length
spam_data['email_length'] = spam_data['cleaned_text'].apply(lambda x: len(x.split()))

# Create a feature for frequency of the word "free" in the email
spam_data['free_word_count'] = spam_data['cleaned_text'].apply(lambda x: x.count('free'))

# Step 3: Vectorizing the text data (using bag of words)
vectorizer = CountVectorizer(max_features=500)  # Limit to top 500 most frequent words
X_text = vectorizer.fit_transform(spam_data['cleaned_text']).toarray()

# Combine the text features with the new numerical features
import numpy as np
X_final = np.hstack((X_text, spam_data[['email_length', 'free_word_count']].values))

# Display the updated dataset
spam_data.head()

Code Explanation:

Here’s what each part of the code does:

1. Text Processing:
   • Remove Special Characters: The re.sub(r'[^a-zA-Z\s]', '', text) line removes any non-alphabetic characters (like digits or punctuation) from the email text. This step helps focus on the meaningful words.
   • Tokenization and Stopword Removal: We split the text into tokens (words) using text.lower().split(). We also remove common stopwords (like "the", "is", "in") using stopwords.words('english') from the NLTK library. This reduces the noise in the data.
   • Stemming: Each word is then reduced to its base form (e.g., "running" becomes "run") using the PorterStemmer (stemmer.stem(word)), which helps treat variations of words as the same feature.
2. Creating New Features:
   • Email Length: The line spam_data['email_length'] = spam_data['cleaned_text'].apply(lambda x: len(x.split())) calculates the length of each email by counting the number of words in the cleaned text. This feature helps identify if longer emails are more likely to be spam.
   • Frequency of "Free": The line spam_data['free_word_count'] = spam_data['cleaned_text'].apply(lambda x: x.count('free')) counts the occurrences of the word "free" in each email. Spam emails often contain such trigger words, so tracking this can be useful.
3. Vectorizing the Text:
   • The CountVectorizer is used to convert the cleaned text into numerical features. The line vectorizer.fit_transform(spam_data['cleaned_text']).toarray() converts the text into a matrix of token counts, where each feature corresponds to the presence of a word in the email.
   • We limit the features to the top 500 most frequent words (max_features=500) to avoid an overwhelming number of features and ensure the model focuses on the most relevant words.
4. Combining Text Features with Numerical Features:
   • The line X_final = np.hstack((X_text, spam_data[['email_length', 'free_word_count']].values)) combines the output from the CountVectorizer (text features) with the new features like "email length" and "free word count" into one final feature matrix (X_final).

Output:

Here’s a preview of the updated dataset after applying the feature engineering steps:

Output Explanation:

• cleaned_text: This column shows the preprocessed version of the email text after stopword removal, stemming, and tokenization.
• email_length: This new feature shows the number of words in each email. For example, the email in row 1 has 6 words after preprocessing.
• free_word_count: This feature counts how many times the word "free" appears in the email. If the email contains "free", it gets a higher count.
• Vectorized Text Data: The transformed email content has been converted into numerical data based on the most frequent words, which will be used for training machine learning models.

These steps not only enhance the accuracy of predictions but also make the model more robust and adaptable to new data.

Also Read: 5 Breakthrough Applications of Machine Learning

Let’s now explore the benefits feature engineering brings to machine learning, along with the challenges that come with it.

Benefits and Challenges of Feature Engineering in Machine Learning

Feature engineering brings several key benefits to machine learning models. Let’s take a look at how it improves model performance and efficiency:

However, while it offers significant advantages, it also comes with some challenges that need to be carefully managed. 

Also Read: 15 Key Techniques for Dimensionality Reduction in Machine Learning

As the field of machine learning evolves, the ability to engineer the right features will continue to be a key differentiator between average and exceptional models.

How Can upGrad Support Your Machine Learning Journey?

With a vast community of learners and access to over 10 million users worldwide, upGrad provides the tools to deepen your knowledge of feature engineering and data manipulation. 

Whether you're just starting out or looking to refine your skills, these courses provide valuable knowledge and tools. They will help you master the techniques needed to elevate your machine learning models.

Here are some of the top courses: 

• Executive PG Diploma in Data Science & AI
• Executive Diploma in Machine Learning and AI with IIIT-B
• Post Graduate Certificate in Machine Learning and Deep Learning (Executive)
• Post Graduate Certificate in Machine Learning & NLP (Executive)
• Analyzing Patterns: Data Storytelling Course (Free) 

For personalized career guidance, consult upGrad’s expert counselors or visit our offline centers to find the best course tailored to your goals!

Expand your expertise with the best resources available. Browse the programs below to find your ideal fit in Best Machine Learning and AI Courses Online.

Best Machine Learning and AI Courses Online

Discover in-demand Machine Learning skills to expand your expertise. Explore the programs below to find the perfect fit for your goals.

In-demand Machine Learning Skills

Discover popular AI and ML blogs and free courses to deepen your expertise. Explore the programs below to find your perfect fit.

Popular AI and ML Blogs & Free Courses