Bagging vs Boosting in Machine Learning: Difference Between Bagging and Boosting

“Which ensemble technique improves model accuracy the most: Bagging or Boosting?”

As machine learning applications grow, so do the strategies for improving model accuracy. Two of the most popular techniques in ensemble learning are bagging and boosting. Both methods combine multiple models to build a more accurate and reliable final model, yet they follow distinct paths to achieve this.

Why use ensemble methods? These techniques combine multiple "weak" models (ones that perform slightly better than random guessing) into a "strong" model that’s capable of high accuracy. Bagging and boosting are two primary methods for achieving this. 

Bagging reduces variance by training models in parallel, while boosting reduces bias by training models in sequence, correcting errors at each step. Both are widely used to refine prediction accuracy and reduce errors in machine learning models. You ca also learn about the various machine learning project ideas here.

This blog will break down bagging and boosting, comparing how they work, when to use each, and how they impact model performance. Whether you're a data scientist refining models or a student curious about ensemble methods, understanding bagging vs. boosting will help you choose the right approach to get the most precise results.

Advance Your Career with Machine Learning Courses – Study online with top universities. Choose from Masters, Executive PG Programs, or Advanced Certificates in ML & AI, all designed to fast-track your career.

What is an Ensemble Method?

In machine learning, ensemble methods combine multiple "weak" models to create a stronger, more accurate model. The main goal of ensemble methods is to improve the model's accuracy by balancing bias and variance, producing more stable and reliable predictions. Each individual model (or "learner") may not perform well on its own, but when combined correctly, they can collectively yield a powerful result.

Key Elements of Ensemble Methods:

1. Base Models:

   Ensemble methods use base models, which can be either:

   • Homogeneous Models:

     These use the same base algorithm, like multiple decision trees.

   • Heterogeneous Models:

     These combine different algorithms, like decision trees and support vector machines, for diverse perspectives.

2. How Ensemble Models Work:

   • Bagging (Bootstrap Aggregating):

     A homogeneous ensemble model where weak learners are trained in parallel on different subsets of data and combined. This method primarily reduces variance.

   • Boosting:

     Also a homogeneous ensemble, boosting trains models sequentially. Each model learns from the errors of the previous one, reducing bias for high accuracy.

Purpose of Ensemble Methods:

• Ensemble methods enhance model accuracy by reducing bias and variance.
• They are highly effective for complex datasets, where single models struggle with capturing all patterns.

Bagging and Boosting: A Quick Overview

Bagging and Boosting in machine learning are two popular approaches to improving predictions. Here’s a quick look at how each method works and what makes it unique.

Bagging: Learning Side-by-Side

In Bagging (short for Bootstrap Aggregating), each model tries their best to make predictions. They work on different parts of the data, analyzing in parallel, without influencing each other’s process. At the end, their individual predictions are averaged out, giving us a final decision. This "parallel" approach helps Bagging reduce variance, which means it handles the randomness in the data better.

Key Points:

• Works in Parallel:

  Each model is trained independently on different data samples.

• Reduces Variance:

  Bagging smooths out extreme variations by averaging different models, producing a more stable result.

Example:

Random Forest is a classic Bagging technique where multiple decision trees (weak learners) each make their own prediction, and the average or majority vote becomes the final output.

Boosting: Learning from Mistakes, Step-by-Step

On the other hand, in Boosting, each model learns from the mistakes of the previous one. Models work sequentially rather than in parallel. Each new model tries to fix what the previous models got wrong. As this process continues, the ensemble gradually gets better at handling bias (systematic errors), becoming more precise with each step.

Key Points:

• Sequential Learning:

  Each model builds on the errors of the previous one.

• Reduces Bias:

  Boosting creates a highly accurate model by focusing on correcting mistakes.

Example:

In Adaptive Boosting (AdaBoost), the model assigns higher weights to data points misclassified in the previous round, training the next learner to pay closer attention to them. The result? A model that’s continuously refining itself to reduce bias.

Source: ScienceDirect

Bagging: How It Works

Definition:

Bagging, or Bootstrap Aggregating, is an ensemble technique in machine learning that improves model stability and accuracy by training multiple base models on varied subsets of the training data. This technique is particularly useful in stabilizing high-variance models like decision trees.

Technique Description

1. Dataset Splitting (Bootstrap Sampling)
Bagging uses row sampling with replacement, meaning each subset drawn from the original dataset can have duplicate rows. This process creates multiple, unique subsets of data, each used to train an independent model.

2. Independent Model Training
Each model is trained in parallel on its subset, ensuring each learns different aspects of the dataset. This parallel training makes Bagging computationally efficient when enough hardware resources are available.

3. Averaging Predictions
The predictions from each model are combined to form the final output. In classification tasks, this might be a majority vote; in regression, it could be the average of predictions. This aggregation reduces overfitting and variance, improving overall accuracy.

Implementation Steps

• Step 1:

  Generate multiple subsets from the original dataset using bootstrap sampling.

• Step 2:

  Train a base model (e.g., decision tree) independently on each subset.

• Step 3:

  Combine the predictions from each trained model to form the final output.

Types of Bagging Algorithms

1. Random Forest

   • Process:

     Builds multiple decision trees using random subsets of data and features, then aggregates the results for a final prediction. Each tree is trained independently on a bootstrap sample, which is a random subset of the training data, and at each split, only a random subset of features is considered.

   • Best For:

     Classification and regression tasks with high variance and large feature sets.

   • Unique Feature:

     Randomly selects rows and columns (features) to reduce correlation among trees, enhancing generalization and reducing overfitting.

2. Pasting

   • Process:

     Similar to Bagging but uses non-overlapping subsets of the original dataset (no replacement). Each subset is unique, ensuring that each model learns from distinct data points without repeated rows.

   • Best For:

     Datasets where sampling with replacement may lead to redundant data and where slight modifications in data points do not improve model diversity.

   • Unique Feature:

     Pasting avoids repeated samples, making it potentially more suitable for smaller datasets where data redundancy would limit model variety.

3. Random Patches

   • Process:

     The process randomly samples both rows and columns from the dataset to create diverse subsets, which are then used to train individual models. This technique combines row and feature sampling in each subset, providing more flexibility in model training.

   • Best For:

     High-dimensional data where feature reduction can help reduce computational costs without sacrificing model performance.

   • Unique Feature:

     Creates subsets with row and feature diversity, which is particularly useful for feature-rich datasets and helps prevent overfitting.

4. Random Subspaces

   • Process:

     A form of feature bagging, where models are trained on different subsets of features rather than rows. This method randomly samples only columns, not rows, to train individual models on varying feature combinations while keeping the full dataset intact for each model.

   • Best For:

     High-dimensional data with many irrelevant features, especially useful for image and text data.

   • Unique Feature:

     Focuses only on feature diversity, which helps reduce the impact of irrelevant features on the model and can lead to more efficient and interpretable models.

Source: ScienceDirect

Must Read: Free NLP Online Course! – Explore foundational NLP skills at no cost.

Example: Random Forest

The Random Forest algorithm is a popular application of Bagging, especially suited to high-variance models like decision trees. Each tree is trained on a bootstrap sample of the data, and features are randomly selected at each split.

python

from sklearn.ensemble import RandomForestClassifier

# Initializing and training the model
model = RandomForestClassifier(n_estimators=10, random_state=42)
model.fit(X_train, y_train)

Detailed Bagging Classifier Implementation

Below is a custom implementation of a Bagging Classifier from scratch, demonstrating how Bagging is applied using a simple base classifier and bootstrap sampling.

BaggingClassifier Class

python

import numpy as np

class BaggingClassifier:
    def __init__(self, base_classifier, n_estimators):
        self.base_classifier = base_classifier
        self.n_estimators = n_estimators
        self.classifiers = []
        
    def fit(self, X, y):
        for _ in range(self.n_estimators):
            # Bootstrap sampling with replacement
            indices = np.random.choice(len(X), len(X), replace=True)
            X_sampled = X[indices]
            y_sampled = y[indices]
            
            # Clone the base classifier and train it
            classifier = self.base_classifier.__class__()
            classifier.fit(X_sampled, y_sampled)
            self.classifiers.append(classifier)
    
    def predict(self, X):
        # Predictions from each classifier
        predictions = [classifier.predict(X) for classifier in self.classifiers]
        # Majority voting for final prediction
        majority_votes = np.apply_along_axis(lambda x: np.bincount(x).argmax(), axis=0, arr=predictions)
        return majority_votes

Using BaggingClassifier with Decision Tree Base Model

python

from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier

# Load dataset and split
digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Base classifier
base_model = DecisionTreeClassifier()

# Bagging with 10 estimators
bagging_model = BaggingClassifier(base_classifier=base_model, n_estimators=10)
bagging_model.fit(X_train, y_train)
y_pred = bagging_model.predict(X_test)

# Accuracy evaluation
accuracy = accuracy_score(y_test, y_pred)
print("Bagging Model Accuracy:", accuracy)

Read: Machine Learning Models Explained – Get insights into essential ML models and techniques.

Implementing a Random Forest: Step-by-Step Guide

Random Forest is an ensemble learning technique that creates multiple decision trees from subsets of data, then combines their predictions for improved accuracy and reduced overfitting. Here’s how it works in practice:

1. Data Preparation

   • Step:

     Start with a dataset containing X observations (rows) and Y features (columns).

   • Purpose:

     This dataset serves as the base for building multiple decision trees, each trained on a unique subset of data.

2. Bootstrap Sampling

   • Step:

     For each tree, create a random subset of the dataset by sampling with replacement. This means some data points may appear multiple times, while others may be excluded.

   • Purpose:

     Bootstrap sampling helps each tree in the forest learn unique data patterns, which increases model diversity.

3. Tree Creation

   • Step:

     Grow each decision tree to its full size without pruning. Each tree learns from the sampled data subset and is not influenced by the other trees.

   • Purpose:

     Unpruned trees capture detailed patterns in their specific subset, contributing to the ensemble’s ability to make accurate predictions.

4. Random Feature Selection

   • Step:

     At each split in a tree, select a random subset of features from the dataset, rather than considering all features.

   • Purpose:

     This feature randomness reduces the correlation between trees, improving the model’s ability to generalize to new data.

5. Prediction Aggregation

   • Step:

     After all trees have been built, use each tree’s prediction to make a final ensemble prediction. For classification tasks, the final prediction is determined by majority voting, while for regression tasks, it’s the average of all trees’ predictions.

   • Purpose:

     Aggregating predictions from multiple trees reduces overfitting and provides a more stable, accurate result.

Pros and Cons of Random Forest

Pros

• Handles High-Dimensional Data:

  Random Forest is well-suited for datasets with many features. Random feature selection at each split ensures all features are considered, reducing overfitting.

• Manages Missing Values:

  Random Forest can handle missing data points by averaging predictions or interpolating values, maintaining accuracy even with incomplete data.

• High Accuracy:

  The ensemble of decision trees yields highly accurate predictions, especially for classification tasks.

Cons

• Less Interpretability:

  With many trees involved, Random Forest can be complex to interpret, as it lacks the simplicity of a single decision tree.

• Averaged Prediction:

  In regression tasks, the final prediction is the average of individual trees’ predictions, which can reduce precision in some cases compared to other models.

Key Benefits

• Reduces Variance:

  Bagging minimizes overfitting in high-variance models like decision trees by averaging predictions.

• Improves Model Stability:

  Models trained on different data subsets produce more reliable predictions.

• Out-of-Bag Evaluation (OOB):

  Bagging enables model validation without separate test data, as models have unused data samples (“out-of-bag” data) for accuracy estimation.

Boosting: How It Works

Definition:
Boosting is a sequential ensemble technique combining multiple weak models to build a powerful model. In Boosting, each model tries to correct the errors made by its predecessor, leading to a highly accurate final model. This approach is especially effective for reducing bias in the model, making it popular for tasks requiring high precision.

Technique Description

1. Initial Model Training

   • Step:

     Start by training a base model on the initial dataset.

   • Purpose:

     This first model serves as the foundation. Errors from this model guide the training of subsequent models.

2. Weighted Dataset Adjustment

   • Step:

     Increase weights for misclassified data points, so these points are prioritized in the next training round.

   • Purpose:

     By focusing on the most difficult instances, the next model will likely learn the areas where the previous model struggled.

3. Sequential Model Training

   • Step:

     Train subsequent models in sequence, with each model aiming to correct the errors of the previous one.

   • Purpose:

     Sequential learning creates a strong model by continuously improving on previous errors, effectively minimizing both bias and error.

4. Final Model Aggregation

   • Step:

     Aggregate predictions from all models, typically using a weighted voting or averaging system.

   • Purpose:

     The combined output from all models results in a highly accurate prediction, leveraging the collective strength of each model.

Source: ScienceDirect

Implementation Steps

• Step 1:

  Train an initial model on the dataset and calculate errors.

• Step 2:

  Assign higher weights to misclassified points, ensuring they receive greater focus in the next model.

• Step 3:

  Sequentially train new models, each correcting errors from the last, until achieving desired accuracy or a maximum number of models.

• Step 4:

  Combine predictions from all models for the final result.

Types of Boosting Algorithms

1. AdaBoost (Adaptive Boosting)

   • Process:

     Focuses on sequential training by dynamically adjusting the weights of misclassified instances.

   • Best For:

     Binary classification problems.

   • Unique Feature:

     Adjusts weights based on errors in each iteration, improving accuracy over rounds.

2. Gradient Boosting

   • Process:

     Uses gradient descent to optimize loss functions over iterations.

   • Best For:

     Tasks requiring high accuracy and complex decision-making.

   • Unique Feature:

     Fits models on residuals of previous models, enhancing predictive accuracy.

3. XGBoost (Extreme Gradient Boosting)

   • Process:

     A fast, regularized variant of Gradient Boosting, optimized for performance.

   • Best For:

     High-dimensional data and large-scale problems.

   • Unique Feature:

     Includes regularization to prevent overfitting and improve speed.

Example: AdaBoost

AdaBoost (Adaptive Boosting) is one of the earliest and most widely used Boosting algorithms. AdaBoost adjusts weights dynamically, assigning higher weights to misclassified points with each iteration, focusing on areas needing improvement.

Python Code Example: AdaBoost with Scikit-Learn

python

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

# Load dataset
data = load_iris()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize AdaBoost with a Decision Tree as the base estimator
model = AdaBoostClassifier(
    base_estimator=DecisionTreeClassifier(max_depth=1), # Weak learner
    n_estimators=50,                                    # Number of boosting rounds
    learning_rate=1.0                                   # Learning rate
)

# Train AdaBoost model
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate model
accuracy = accuracy_score(y_test, y_pred)
print("AdaBoost Accuracy:", accuracy)

Detailed Boosting Implementation: AdaBoost Classifier

Below is a custom AdaBoost implementation demonstrating how Boosting works by focusing on misclassified points in each iteration.

python

import numpy as np

class AdaBoostClassifier:
    def __init__(self, base_classifier, n_estimators):
        self.base_classifier = base_classifier
        self.n_estimators = n_estimators
        self.models = []
        self.model_weights = []

    def fit(self, X, y):
        n_samples = X.shape[0]
        weights = np.full(n_samples, (1 / n_samples))  # Initial equal weight distribution

        for _ in range(self.n_estimators):
            # Train model on weighted dataset
            model = self.base_classifier.__class__()
            model.fit(X, y, sample_weight=weights)
            predictions = model.predict(X)

            # Calculate model error
            error = np.sum(weights * (predictions != y)) / np.sum(weights)

            # Calculate model weight
            model_weight = np.log((1 - error) / error)
            self.models.append(model)
            self.model_weights.append(model_weight)

            # Update weights
            weights *= np.exp(model_weight * (predictions != y))

    def predict(self, X):
        model_preds = np.array([model.predict(X) for model in self.models])
        weighted_preds = np.dot(self.model_weights, model_preds)
        return np.sign(weighted_preds)

Implementing AdaBoost: Step-by-Step Guide

AdaBoost, or Adaptive Boosting, is an ensemble learning technique that builds a strong classifier by sequentially combining multiple weak learners. Each subsequent learner focuses more on instances that previous models misclassified, gradually improving overall accuracy and reducing bias. Here’s how AdaBoost works in practice:

1. Data Preparation

   • Step:

     Start with a labeled dataset containing X observations (rows) and Y features (columns), which serves as the foundation for training the weak classifiers. Each data point is initially assigned an equal weight.

   • Purpose:

     The dataset provides the base for sequentially training multiple weak models (typically decision stumps or shallow trees), with each model focusing on the misclassified data points from its predecessor.

2. Initial Model Training and Error Calculation

   • Step:

     Train the first weak learner on the dataset and calculate its error rate by assessing the instances it misclassifies. The error rate is calculated as the weighted sum of misclassified points.

   • Purpose:

     By calculating error, AdaBoost determines how effective each model is and adjusts the focus of subsequent models on harder-to-predict instances.

3. Adjusting Weights for Misclassified Points

   • Step:

     Increase the weights of the misclassified data points, ensuring that these instances receive higher emphasis in the next training iteration. This process guides the model’s attention toward correcting previous mistakes.

   • Purpose:

     Focusing on misclassified data points helps each subsequent model correct the errors of its predecessor, refining the ensemble’s performance.

4. Sequential Model Training

   • Step:

     Train a new model using the updated weights, focusing more on the difficult-to-predict instances. Repeat this process iteratively, with each model further correcting the errors from the previous models.

   • Purpose:

     Sequentially building models in this manner reduces bias and continuously improves model accuracy.

5. Aggregating Model Predictions

   • Step:

     Combine the weak learners' predictions using a weighted voting system. Each model’s influence on the final prediction is based on its accuracy, with more accurate models having a greater weight.

   • Purpose:

     Weighted aggregation of predictions strengthens the overall accuracy, as each model contributes according to its performance.

Example: AdaBoost Implementation in Python

AdaBoost often uses decision stumps (one-level decision trees) as weak learners, but other classifiers can be used. Here’s an example of AdaBoost using a Decision Tree as the base classifier with Scikit-Learn:

python

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score

# Load dataset
data = load_iris()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize AdaBoost with a Decision Tree as the base estimator
model = AdaBoostClassifier(
    base_estimator=DecisionTreeClassifier(max_depth=1), # Weak learner
    n_estimators=50,                                    # Number of boosting rounds
    learning_rate=1.0                                   # Learning rate
)

# Train AdaBoost model
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate model
accuracy = accuracy_score(y_test, y_pred)
print("AdaBoost Model Accuracy:", accuracy)

Pros and Cons of AdaBoost

Pros

• High Accuracy:

  Sequentially correcting errors helps achieve better accuracy, making AdaBoost highly effective on structured data.

• Bias Reduction:

  Boosting reduces bias by iteratively adjusting to the data, which enhances its generalization ability.

• Handles Noisy Data Well:

  AdaBoost is less prone to overfitting than other ensemble methods when applied to structured, noisier data.

Cons

• Sensitive to Noisy Data and Outliers:

  Because AdaBoost focuses on misclassified points, noisy data points and outliers can negatively impact model accuracy if not managed well.

• High Computational Cost:

  AdaBoost’s sequential nature can be computationally intensive, especially with large datasets or high complexity.

• Model Interpretability:

  The reliance on multiple weak learners can make the final model less interpretable.

Key Benefits

• Adaptive Learning:

  AdaBoost dynamically adjusts to misclassified instances, helping models learn efficiently from errors.

• Enhanced Performance on Imbalanced Data:

  AdaBoost can focus more on minority classes in imbalanced datasets, improving predictive performance.

• Versatile Application:

  Works well across various weak learners, making it adaptable for classification and regression tasks.

Similarities Between Bagging and Boosting in Machine Learning

Description:

Both Bagging and Boosting are ensemble learning techniques that improve model performance by combining multiple weak learners to enhance prediction accuracy and stability.

Differences Between Bagging and Boosting in Machine Learning

Description:

Bagging and Boosting differ significantly in their approach to ensemble learning, data handling, and model training processes. Here’s a side-by-side comparison:

Upskill in Machine Learning with upGrad

Transform Your Career in Machine Learning – Get Roles with Salaries Up to ₹15 LPA

Jumpstart a career as a Data Scientist, AI Specialist, or Machine Learning Engineer, with salaries ranging from ₹6 LPA for entry-level positions to ₹23 LPA and beyond for experienced roles. 

Gain in-demand skills through hands-on projects, industry mentorship, and flexible learning schedules designed for busy professionals.

Unlock Top Career Roles

• Data Scientist | ₹8–20 LPA
• Machine Learning Engineer | ₹6–17 LPA
• AI Specialist | ₹7–23 LPA

Why Machine Learning Skills Matter

• High Demand:

  Industries from finance to healthcare rely on ML experts.

• Competitive Salaries:

  Upskill to qualify for high-paying roles.

• Job Security:

  Machine learning roles rank among the fastest-growing tech careers.

What upGrad Offers

• Real-World Projects:

  Work on datasets from leading industries.

• Mentorship from Experts:

  Gain insights from top professionals.

• Flexible Learning:

  Balance studies with work through our online courses.

Find your perfect learning path with our Best Online AI and Machine Learning Courses—each course crafted to provide in-depth knowledge, practical experience, and the tools you need to excel in the field!

Master in-demand machine learning skills like data preprocessing, model building, deep learning, and natural language processing to stay ahead in the AI-driven world!

Kickstart your AI and Machine Learning journey with our Free Courses! Dive into cutting-edge topics, from foundational concepts to advanced techniques, and learn at your own pace—completely free!