Bagging vs Boosting in Machine Learning: Difference Between Bagging and Boosting
Updated on Mar 01, 2025 | 18 min read | 93.3k views
Share:
For working professionals
For fresh graduates
More
Updated on Mar 01, 2025 | 18 min read | 93.3k views
Share:
Table of Contents
“Which ensemble technique improves model accuracy the most: Bagging or Boosting?”
As machine learning evolves, bagging and boosting in machine learning have become essential techniques for improving model accuracy. These ensemble methods combine multiple models to create a stronger, more reliable predictor.
Bagging (Bootstrap Aggregating) reduces variance by training models in parallel on different data subsets, averaging their predictions for stability. Boosting, on the other hand, reduces bias by training models sequentially, where each model corrects the errors of the previous one, leading to better accuracy.
Both methods enhance prediction performance, but they serve different purposes. This blog will compare bagging and boosting in machine learning, explaining how they work, when to use them, and their impact on model performance—helping you choose the right approach.
You can also learn about the various machine learning project ideas here.
Advance Your Career with Machine Learning Courses – Study online with top universities. Choose from Masters, Executive PG Programs, or Advanced Certificates in ML & AI, all designed to fast-track your career.
In machine learning, ensemble methods combine multiple "weak" models to create a stronger, more accurate model. The main goal of ensemble methods is to improve the model's accuracy by balancing bias and variance, producing more stable and reliable predictions. Each individual model (or "learner") may not perform well on its own, but when combined correctly, they can collectively yield a powerful result.
1. Base Models: Ensemble methods use base models, which can be either:
2. How Ensemble Models Work:
Purpose of Ensemble Methods:
Bagging and boosting in machine learning are two essential ensemble methods used to improve model accuracy and reliability. Bagging trains multiple models in parallel on different data subsets to reduce variance, while boosting trains models sequentially, correcting errors at each step to reduce bias.
Here’s a quick look at how each method works and what makes it unique.
In Bagging (short for Bootstrap Aggregating), each model tries their best to make predictions. They work on different parts of the data, analyzing in parallel, without influencing each other’s process. At the end, their individual predictions are averaged out, giving us a final decision. This "parallel" approach helps Bagging reduce variance, which means it handles the randomness in the data better.
Example:
Random Forest is a classic Bagging technique where multiple decision trees (weak learners) each make their own prediction, and the average or majority vote becomes the final output.
On the other hand, in Boosting, each model learns from the mistakes of the previous one. Models work sequentially rather than in parallel. Each new model tries to fix what the previous models got wrong. As this process continues, the ensemble gradually gets better at handling bias (systematic errors), becoming more precise with each step.
Example:
In Adaptive Boosting (AdaBoost), the model assigns higher weights to data points misclassified in the previous round, training the next learner to pay closer attention to them. The result? A model that’s continuously refining itself to reduce bias.
Source: ScienceDirect
Definition:
Bagging, or Bootstrap Aggregating, is an ensemble technique in machine learning that improves model stability and accuracy by training multiple base models on varied subsets of the training data. This technique is particularly useful in stabilizing high-variance models like decision trees.
1. Dataset Splitting (Bootstrap Sampling)
Bagging uses row sampling with replacement, meaning each subset drawn from the original dataset can have duplicate rows. This process creates multiple, unique subsets of data, each used to train an independent model.
2. Independent Model Training
Each model is trained in parallel on its subset, ensuring each learns different aspects of the dataset. This parallel training makes Bagging computationally efficient when enough hardware resources are available.
3. Averaging Predictions
The predictions from each model are combined to form the final output. In classification tasks, this might be a majority vote; in regression, it could be the average of predictions. This aggregation reduces overfitting and variance, improving overall accuracy.
Step 1:Generate multiple subsets from the original dataset using bootstrap sampling.
Step 2:Train a base model (e.g., decision tree) independently on each subset.
Step 3:Combine the predictions from each trained model to form the final output.
Process: Builds multiple decision trees using random subsets of data and features, then aggregates the results for a final prediction. Each tree is trained independently on a bootstrap sample, which is a random subset of the training data, and at each split, only a random subset of features is considered.
Best For: Classification and regression tasks with high variance and large feature sets.
Unique Feature: Randomly selects rows and columns (features) to reduce correlation among trees, enhancing generalization and reducing overfitting.
Process: Similar to Bagging but uses non-overlapping subsets of the original dataset (no replacement). Each subset is unique, ensuring that each model learns from distinct data points without repeated rows.
Best For: Datasets where sampling with replacement may lead to redundant data and where slight modifications in data points do not improve model diversity.
Unique Feature: Pasting avoids repeated samples, making it potentially more suitable for smaller datasets where data redundancy would limit model variety.
Process: The process randomly samples both rows and columns from the dataset to create diverse subsets, which are then used to train individual models. This technique combines row and feature sampling in each subset, providing more flexibility in model training.
Best For: High-dimensional data where feature reduction can help reduce computational costs without sacrificing model performance.
Unique Feature: Creates subsets with row and feature diversity, which is particularly useful for feature-rich datasets and helps prevent overfitting.
Process: A form of feature bagging, where models are trained on different subsets of features rather than rows. This method randomly samples only columns, not rows, to train individual models on varying feature combinations while keeping the full dataset intact for each model.
Best For: High-dimensional data with many irrelevant features, especially useful for image and text data.
Unique Feature: Focuses only on feature diversity, which helps reduce the impact of irrelevant features on the model and can lead to more efficient and interpretable models.
Source: ScienceDirect
Must Read: Free NLP Online Course! – Explore foundational NLP skills at no cost.
The Random Forest algorithm is a popular application of Bagging, especially suited to high-variance models like decision trees. Each tree is trained on a bootstrap sample of the data, and features are randomly selected at each split.
python
from sklearn.ensemble import RandomForestClassifier
# Initializing and training the model
model = RandomForestClassifier(n_estimators=10, random_state=42)
model.fit(X_train, y_train)
Below is a custom implementation of a Bagging Classifier from scratch, demonstrating how Bagging is applied using a simple base classifier and bootstrap sampling.
python
import numpy as np
class BaggingClassifier:
def __init__(self, base_classifier, n_estimators):
self.base_classifier = base_classifier
self.n_estimators = n_estimators
self.classifiers = []
def fit(self, X, y):
for _ in range(self.n_estimators):
# Bootstrap sampling with replacement
indices = np.random.choice(len(X), len(X), replace=True)
X_sampled = X[indices]
y_sampled = y[indices]
# Clone the base classifier and train it
classifier = self.base_classifier.__class__()
classifier.fit(X_sampled, y_sampled)
self.classifiers.append(classifier)
def predict(self, X):
# Predictions from each classifier
predictions = [classifier.predict(X) for classifier in self.classifiers]
# Majority voting for final prediction
majority_votes = np.apply_along_axis(lambda x: np.bincount(x).argmax(), axis=0, arr=predictions)
return majority_votes
python
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
# Load dataset and split
digits = load_digits()
X, y = digits.data, digits.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Base classifier
base_model = DecisionTreeClassifier()
# Bagging with 10 estimators
bagging_model = BaggingClassifier(base_classifier=base_model, n_estimators=10)
bagging_model.fit(X_train, y_train)
y_pred = bagging_model.predict(X_test)
# Accuracy evaluation
accuracy = accuracy_score(y_test, y_pred)
print("Bagging Model Accuracy:", accuracy)
Read: Machine Learning Models Explained – Get insights into essential ML models and techniques.
Random Forest is an ensemble learning technique that creates multiple decision trees from subsets of data, then combines their predictions for improved accuracy and reduced overfitting. Here’s how it works in practice:
1. Data Preparation
2. Bootstrap Sampling
3. Tree Creation
4. Random Feature Selection
5. Prediction Aggregation
Definition:
Boosting is a sequential ensemble technique combining multiple weak models to build a powerful model. In Boosting, each model tries to correct the errors made by its predecessor, leading to a highly accurate final model. This approach is especially effective for reducing bias in the model, making it popular for tasks requiring high precision.
1. Initial Model Training
2. Weighted Dataset Adjustment
3. Sequential Model Training
4. Final Model Aggregation
Source: ScienceDirect
AdaBoost (Adaptive Boosting) is one of the earliest and most widely used Boosting algorithms. AdaBoost adjusts weights dynamically, assigning higher weights to misclassified points with each iteration, focusing on areas needing improvement.
python
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
# Load dataset
data = load_iris()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Initialize AdaBoost with a Decision Tree as the base estimator
model = AdaBoostClassifier(
base_estimator=DecisionTreeClassifier(max_depth=1), # Weak learner
n_estimators=50, # Number of boosting rounds
learning_rate=1.0 # Learning rate
)
# Train AdaBoost model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate model
accuracy = accuracy_score(y_test, y_pred)
print("AdaBoost Accuracy:", accuracy)
Below is a custom AdaBoost implementation demonstrating how Boosting works by focusing on misclassified points in each iteration.
python
import numpy as np
class AdaBoostClassifier:
def __init__(self, base_classifier, n_estimators):
self.base_classifier = base_classifier
self.n_estimators = n_estimators
self.models = []
self.model_weights = []
def fit(self, X, y):
n_samples = X.shape[0]
weights = np.full(n_samples, (1 / n_samples)) # Initial equal weight distribution
for _ in range(self.n_estimators):
# Train model on weighted dataset
model = self.base_classifier.__class__()
model.fit(X, y, sample_weight=weights)
predictions = model.predict(X)
# Calculate model error
error = np.sum(weights * (predictions != y)) / np.sum(weights)
# Calculate model weight
model_weight = np.log((1 - error) / error)
self.models.append(model)
self.model_weights.append(model_weight)
# Update weights
weights *= np.exp(model_weight * (predictions != y))
def predict(self, X):
model_preds = np.array([model.predict(X) for model in self.models])
weighted_preds = np.dot(self.model_weights, model_preds)
return np.sign(weighted_preds)
AdaBoost, or Adaptive Boosting, is an ensemble learning technique that builds a strong classifier by sequentially combining multiple weak learners. Each subsequent learner focuses more on instances that previous models misclassified, gradually improving overall accuracy and reducing bias. Here’s how AdaBoost works in practice:
1. Data Preparation
2. Initial Model Training and Error Calculation
3. Adjusting Weights for Misclassified Points
4. Sequential Model Training
5. Aggregating Model Predictions
AdaBoost often uses decision stumps (one-level decision trees) as weak learners, but other classifiers can be used. Here’s an example of AdaBoost using a Decision Tree as the base classifier with Scikit-Learn:
python
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_iris
from sklearn.metrics import accuracy_score
# Load dataset
data = load_iris()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)
# Initialize AdaBoost with a Decision Tree as the base estimator
model = AdaBoostClassifier(
base_estimator=DecisionTreeClassifier(max_depth=1), # Weak learner
n_estimators=50, # Number of boosting rounds
learning_rate=1.0 # Learning rate
)
# Train AdaBoost model
model.fit(X_train, y_train)
# Make predictions
y_pred = model.predict(X_test)
# Evaluate model
accuracy = accuracy_score(y_test, y_pred)
print("AdaBoost Model Accuracy:", accuracy)
Description:
Both Bagging and Boosting are ensemble learning techniques that improve model performance by combining multiple weak learners to enhance prediction accuracy and stability.
Feature |
Bagging & Boosting |
Ensemble Learning |
Both are ensemble techniques designed to combine multiple models, leveraging the strengths of each for a stronger overall model. |
Base Model Usage |
They both use base learners, typically weak classifiers, and aggregate their results to produce a strong prediction. |
Reduction of Variance |
Both aim to reduce errors in the final model, with Bagging reducing variance and Boosting reducing bias. |
Combining Predictions |
The final prediction is made by averaging or taking the majority vote of the models’ predictions. |
Application |
Both are commonly used in supervised learning tasks like classification and regression. |
Model Improvement |
They enhance model performance by iterating over weak models to correct or reduce errors. |
Feature Importance |
Both methods can produce feature importance scores, helping identify the key drivers of the model. |
Parallelism |
Bagging builds models in parallel, while Boosting typically builds them sequentially. Both involve multiple models that work together. |
Final Prediction |
The final prediction combines the output of all models to achieve higher accuracy and robustness. |
Description:
Bagging and Boosting differ significantly in their approach to ensemble learning, data handling, and model training processes. Here’s a side-by-side comparison:
Feature |
Bagging |
Boosting |
Purpose |
Reduces variance in high-variance models |
Reduces bias by sequentially correcting model errors |
Model Independence |
Models are trained independently, in parallel |
Models are dependent, with each model correcting the last |
Weighting of Models |
Equal weight given to all models |
Weights are adjusted based on performance |
Training Data |
Each model uses random subsets with replacement |
Each model focuses on data points misclassified by the previous model |
Popular Example |
Random Forest |
AdaBoost, Gradient Boosting |
Best For |
High-variance models like decision trees |
High-bias models where sequential adjustments are helpful |
Iterative Process |
Not iterative; models don’t depend on one another |
Iterative; each model is trained based on previous results |
Combining Predictions |
Aggregates predictions by voting or averaging |
Combines weighted predictions based on accuracy |
Application Use Cases |
Suitable for data with more noise and variability |
Suitable for datasets where accuracy improvement is needed over multiple iterations |
Parallelism |
Parallel processing of models |
Sequential processing for error correction |
Transform Your Career in Machine Learning – Get Roles with Salaries Up to ₹15 LPA
Jumpstart a career as a Data Scientist, AI Specialist, or Machine Learning Engineer, with salaries ranging from ₹6 LPA for entry-level positions to ₹23 LPA and beyond for experienced roles.
Gain in-demand skills through hands-on projects, industry mentorship, and flexible learning schedules designed for busy professionals.
Unlock Top Career Roles
Why Machine Learning Skills Matter
What upGrad Offers
Both bagging and boosting in machine learning are powerful ensemble methods that enhance model performance, but they work differently. Bagging vs boosting comes down to their approach—bagging reduces variance by training models in parallel, making it ideal for high-variance models like decision trees, while boosting reduces bias by correcting errors sequentially, improving accuracy on complex datasets.
Choosing between bagging vs boosting depends on your needs—bagging enhances model stability, while boosting improves precision. Understanding these differences helps you select the right technique to optimize machine learning models for better accuracy and performance.
Find your perfect learning path with our Best Online AI and Machine Learning Courses—each course crafted to provide in-depth knowledge, practical experience, and the tools you need to excel in the field!
Master in-demand machine learning skills like data preprocessing, model building, deep learning, and natural language processing to stay ahead in the AI-driven world!
Kickstart your AI and Machine Learning journey with our Free Courses! Dive into cutting-edge topics, from foundational concepts to advanced techniques, and learn at your own pace—completely free!
Get Free Consultation
By submitting, I accept the T&C and
Privacy Policy
Top Resources