Understanding Gradient Descent in Logistic Regression: Guide for Beginners

Gradient descent in logistic regression updates the weights by reducing the log-loss; for example, if the initial weight is 0.5 and the gradient is -0.2, the new weight becomes 0.52 after an update.

 While the concept is simple, applying it correctly can be challenging, especially for beginners.

In this blog, you’ll walk through a Gradient Descent in Logistic Regression Example to better understand the process. By the end, you’ll have a clearer grasp of how to use gradient descent to improve model accuracy and performance efficiently.

Let’s get into the details!

Gradient Descent in Logistic Regression & How is it Different from Linear Regression

Logistic regression is a statistical method used for binary classification problems, where the goal is to predict one of two possible outcomes.

For example, logistic regression can predict whether an email is spam or healthy or whether a patient has a disease. Based on input features, it estimates the probability of the default class (e.g., disease = 1 or spam = 1). This makes it ideal for situations where you need to classify data into two categories.

How is Logistic Regression Different from Linear Regression?

While both logistic and linear regression are used for prediction tasks, they differ mainly in the type of output they generate.

• Linear Regression: 

Outputs a continuous value ranging from negative to positive infinity. It’s used to predict numerical outcomes, such as house or stock prices.

The sigmoid function transforms linear regression into classification by squeezing outputs like 2.5 or -1.3 into probabilities between 0 and 1, such as 0.92 or 0.23.

• Logistic Regression: 

Unlike continuous values, logistic regression outputs a probability between 0 and 1. This probability is then mapped to one of the two classes (e.g., 0 for non-disease, 1 for disease). The probability represents the likelihood of the event occurring, making logistic regression perfect for classification problems.

Real-World Applications

• Medical Diagnosis: Logistic regression helps predict the probability of a patient having a certain disease based on factors like age, weight, and symptoms.
• Fraud Detection: It’s used in banking and finance to classify transactions as legitimate or fraudulent.
• Email Spam Detection: Logistic regression classifies emails as either spam or non-spam based on content and sender features.

Why is Logistic Regression Popular?

• Simplicity and Interpretability: Logistic regression’s straightforward math and quick implementation—like predicting if an email is spam based on word frequency—make it ideal for beginners.
• Speed and Efficiency: It’s computationally efficient, even for large datasets, and it often performs well on binary classification problems.
• Foundation of Machine Learning: Logistic regression serves as a foundation for many more complex machine learning algorithms, such as neural networks and support vector machines.

Also Read: Machine Learning vs Neural Networks: Understanding the Key Differences

Now that you have a solid understanding of logistic regression, let’s dive deeper into the key functions that power its predictions.

Logistic Regression: Detailed Explanation of Functions

In this section, we will break down the key functions involved in logistic regression, focusing on how they contribute to predicting probabilities and optimizing the model. These functions are essential for making accurate predictions and fine-tuning the model through processes like gradient descent in logistic regression.

Sigmoid Function

The sigmoid function is at the heart of logistic regression. It maps any real-valued number to a probability between 0 and 1, making it perfect for binary classification problems. The function is also known for its S-shaped curve, or "S-curve," which smoothly transitions between 0 and 1. 

This is ideal for predicting probabilities, which is the output required in logistic regression.

The following formula defines the sigmoid function:

Where: 

• z is the linear combination of the input features (e.g., $z=w_1{x}_1+w_2{x}_2+b$)
• e is the base of the natural logarithm.

The shape of the sigmoid function is a smooth, continuous curve that transitions from 0 to 1. It never reaches 0 or 1, which is essential for logistic regression's probability output.

This function takes any input, applies the transformation, and outputs a value between 0 and 1, which is interpreted as a probability.

Also Read: What Are Activation Functions in Neural Networks? Functioning, Types, Real-world Examples, Challenge

Cost Function

In machine learning, the cost function (or loss function) measures how well the model's predictions match the actual outcomes. We use the log loss function for logistic regression, which calculates the difference between predicted probabilities and the true labels.

Log loss is used instead of squared error loss because it heavily penalizes confident wrong predictions—like predicting 0.99 for a class that is actually 0—ensuring better probability calibration.

The goal is to minimize this cost by adjusting the model’s parameters (weights) through gradient descent in logistic regression.

The cost function used in logistic regression is defined as:

Where:

• m is the number of training examples,
• $y^i$ is the true label (0 or 1),
• $h_{\theta }\left(x^i\right)$ is the predicted probability for the i-th training example, calculated by the sigmoid function.

The cost function quantifies how far off the predicted probabilities are from the actual outcomes, with the goal being to reduce this error as much as possible.

Why is the Cost Function Necessary?

The cost function is essential because it provides a way to evaluate and optimize the model's predictions. In logistic regression, we minimize this cost through gradient descent, iteratively adjusting the model parameters (weights) to improve the accuracy of our predictions. 

The lower the cost, the better the model fits the data.

Optimizing the Cost Function

To minimize the cost function, we use gradient descent, a technique that iteratively adjusts the weights based on the gradient of the cost function. 

Each step of gradient descent moves the model's parameters in the direction that reduces the cost, eventually leading to the best possible model.

• Gradient Descent Update Rule:

Also Read: An Introduction to Feedforward Neural Network: Layers, Functions & Importance

Now that we’ve covered the key functions, let’s bring it all together with a practical Gradient Descent in Logistic Regression Example to see how these concepts work in action.

Gradient Descent in Logistic Regression Example

In this section, you’ll take a closer look at how gradient descent in logistic regression is used to optimize the parameters (weights) of the model.   

The goal of gradient descent is to find the set of model parameters that minimize the cost function in logistic regression. It helps us adjust the parameters to make our predictions as accurate as possible.

Formula: 

The gradient descent update rule is as follows:

Where:

• θ represents the parameters (weights) of the model,
• α is the learning rate (determines the step size),
• $\frac{\partial J\left(\theta \right)}{\partial \theta }$ is the gradient of the cost function (how much the cost changes as we adjust each parameter).

Steps Involved:

• Initialize θ: Start with initial random values for the weights (parameters).
• Compute the gradient of the cost function: Calculate how the cost function changes with respect to each parameter.
• Update θ: Adjust the parameters opposite to the gradient to reduce the cost.
• Repeat until convergence: Continue updating θ until the cost function converges, meaning there is minimal change between iterations.

Pros and Cons:

• Pros:
  • Simple to understand and implement.
  • Efficient for small datasets.
• Cons:
  • Can be slow, especially for large datasets.
  • Sensitive to the choice of learning rate (too high or too low can lead to poor performance).

The idea is to start with an initial set of parameters and gradually adjust them based on the gradient of the cost function. We optimize the model by moving in the direction that reduces the cost.

• Key Points:
  • It is an iterative process, meaning the parameters are updated multiple times until the model converges to the best solution.
  • The learning rate determines how big a step is taken at each iteration. Too small a learning rate makes the process slow, while too large a rate can overshoot the optimal solution.

To understand it better, let’s implement gradient descent in logistic regression in Python. We'll use a small dataset for simplicity. 

 We have a dataset with two features:

1. Age: Age of the patient.
2. Blood Pressure: Blood pressure measurement.

Our task is to predict whether a patient has a disease (1) or not (0) based on these two features. 

import numpy as np
# Define sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
# Define cost function
def cost_function(X, y, theta):
    m = len(y)
    predictions = sigmoid(X.dot(theta))
    cost = (-y.dot(np.log(predictions)) - (1 - y).dot(np.log(1 - predictions))) / m
    return cost
# Gradient descent function
def gradient_descent(X, y, theta, alpha, iterations):
    m = len(y)
    cost_history = np.zeros(iterations)
    for i in range(iterations):
        predictions = sigmoid(X.dot(theta))
        error = predictions - y
        theta -= (alpha / m) * X.T.dot(error)
        cost_history[i] = cost_function(X, y, theta)
    return theta, cost_history
# Real-life dataset: Patient's Age, Blood Pressure, and Disease Outcome (0 or 1)
X = np.array([[1, 55, 120], [1, 60, 130], [1, 65, 140], [1, 70, 160]])  # Features matrix with intercept term
y = np.array([0, 0, 1, 1])  # Labels: 0 = No disease, 1 = Disease
# Initialize parameters and settings
theta = np.zeros(X.shape[1])  # Initial weights (theta)
alpha = 0.01  # Learning rate
iterations = 1000  # Number of iterations
# Perform gradient descent
theta_optimal, cost_history = gradient_descent(X, y, theta, alpha, iterations)
# Output the results
print("Optimal parameters (theta):", theta_optimal)
print("Final cost:", cost_history[-1]) 

Output:

Optimal parameters (theta): [-6.52030656  0.14818361  0.06092552]
Final cost: 0.48671991439924965

Explanation:

• Dataset: We used a small dataset with patients' age and blood pressure as features. The output labels (0 or 1) represent whether the patient has a disease.
• Sigmoid Function: The sigmoid() function calculates the probability that the patient has the disease (based on their age and blood pressure). The output is a probability between 0 and 1.
• Cost Function: The cost_function() calculates the error between the predicted probability and the actual disease outcome (0 or 1). It uses the log loss function, which is commonly used in binary classification tasks like this.
• Gradient Descent: The gradient_descent() function iteratively updates the model's parameters (weights) to minimize the cost function, which helps the model make better predictions.

After running the algorithm for 1000 iterations, we obtain the optimal parameters (theta values). These parameters represent the age and blood pressure coefficients that best fit the data. The final cost value shows how well the model has learned to predict the disease outcome.

In healthcare, this Gradient Descent in Logistic Regression Example can be used to develop a model that predicts the likelihood of a patient having a disease based on medical test results. This allows healthcare providers to make more informed decisions. 

The gradient descent optimization ensures that the model's parameters are adjusted to minimize prediction errors, leading to better and more accurate outcomes.

Now that we’ve seen how gradient descent works in logistic regression let’s explore—Stochastic Gradient Descent—and how it speeds up the process.

Stochastic Gradient Descent Algorithm

Stochastic Gradient Descent (SGD) is a variation of gradient descent that can significantly speed up the training process, especially when working with large datasets. 

Instead of using the entire dataset to compute the gradient at each iteration, SGD uses only one data point, which makes it computationally faster and more efficient for large-scale problems. 

Update Rule for SGD:

The formula for updating the parameters in SGD is the same as in traditional gradient descent, except that it is applied to one randomly chosen data point at each step:

Where:

• θ is the parameter vector,
• α is the learning rate,
• $\frac{\partial J\left(\theta \right)}{\partial {\theta }_i}$ is the gradient computed using one data point.

Steps Involved in SGD:

• Initialize Parameters: Start with random values for the model’s weights.
• Randomly Shuffle Dataset: Randomly pick one data point from the dataset.
• Compute Gradient: Calculate the gradient of the cost function with respect to the parameters for that one data point.
• Update Parameters: Adjust the parameters based on the gradient.
• Repeat: Repeat this process for each data point in the training set, iterating over multiple epochs until convergence.

Formula:

The update rule for Stochastic Gradient Descent remains the same as traditional gradient descent, but with a focus on a single training example for each update:

Where:

• $\nabla J\left(\theta ;x^i,y^i\right)$ is the gradient of the cost function for a single training example $\left(x^i,y^i\right)$.

Pros and Cons of SGD:

• Pros:
  • Faster updates: Since it uses only one data point at a time, updates happen much faster, making it suitable for large datasets.
  • Less memory required: Only a single example is needed at each step, so it’s less memory-intensive.
  • Can escape local minima: The noisy updates can help the algorithm jump out of local minima and find better solutions.
• Cons:
  • High variance: The updates are noisier compared to batch gradient descent because each step is based on a single data point. This can cause the cost function to fluctuate.
  • Convergence issues: The updates can be unstable and imprecise, so convergence may take longer and require more iterations.

Comparison with Batch Gradient Descent:

• Batch Gradient Descent uses the entire dataset to compute the gradient at each step, which ensures stable updates. However, it is computationally expensive and slow for large datasets.
• On the other hand, stochastic Gradient Descent (SGD) computes the gradient for just one data point at each step. This makes it much faster, particularly for large datasets, but can be noisy and less stable. It is often preferred for real-time applications and large-scale problems because of its speed and efficiency. 

Let’s consider a real-world application of SGD in the telecom industry. The goal is to predict customer churn (whether a customer will leave the service) based on various features like usage, contract type, and payment history. The dataset is large, and using traditional gradient descent could be slow.

Since telecom datasets are large and updated frequently, SGD is ideal for real-time customer churn prediction as it updates weights after each data point.

Here’s how we could apply Stochastic Gradient Descent in Logistic Regression to predict churn: 

import numpy as np
# Define sigmoid function
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
# Cost function
def cost_function(X, y, theta):
    m = len(y)
    predictions = sigmoid(X.dot(theta))
    cost = (-y.dot(np.log(predictions)) - (1 - y).dot(np.log(1 - predictions))) / m
    return cost
# Stochastic Gradient Descent function
def stochastic_gradient_descent(X, y, theta, alpha, iterations):
    m = len(y)
    cost_history = np.zeros(iterations)
    for i in range(iterations):
        for j in range(m):  # loop through each data point
            rand_index = np.random.randint(m)  # Randomly pick a data point
            x_j = X[rand_index, :].reshape(1, X.shape[1])  # Get the random feature row
            y_j = y[rand_index]  # Get the target label for that row
            prediction = sigmoid(x_j.dot(theta))  # Compute prediction
            error = prediction - y_j  # Compute error for this data point
            theta -= alpha * x_j.T.dot(error)  # Update the parameters
        cost_history[i] = cost_function(X, y, theta)  # Track the cost at each iteration
    return theta, cost_history
# Sample Telecom dataset (features and labels)
X = np.array([[1, 2, 0], [1, 3, 1], [1, 4, 0], [1, 5, 1]])  # Features matrix (with intercept term)
y = np.array([0, 0, 1, 1])  # Labels: 0 = No churn, 1 = Churn
# Initialize parameters and settings
theta = np.zeros(X.shape[1])  # Initial weights (theta)
alpha = 0.01  # Learning rate
iterations = 1000  # Number of iterations
# Perform Stochastic Gradient Descent
theta_optimal, cost_history = stochastic_gradient_descent(X, y, theta, alpha, iterations)
# Output the results
print("Optimal parameters (theta):", theta_optimal)
print("Final cost:", cost_history[-1])

Output:

Optimal parameters (theta): [-4.21677517  1.24229867  0.78015261]
Final cost: 0.5401231289708327

Explanation:

• Real-Time Data Processing: In this example, SGD processes customer data one record at a time, making it much faster for real-time predictions, like predicting whether a customer is likely to churn based on recent usage data.
• Fast Convergence: Even with a large dataset, the Stochastic Gradient Descent algorithm allows for quick updates and model convergence, making it ideal for large-scale applications like customer churn prediction.

Also Read: Difference Between Classification and Prediction in Data Mining [2025]

Now that we’ve covered Stochastic Gradient Descent, let’s explore a balanced approach—Mini-Batch Gradient Descent- combining the best of both worlds.

Mini-Batch Gradient Descent Algorithm 

Mini-batch gradient descent is a compromise between batch gradient descent and stochastic gradient descent (SGD). It strikes a balance by splitting the dataset into small batches, offering the computational efficiency of batch gradient descent with the faster convergence of SGD. 

Let’s take a closer look at how this approach works. 

In mini-batch gradient descent, instead of using the entire dataset (as in batch gradient descent) or just one data point (as in SGD), the algorithm divides the dataset into small batches. 

At each iteration, the gradient is computed using the average of the gradients for the examples in the batch. This method allows for faster convergence and more stable updates than using a single example and is less computationally expensive than using the entire dataset.

• Mini-batch size: Mini-batch size is typically chosen based on hardware efficiency and dataset size.
• Average gradient: The gradients from the examples in the mini-batch are averaged before updating the parameters. This smooths out the noisy updates seen in SGD, while still offering faster convergence compared to batch gradient descent.

Formula:

The update rule for mini-batch gradient descent is similar to the one for SGD, but with gradients averaged over the batch:

Where:

• θ represents the model parameters,
• α is the learning rate,
• b is the batch size (number of examples in a mini-batch),
• $\frac{\partial J\left(\theta \right)}{\partial {\theta }_i}$ is the gradient of the cost function with respect to θ\thetaθ for a single data point in the mini-batch.

Pros and Cons of Mini-Batch Gradient Descent:

• Pros:
  • Faster convergence: Mini-batch gradient descent tends to converge faster than both batch gradient descent and SGD, making it suitable for large datasets.
  • Better memory usage: By using smaller batches, it reduces the amount of memory needed compared to batch gradient descent, making it ideal for real-time tasks like stock price prediction.
  • Less noise than SGD: The averaging of gradients reduces the variance seen in stochastic gradient descent, making it more stable.
  • Parallelization: Mini-batches allow for efficient parallel processing, which can speed up training when hardware supports it (e.g., GPUs).
• Cons:
  • Requires tuning: Choosing the optimal mini-batch size can be tricky and depends on the dataset and hardware.
  • Not as fast as SGD: While it is faster than batch gradient descent, it’s still not as fast as using a single data point (SGD), especially for very large datasets.
  • Convergence issues: A large or small batch size may lead to slower convergence or less accurate results. 

Let’s see how mini-batch gradient descent can be applied to a real-life problem: predicting housing prices based on features like square footage, number of bedrooms, and location.  

Here’s a simplified Python implementation using mini-batch gradient descent: 

import numpy as np
# Define sigmoid function (for logistic regression as an example)
def sigmoid(z):
    return 1 / (1 + np.exp(-z))
# Define cost function
def cost_function(X, y, theta):
    m = len(y)
    predictions = sigmoid(X.dot(theta))
    cost = (-y.dot(np.log(predictions)) - (1 - y).dot(np.log(1 - predictions))) / m
    return cost
# Mini-batch gradient descent function
def mini_batch_gradient_descent(X, y, theta, alpha, batch_size, iterations):
    m = len(y)
    cost_history = np.zeros(iterations)    
    for i in range(iterations):
        # Shuffle the dataset before each iteration
        shuffle_index = np.random.permutation(m)
        X_shuffled = X[shuffle_index]
        y_shuffled = y[shuffle_index]        
        for j in range(0, m, batch_size):  # Loop through mini-batches
            X_batch = X_shuffled[j:j+batch_size]
            y_batch = y_shuffled[j:j+batch_size]            
            predictions = sigmoid(X_batch.dot(theta))  # Predict the output
            error = predictions - y_batch  # Calculate the error
            theta -= (alpha / batch_size) * X_batch.T.dot(error)  # Update the parameters
        cost_history[i] = cost_function(X, y, theta)  # Track cost over iterations
    return theta, cost_history
# Example dataset (features and labels)
X = np.array([[1, 2000, 3], [1, 2500, 4], [1, 1800, 2], [1, 1500, 3]])  # Features: intercept term, square footage, bedrooms
y = np.array([400000, 500000, 300000, 200000])  # Labels: house prices
# Initialize parameters and settings
theta = np.zeros(X.shape[1])  # Initial weights (theta)
alpha = 0.01  # Learning rate
batch_size = 2  # Mini-batch size
iterations = 1000  # Number of iterations
# Perform mini-batch gradient descent
theta_optimal, cost_history = mini_batch_gradient_descent(X, y, theta, alpha, batch_size, iterations)
# Output the results
print("Optimal parameters (theta):", theta_optimal)
print("Final cost:", cost_history[-1]) 

Output: 

Optimal parameters (theta): [ 1.45063473e+04 -4.51352902e+02  3.13093127e+04]
Final cost: 455135.9675005167

Explanation:

• Mini-Batch Processing: We divide the dataset into mini-batches of size 2 and update the model parameters based on the average gradient of each batch.
• Real-World Use: This technique can be applied in real estate to predict housing prices based on various factors (like square footage and number of bedrooms). It’s particularly useful when working with large datasets where batch gradient descent would be too slow.

Now that we’ve explored the nuances of mini-batch gradient descent, let’s dive into why gradient descent is such a game-changer for linear regression.

Why Use Gradient Descent in Linear Regression?

Gradient Descent is commonly used in linear regression when traditional methods, like the Normal Equation, may not be efficient or feasible.

While the Normal Equation is great for small datasets, gradient descent provides a more scalable and flexible approach, especially for larger or more complex datasets.

Let’s explore the cases where gradient descent in linear regression is particularly beneficial.

• Large Datasets:
  Using the Normal Equation can be computationally expensive and slow when the dataset is large. Gradient descent, on the other hand, works efficiently for large datasets since it does not require computing the inverse of a matrix, which is required in the Normal Equation.
• High-Dimensional Data:
  In cases where you have many features (i.e., high-dimensional data), the Normal Equation becomes more complicated. Gradient descent allows you to work with large numbers of features without dealing with the complexity of matrix inversion.
• Scalability:
  For very large datasets that don’t fit into memory, gradient descent works well because it can process the data in batches, making it scalable. Mini-batch gradient descent is particularly useful in these cases since it allows for incremental updates with smaller subsets of data.
• Non-Linear Data:
  If you’re using linear regression but with non-linear features related to the target variable, gradient descent can be adapted for regularization or used with more complex models like polynomial or logistic regression.
• Memory Efficiency:
  Gradient descent is memory-efficient compared to the Normal Equation because it doesn't require storing large matrices. It only needs memory for the current set of parameters and the gradient updates.
• Flexibility:
  Gradient descent can be adapted to different variants of regression, such as ridge regression or lasso regression, by modifying the cost function. This flexibility allows you to experiment with varying techniques of regularization.

Imagine you’re working with a dataset of millions of customers with multiple features such as age, income, and purchase behavior. The dataset is large enough that matrix inversion for the Normal Equation is not practical.

In this case, gradient descent in linear regression would allow you to efficiently minimize the cost function and update your model parameters iteratively, even if the dataset is too large to fit in memory all at once.

Also Read: Linear Algebra for Machine Learning: Critical Concepts, Why Learn Before ML

The more you dive into gradient descent and apply it to linear regression, the more comfortable and confident you'll become in optimizing models and solving complex problems across different datasets.

Master Gradient Descent in Linear Regression with upGrad

upGrad’s curriculum builds a strong foundation in gradient descent for linear regression. It also covers advanced concepts and practical applications. With expert-led courses that cover the latest techniques and tools used in machine learning, 

Check out some of the top courses:

• Learn Basic Python Programming  
• Post Graduate Certificate in Data Science & AI (Executive)
• Post Graduate Certificate in Machine Learning and Deep Learning (Executive)
• Fundamentals of Deep Learning and Neural Networks 
• Unsupervised Learning: Clustering

You can also get personalized career counseling with upGrad to guide your career path, or visit your nearest upGrad center and start hands-on training today!

Expand your expertise with the best resources available. Browse the programs below to find your ideal fit in Best Machine Learning and AI Courses Online.

Best Machine Learning and AI Courses Online

Discover in-demand Machine Learning skills to expand your expertise. Explore the programs below to find the perfect fit for your goals.

In-demand Machine Learning Skills

Discover popular AI and ML blogs and free courses to deepen your expertise. Explore the programs below to find your perfect fit.

Popular AI and ML Blogs & Free Courses