Image Segmentation Techniques [Step By Step Implementation]

What happens when you look at a selfie? Your eyes go straight to your face, right? You can easily spot it because your brain naturally separates your face from the background. But for a computer to do this, it needs a process called image segmentation.

Image segmentation is the first step in helping computers “see” objects in photos. It is a technique that helps computers break an image into parts. Each part shares similar features—maybe it’s the same color, texture, or shape. This separation makes it much easier for a computer to understand and work with images.

Here’s how it helps:

• Makes Images Easier to Understand:

  Instead of one big, complicated picture, the image is broken into simpler sections.

• Creates “Image Objects”:

  Groups parts of the image that look alike, like faces, trees, or cars.

• First Step for Analysis:

  Helps computers recognize and work with specific parts of the image.

This is how your phone recognizes faces, how self-driving cars spot road signs, and how medical scans identify specific areas. 

To make image segmentation in image processing work, there are different techniques we can use, each with its unique way of breaking down an image.

Now, let’s cover these techniques and how they work.

Image Segmentation Techniques

Image segmentation techniques break down an image into meaningful parts, which allow computers to identify objects and features. Here’s a look at some popular methods, their applications, and how to implement them step-by-step in Python.

1. Thresholding Segmentation

Overview: Thresholding is a basic image segmentation technique that converts an image into a binary form by comparing each pixel’s intensity to a set threshold value. Pixels above the threshold become white (255), while those below become black (0). This method works best when the object of interest is distinctly brighter or darker than the background.

• Types of Thresholding:

  • Simple Thresholding:

    Applies a single, fixed threshold to the entire image.

  • Otsu’s Binarization:

    Automatically determines an optimal threshold value by analyzing the histogram of the image, effective for images with clear foreground and background.

  • Adaptive Thresholding:

    Calculates varying thresholds for different sections, ideal for images with inconsistent lighting or complex backgrounds.

• Applications:

  Commonly used in medical imaging to highlight specific features, like cells, in quality control to spot defects, and in document scanning to separate text from the background.

Objective:Convert an image into a binary image by applying a threshold value, where pixels above the threshold are set to one value (e.g., white) and those below to another (e.g., black).

Steps:

Install OpenCV (if not installed):

bash

pip install opencv-python

• Load the Image and Convert to Grayscale:
  • This is necessary because thresholding works best on grayscale images.

python

import cv2
# Load the image in grayscale
image = cv2.imread('example.jpg', 0)  # 0 converts the image to grayscale

• Apply Simple Thresholding:
  • Choose a threshold value (e.g., 127). Pixels above this value are set to 255 (white), and those below are set to 0 (black).

python

# Apply simple thresholding
_, thresh_image = cv2.threshold(image, 127, 255, cv2.THRESH_BINARY)

• Display the Result:
  • The result is a binary image where regions above the threshold are white, and those below are black.

python

cv2.imshow('Thresholded Image', thresh_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The output will be a binary image where areas above the threshold are white (255), and areas below are black (0). For example, in a grayscale image of a document, the text appears black on a white background, effectively separating the text from the page.

Must Read: Free deep learning course!

2. Edge-Based Segmentation

Overview: Edge-based segmentation detects object boundaries by identifying significant changes in pixel intensity. This technique highlights edges where colors or shades transition sharply, making it easier to define object outlines. The Canny edge detector is commonly used as it computes intensity gradients and applies non-maximum suppression for clearer edges.

• Types of Edge Detection:

  • Search-Based Edge Detection:

    Finds edges by calculating intensity gradients and marking points of high contrast.

  • Zero-Crossing Edge Detection:

    Detects edges by finding zero-crossings in the second derivative of pixel intensity.

• Applications:

  Essential for applications that require clear boundary detection, such as facial recognition, autonomous driving, and industrial inspection. Edge detection can highlight object contours, reducing data for faster analysis.

Objective:

Detect edges in an image by identifying changes in intensity, which can be used to outline objects.

Steps:

Install OpenCV:
bash

pip install opencv-python

• Load the Image and Convert to Grayscale:
  • This is necessary for the edge detection algorithm to work effectively.

python

import cv2
# Load the image in grayscale
image = cv2.imread('example.jpg', 0)

• Apply Canny Edge Detection:
  • Canny edge detection is one of the most popular methods for edge detection, as it applies multiple steps to clean up and enhance edges.
  • The 100 and 200 values define the thresholds for edge detection.

python

# Apply Canny edge detection
edges = cv2.Canny(image, 100, 200)

• Display the Result:
  • The output will show only the edges of objects in the image, helping to highlight boundaries.

python

cv2.imshow('Canny Edge Detection', edges)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The result is an image displaying only the edges of objects. For example, in a photo of a car, Canny edge detection will highlight the car’s outline and key features like windows and wheels, making the structure of the object clearer.

3. Region-Based Segmentation

Overview: Region-based segmentation groups pixels into regions based on similar attributes, such as color or intensity. The process usually starts with a “seed” pixel, and the algorithm expands the region by adding neighboring pixels with similar properties. This method is especially useful for segmenting areas in images with clear, distinct regions.

• Types of Region-Based Segmentation:

  • Region Growing:

    Expands regions from initial seed points by merging neighboring pixels with similar properties.

  • Region Splitting and Merging:

    Divides the image into sections, then merges adjacent regions with similar characteristics.

• Applications:

  Widely used in medical imaging to identify organs, tumors, or abnormalities, as well as in remote sensing to distinguish land types and urban areas. Region-based methods are also valuable in applications where precise area identification is critical.

Objective:

Group pixels into regions based on similarity, typically starting from a “seed” point and growing the region by adding pixels with similar properties.

Steps:

Install OpenCV:
bash

pip install opencv-python

• Load the Image and Convert to Grayscale:
  • Region-based segmentation often begins with grayscale images for simplicity.

python

import cv2
import numpy as np
# Load the image in grayscale
image = cv2.imread('example.jpg', 0)

• Apply Thresholding to Simulate Region-Based Segmentation:
  • For demonstration, we use simple thresholding to create binary regions as a form of region-based segmentation.

python

# Apply simple thresholding to simulate region-based segmentation
_, segmented_image = cv2.threshold(image, 127, 255, cv2.THRESH_BINARY)

• Display the Result:
  • The result groups regions with similar intensities, making it easier to identify distinct areas.

python

cv2.imshow('Region-Based Segmentation', segmented_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The output is an image where regions with similar intensities are grouped together. In a medical brain scan, for example, regions with different tissue densities may appear as distinct areas, helping to identify and separate key structures, such as tumors or organs.

4. Watershed Segmentation

Overview:

Watershed segmentation views a grayscale image as a topographic map, treating brighter pixels as peaks and darker pixels as valleys. The algorithm identifies "catchment basins" (valleys) and "watershed lines" (ridges) to divide the image into distinct regions. The watershed approach is particularly useful for separating overlapping objects by defining boundaries based on pixel height.

• Process:

  Watershed fills basins with markers that expand until they meet at ridges, forming clear boundaries. It effectively segments images into regions based on pixel intensity.

• Applications:

  Watershed segmentation is widely used in medical imaging to separate touching anatomical structures, like identifying overlapping cells in MRI or CT scans.

Objective:Use the watershed algorithm to segment overlapping objects by treating the image as a topographic map. Pixels with higher intensity are treated as "higher elevation."

Steps:

Install OpenCV (if not installed):
bash

pip install opencv-python

• Load the Image and Convert to Grayscale:
  • The grayscale conversion is necessary for the watershed algorithm.

python

import cv2
import numpy as np
# Load the image and convert to grayscale
image = cv2.imread('example.jpg')
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

• Apply Thresholding to Create a Binary Image:
  • We use thresholding to create a binary image where objects of interest are separated from the background.

python

_, binary = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV + cv2.THRESH_OTSU)

• Remove Noise with Morphological Operations:
  • Morphological operations help clean up the binary image by removing small spots of noise.

python

kernel = np.ones((3, 3), np.uint8)
opening = cv2.morphologyEx(binary, cv2.MORPH_OPEN, kernel, iterations=2)

• Define Background and Foreground Areas:
  • Use dilation to define sure background and distance transformation for sure foreground.

python

sure_bg = cv2.dilate(opening, kernel, iterations=3)
dist_transform = cv2.distanceTransform(opening, cv2.DIST_L2, 5)
_, sure_fg = cv2.threshold(dist_transform, 0.7 * dist_transform.max(), 255, 0)

• Label Markers for Watershed Segmentation:

python

sure_fg = np.uint8(sure_fg)
unknown = cv2.subtract(sure_bg, sure_fg)
_, markers = cv2.connectedComponents(sure_fg)
markers = markers + 1
markers[unknown == 255] = 0

• Apply the Watershed Algorithm:
  • The watershed algorithm is applied to label regions. Borders between regions are marked with a red line.

python

markers = cv2.watershed(image, markers)
image[markers == -1] = [0, 0, 255]  # Mark boundaries in red
# Display result
cv2.imshow('Watershed Segmentation', image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The output image will display red lines marking boundaries between segmented regions. For instance, if you use an image with overlapping cells, watershed segmentation will create distinct boundaries around each cell, clearly separating them.

5. Clustering-Based Segmentation

Overview:

Clustering-based segmentation groups pixels based on their similarities using clustering algorithms like K-Means or Fuzzy C-Means. In image segmentation, clustering divides pixels into "clusters" where each cluster has similar characteristics, such as color, intensity, or texture.

• Techniques:

  • K-Means Clustering:

    A popular unsupervised algorithm that groups pixels by minimizing variance within clusters. It’s simple and effective for color-based segmentation.

  • Fuzzy C-Means (FCM):

    Allows pixels to belong to multiple clusters with varying degrees, making it flexible but more complex.

• Applications:

  Commonly used in social network analysis, market research, and object classification.

Objective:

Segment an image by grouping similar pixels using clustering. K-Means clustering is commonly used to separate colors or textures.

Steps:

Install OpenCV and NumPy:
bash

pip install opencv-python numpy

• Load Image and Reshape for Clustering:
  • Convert the image to a format compatible with K-Means clustering by flattening pixel values.

python

import cv2
import numpy as np
# Load the image
image = cv2.imread('example.jpg')
pixel_values = image.reshape((-1, 3))  # Flatten the 2D image into a 1D array of pixels
pixel_values = np.float32(pixel_values)  # Convert to float for K-Means

• Define K-Means Criteria and Apply Clustering:
  • Set the criteria for the K-Means algorithm and define the number of clusters.

python

# Define K-Means criteria and number of clusters (k)
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 100, 0.2)
k = 3  # Number of color clusters
_, labels, centers = cv2.kmeans(pixel_values, k, None, criteria, 10, cv2.KMEANS_RANDOM_CENTERS)

• Convert Cluster Centers to Integers:
  • Convert the centers to uint8 format and use them to replace pixel values.

python

centers = np.uint8(centers)
segmented_image = centers[labels.flatten()]
segmented_image = segmented_image.reshape(image.shape)  # Reshape to original image dimensions

• Display the Result:
  • The output shows the image segmented by color clusters.

python

cv2.imshow('K-Means Segmentation', segmented_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The output will be an image divided into k color-based clusters. For instance, in an image of a forest, K-Means might segment the image into areas representing tree canopies, trunks, and ground cover based on color clusters.

6. Neural Networks for Segmentation (e.g., Mask R-CNN)

Overview:

Neural networks, especially Convolutional Neural Networks (CNNs), are increasingly popular for complex image segmentation in image processing tasks. Advanced models like Mask R-CNN go further by generating pixel-level masks for objects, providing precise segmentation. Mask R-CNN builds on Faster R-CNN by adding a mask prediction to object detection, allowing it to classify and locate each object individually.

• Process:

  Mask R-CNN performs segmentation by creating bounding boxes and object masks for each detected object.

• Applications:

  Widely used in applications requiring high accuracy, like self-driving cars (identifying road features and pedestrians), medical image analysis (segmenting tumors), and interactive photo editing.

Objective:

Use a Convolutional Neural Network (CNN) for semantic segmentation, assigning each pixel a class label (e.g., road, car, person).

Steps:

Install TensorFlow:
bash

pip install tensorflow

• Define a Simple CNN Model for Semantic Segmentation:
  • We create a simple model with convolutional and upsampling layers to classify pixels.

python
 

import tensorflow as tf
from tensorflow.keras.layers import Conv2D, MaxPooling2D, UpSampling2D, Input
from tensorflow.keras.models import Model

# Define a simple CNN model for semantic segmentation
inputs = Input(shape=(256, 256, 3))  # Input shape can be adjusted based on dataset
x = Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(128, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
outputs = Conv2D(3, (1, 1), activation='softmax', padding='same')(x)  # 3 classes

# Compile the model
model = Model(inputs, outputs)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()

• Prepare Training Data:
  • Semantic segmentation requires labeled images, where each pixel is labeled with a class. Datasets like PASCAL VOC or Cityscapes provide such labeled data.
• Train the Model:
  • Train the model on a labeled dataset. Here’s a placeholder for the training code.

python

# model.fit(train_images, train_labels, batch_size=16, epochs=10)

• Evaluate/Use the Model:
  • After training, you can use the model to predict pixel classes for new images.

• python

  # predictions = model.predict(test_image)

Expected Output:
The model architecture defined here provides a structure for binary segmentation (e.g., distinguishing an object from the background). After training on labeled images, the model can separate objects in new images. For example, given a dataset with labeled "cat" and "background" regions, the model would learn to segment cat pixels from the background.

7. Semantic Image Segmentation

Overview:

Semantic image segmentation assigns each pixel in an image a class label, allowing for a detailed understanding of the scene. Unlike object detection, which provides bounding boxes around objects, semantic segmentation identifies each pixel as belonging to a specific category, such as road, car, or person. This technique is commonly used in applications where a high level of detail is required.

• Applications:

  Essential for tasks like autonomous driving (classifying roads, vehicles, pedestrians), medical imaging, and environmental monitoring. This technique provides a pixel-level understanding of scenes, which is crucial for accurate object recognition.

Objective:

Classify each pixel in an image into specific categories, providing a detailed understanding of the entire scene (e.g., labeling pixels as road, car, person, etc.).

Steps:

Install TensorFlow:
bash

pip install tensorflow

• Define a Simple CNN Model for Semantic Segmentation:

  • This model assigns class labels to each pixel in an image, suitable for semantic segmentation.

python

import tensorflow as tf
from tensorflow.keras.layers import Conv2D, MaxPooling2D, UpSampling2D, Input
from tensorflow.keras.models import Model

# Define a CNN model
inputs = Input(shape=(256, 256, 3))  # Adjust input shape as needed
x = Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)
x = MaxPooling2D((2, 2))(x)
x = Conv2D(128, (3, 3), activation='relu', padding='same')(x)
x = UpSampling2D((2, 2))(x)
outputs = Conv2D(3, (1, 1), activation='softmax', padding='same')(x) # 3 classes for example

model = Model(inputs, outputs)
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
model.summary()

• Prepare Training Data:

  • Semantic segmentation requires a labeled dataset with pixel-level annotations. You can use datasets like PASCAL VOC, Cityscapes, or any custom dataset with similar annotations.

Train the Model:
python

# Placeholder training code - replace `train_images` and `train_labels` with actual data
# model.fit(train_images, train_labels, batch_size=16, epochs=10)

• Use the Model for Prediction:

  • Once trained, use the model to predict the class labels for each pixel in a test image.

python

# predictions = model.predict(test_image)

Expected Output:
After training, this model can classify each pixel in an image. For instance, in a street scene, pixels might be labeled as road, car, or pedestrian, providing a detailed map of each object type across the scene.

8. Color-Based Segmentation

Overview:

Color-based segmentation divides an image by grouping pixels with similar color properties. By using color spaces like HSV (Hue, Saturation, Value), this technique can target specific colors, making it effective for applications where color is a defining feature of objects.

• Applications:

  Widely used in image editing, computer graphics, and any application where color is essential for object identification, like detecting ripe fruits in agriculture or identifying colored markers in robotics.

Objective:

Segment objects based on color characteristics using the HSV color space, which is useful for isolating objects of a particular color.

Steps:

Install OpenCV:
bash

pip install opencv-python

• Load the Image and Convert to HSV:
  • Convert the image to HSV color space for easier color filtering.

python

import cv2
import numpy as np

# Load the image
image = cv2.imread('example.jpg')

# Convert the image to HSV color space
hsv_image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)

• Define Color Range for Segmentation:
  • Define the range of the target color (e.g., red) using HSV values.

python

# Define color range for red (adjust as needed)
lower_red = np.array([0, 120, 70])
upper_red = np.array([10, 255, 255])

• Create a Mask for the Specified Color Range:
  • Create a binary mask that highlights only the specified color.

python

# Create a binary mask for the red color
mask = cv2.inRange(hsv_image, lower_red, upper_red)

• Apply the Mask to the Original Image:
  • Use bitwise operations to apply the mask and isolate the color.

python

segmented_image = cv2.bitwise_and(image, image, mask=mask)

# Display result
cv2.imshow('Color-Based Segmentation', segmented_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The output will be an image showing only the red-colored objects, with everything else masked out. For example, in an image with various colored objects, only red items will remain visible, effectively isolating them from the background.

9. Texture-Based Segmentation

Overview:

Texture-based segmentation groups pixels by analyzing patterns and textures, such as smoothness or roughness. Filters like Gabor are used to detect variations in texture by analyzing spatial frequency, orientation, and scale. This method is particularly helpful in distinguishing areas with distinct textural differences.

• Applications:

  Commonly used in medical imaging to identify different tissue types, as each tissue may exhibit unique textural properties. Also used in industrial quality control to differentiate surface finishes or detect defects.

Objective:

Segment an image based on texture patterns using Gabor filters, which are useful for distinguishing regions with different textures (e.g., rough vs. smooth surfaces).

Steps:

Install OpenCV:
bash

pip install opencv-python

• Load the Image in Grayscale:
  • Convert the image to grayscale, as texture-based segmentation works effectively on grayscale images.

python

import cv2
import numpy as np

# Load the image in grayscale
image = cv2.imread('example.jpg', 0)

• Define Gabor Filter Parameters:
  • Gabor filters detect textures by analyzing spatial frequency and orientation.

python

# Define Gabor filter parameters (size, orientation, frequency, etc.)
kernel = cv2.getGaborKernel((21, 21), 8.0, np.pi/4, 10.0, 0.5, 0, ktype=cv2.CV_32F)

• Apply Gabor Filter to the Image:
  • Use the defined Gabor filter to analyze the texture in the image.

python

# Apply the Gabor filter to the image
filtered_image = cv2.filter2D(image, cv2.CV_8UC3, kernel)

• Display the Result:
  • The output highlights areas with the specified texture characteristics.

python

# Display the result
cv2.imshow('Texture-Based Segmentation', filtered_image)
cv2.waitKey(0)
cv2.destroyAllWindows()

Expected Output:
The result will display areas that match the texture pattern defined by the Gabor filter. For instance, it may highlight rough areas in an industrial image or specific textures in medical imaging.

Modes and Types of Image Segmentation

Image segmentation in image processing comes in several forms, each designed for specific tasks. Here’s an overview of the three primary modes—semantic, instance, and panoptic segmentation:

Comparing Image Classification, Object Detection, and Image Segmentation

It's useful to know how image classification, object detection, and image segmentation are different. What makes them unique helps you choose the right approach, whether you’re trying to analyze or use images.

Deep Learning Image Segmentation Models

Deep learning models for image segmentation in image processing use neural networks to divide an image into segments and identify important features within each segment. These models are especially useful in fields like medical imaging, autonomous driving, and robotics.

Below are some popular deep learning models for image segmentation:

1. U-Net

U-Net is a U-shaped network designed specifically for biomedical image segmentation. Its architecture uses an encoder-decoder path, where the encoder extracts features and the decoder refines localization.

• Purpose:

  Highly effective for small datasets and detailed segmentation tasks.

• Applications:

  Biomedical image analysis, like segmenting cells, organs, and other biological structures.

2. Fully Convolutional Network (FCN)

FCN replaces fully connected layers in a CNN with convolutional layers to output spatial predictions, enabling pixel-wise segmentation.

• Purpose:

  Allows for pixel-wise prediction across the entire image.

• Applications:

  Scene segmentation in autonomous driving, satellite image analysis.

3. SegNet

SegNet uses an encoder-decoder network structure where the encoder captures the context, and the decoder performs precise localization.

• Purpose:

  Effective for semantic segmentation tasks where both context and detail are crucial.

• Applications:

  Scene understanding in robotics, autonomous driving.

4. DeepLab

DeepLab uses atrous (dilated) convolutions to handle multi-scale context by applying multiple parallel filters. It’s known for its flexibility in handling complex segmentation tasks.

• Purpose:

  Captures multi-scale information, making it suitable for detailed, complex segmentation.

• Applications:

  Satellite image analysis, medical imaging, fine-grained scene segmentation.

5. Mask R-CNN

An extension of Faster R-CNN for object detection, Mask R-CNN adds a branch for predicting segmentation masks for each detected object.

• Purpose:

  Provides instance segmentation by detecting objects and creating a mask for each.

• Applications:

  Instance segmentation in tasks like identifying individual people or objects in a scene.

6. Vision Transformer (ViT)

ViT adapts transformer architectures for image segmentation by dividing an image into patches and processing them sequentially.

• Purpose:

  Provides global context understanding, useful for capturing details across the entire image.

• Applications:

  Advanced segmentation tasks that benefit from sequential patch processing, such as satellite imagery and high-resolution image analysis.

Where Can Image Segmentation Be Used?

Image segmentation is helpful in many fields. It helps to find, separate, and label objects in images. With most new data being visual, businesses and researchers rely on image segmentation to make sense of it all. Let’s look at some ways different industries use image segmentation and why it’s so valuable:

Looking to get into AI and Machine Learning? upGrad has a range of courses designed to make learning straightforward and practical. Here are some of our top programs:

• Master of Science in Machine Learning & AI

• Master of Science in Machine Learning & Artificial Intelligence

• PG Certification in Machine Learning and Deep Learning

• PG Certification in Machine Learning and NLP

Accelerate your career with our best-in-class Machine Learning and AI courses, designed to provide you with cutting-edge knowledge and hands-on experience for real-world impact.

Stay ahead in the AI era by developing crucial machine learning skills like predictive modeling, deep learning frameworks, and algorithm optimization to transform data into actionable intelligence.

Discover our popular AI and ML blogs and free courses, packed with expert insights and practical guidance to help you master cutting-edge technologies and excel in your career.