Apache Spark Streaming Tutorial: A Step-by-Step Guide to Real-Time Data Processing

Apache Spark Streaming processes real-time data from sources like Kafka, Flume, and cloud storage using micro-batches for efficient, fault-tolerant processing.

Unlike traditional batch systems, which process data in intervals, Spark Streaming handles data incrementally for lower latency and higher scalability. It excels in integrating with the broader Spark ecosystem for large-scale, real-time data processing compared to frameworks like Flink or Kafka Streams.

This Apache Spark Streaming tutorial covers architecture, key concepts, and applications—an essential skill for data engineers and big data professionals.

Apache Spark Streaming Tutorial: Step-by-Step Guide

Apache Spark Streaming is a powerful real-time data processing engine built on Apache Spark’s distributed computing framework. It processes continuous data streams using micro-batching, breaking incoming data into small batches for near real-time computation. 

It works by ingesting real-time data streams from sources like Kafka, Flume, or IoT sensors, processing them in micro-batches using Spark’s Resilient Distributed Dataset (RDD) model, and applying transformations (map, filter, reduce) to extract insights.

The processed data is then stored in HDFS, NoSQL databases, or indexed in ElasticSearch for further analysis, enabling low-latency, fault-tolerant, and scalable real-time analytics.

Now, let’s dive into the Apache Spark Streaming tutorial and learn what Spark Streaming is, its key components, and how to set up and run a streaming application step by step:

Step 1: Prerequisites

Before running a Spark Streaming application, ensure you have the following:

• Apache Spark (Latest Version, 3.x+) installed on your system.
• Java 8+ or Python 3.x for running Spark applications.
• Kafka, Flume, or Socket Streams as data sources.
• Hadoop or a cloud-based data lake (AWS S3, Azure Blob Storage, or Google Cloud Storage) for storing intermediate results.

When using cloud-based data lakes, you benefit from scalable storage and seamless integration with cloud-native services, reducing infrastructure overhead.

In contrast, local Hadoop-based setups may require more manual configuration and scaling to handle large data volumes efficiently.

Also Read: Apache Spark Architecture: Everything You Need to Know

Step 2: Create a Spark Streaming Context

The Spark Streaming Context is the entry point for any Spark Streaming application. It defines the streaming job and manages data flow.

Example (Python – PySpark):

from pyspark.sql import SparkSession
from pyspark.streaming import StreamingContext

# Initialize Spark Session
spark = SparkSession.builder.appName("SparkStreamingApp").getOrCreate()

# Define Streaming Context with 5-second batch intervals
ssc = StreamingContext(spark.sparkContext, 5)

# Enable checkpointing for fault tolerance
ssc.checkpoint("hdfs://namenode:8020/spark_checkpoint")

# Print the state of the StreamingContext to ensure it's active
print(ssc)
# Alternatively, you can use ssc.getState() to check the status
print(ssc.getState())

Explanation:

1. Import Libraries:

• SparkSession: Manages the Spark application.
• StreamingContext: Enables real-time data processing.

2. Initialize Spark Session:

• Creates a SparkSession named "SparkStreamingApp".
• getOrCreate() ensures an existing session is reused if available.

3. Define Streaming Context:

• Uses sparkContext to set up streaming context.
• The batch interval is set to 5 seconds, meaning Spark Streaming will collect and process data in batches every 5 seconds.

4. Enable Checkpointing for Fault Tolerance:

• The ssc.checkpoint() method is crucial for ensuring fault tolerance in your streaming application. 
• It saves the state of the computation to a reliable storage (e.g., HDFS). In case of a failure, this checkpoint helps recover the application to its last successful state.

5. Verify Streaming Context:

• To ensure the streaming context is active, you can print ssc or use ssc.getState() to check its status.
• This will give you information about whether the streaming context has been initialized and is running correctly.

Expected Output: This code initializes Spark Streaming but doesn’t produce direct output. The console logs will confirm the setup:

INFO StreamingContext: StreamingContext started with batch interval 5s
INFO SparkSession: Initialized SparkSession: SparkStreamingApp

If you print ssc or check its state, you will see output similar to:

StreamingContext[streamingContextId, active=false]

Instead of processing each event individually, Spark Streaming groups incoming data into small batches at fixed intervals. In this case, it processing logs every 5 seconds

• T = 0-5s: Collect logs and process in batch 1.
• T = 5-10s: Collect new logs and process in batch 2.
• T = 10-15s: Repeat, ensuring continuous real-time analysis.

Micro-batching balances latency and throughput, making it more efficient than event-driven systems like Apache Flink for high-throughput applications.

Also Read: Apache Kafka Architecture: Comprehensive Guide For Beginners

Step 3: Define Data Input Sources

Spark Streaming supports multiple real-time data sources, including:

• Kafka: For high-throughput real-time event processing (e.g., Twitter's real-time analytics pipeline uses Kafka and Spark to process millions of tweets per second, enabling trend analysis and sentiment detection).
• Flume: Used for ingesting large volumes of event-driven log data (e.g., application logs in cloud environments).
• Socket Streams: For real-time text data (e.g., live server logs or IoT telemetry).
• Cloud Storage & Databases: Streaming from AWS S3, Google Cloud, or NoSQL databases.

Example (Reading from Kafka in Python – Using Structured Streaming API):

from pyspark.sql import SparkSession

# Initialize Spark Session
spark = SparkSession.builder.appName("SparkStreamingApp").getOrCreate()

# Read data from Kafka using Structured Streaming API
kafkaStream = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "localhost:9092") \
    .option("subscribe", "events") \
    .load() 

Explanation:

1. Structured Streaming:

• Instead of using the deprecated KafkaUtils.createDirectStream(), you should now use Structured Streaming to process data from Kafka in a more reliable and scalable manner.
• spark.readStream is the new approach to streaming data, providing a higher-level abstraction for handling real-time data streams.

2. Create a Direct Kafka Stream:

• spark.readStream.format("kafka") specifies that you're reading from Kafka as a stream.
• .option("kafka.bootstrap.servers", "localhost:9092") sets the Kafka broker address (e.g., localhost:9092) where Kafka is running.
• .option("subscribe", "events") defines the Kafka topic ("events") that Spark will consume data from.
• .load() triggers the loading of the stream from Kafka.

Note: This approach eliminates the need for KafkaUtils.createDirectStream(), simplifying the code and ensuring better integration with Spark's built-in structured streaming capabilities.

Expected Output:

This code does not produce immediate output but starts consuming messages from Kafka. If successful, the console will show logs like:

INFO KafkaUtils: Connected to Kafka topic: events
INFO Kafka Stream started successfully

Once Kafka starts sending messages, Spark will begin processing real-time data from the "events" topic. 

Also Read: Kafka vs RabbitMQ: What Are the Biggest Differences and Which Should You Learn

Step 4: Apply Transformations

Transformations are operations that modify the incoming data. Spark Streaming offers a variety of transformations, including map(), filter(), reduceByKey(), and window(). 

These operations allow you to transform, filter, and aggregate data as it streams in real time.

• map(): Applies a function to each data element.
• filter(): Filters data based on a condition.
• window(): Aggregates data over a fixed time window (e.g., compute moving averages).
• updateStateByKey(): A stateful transformation that allows maintaining state across batches, useful for tasks like running totals or counting unique items.

Example (Filtering error logs in real-time):

from pyspark.sql.functions import col

# Convert Kafka messages to STRING and filter for "ERROR" logs
error_logs = kafkaStream.selectExpr("CAST(value AS STRING) as value") \
    .filter(col("value").contains("ERROR"))

Explanation:

1. Convert Kafka Messages: Kafka messages are typically in binary format (e.g., bytes). To filter based on the content of the messages, you first need to convert the value field from binary to string using CAST(value AS STRING).

2. Filter Streaming Data:

• This line processes the real-time Kafka stream and filters messages containing the word "ERROR".
• col("value").contains("ERROR") checks each message in the Kafka stream for the keyword "ERROR".
• Only logs that contain the word "ERROR" will be retained for further processing.

3. Stateful Transformations (optional note): If you want to keep track of state across different batches (e.g., running totals or counts), you can use updateStateByKey(). This is a useful stateful transformation to track ongoing calculations in streaming data.

4. Streaming Data Processing:

• This is a real-time filtering operation, meaning only error logs from Kafka’s "events" topic will be processed.
• The resulting error_logs stream can be saved, displayed, or sent to monitoring tools for alerting.

Expected Output:

The stream will process Kafka messages in real time, and only logs that contain the word "ERROR" will be retained. If successful, the console will show logs like:

INFO StreamingQuery: Processing 100 records...

Once the filtering is applied, the error_logs stream will be ready for further transformations or actions (like writing to a file or a database).

Also Read: Apache Kafka Architecture: Comprehensive Guide For Beginners

Step 5: Define Output Operations

Once data is processed, it must be stored or sent to a sink. Common output destinations (sinks) include:

• HDFS or Cloud Storage (AWS S3, Google Cloud Storage) for long-term storage.
• NoSQL databases (Cassandra, MongoDB) for real-time dashboards.
• Elasticsearch for indexing and full-text search (e.g., log analysis).

Example (Saving results to HDFS with append mode):

error_logs.writeStream \
    .format("text") \
    .option("path", "hdfs://namenode:8020/logs/errors") \
    .option("checkpointLocation", "hdfs://namenode:8020/spark_checkpoint/errors") \
    .outputMode("append") \
    .start()

Explanation:

1. Saves Filtered Logs to HDFS:

• This line writes the filtered error logs to a Hadoop Distributed File System (HDFS) location.
• The directory path "hdfs://namenode:8020/logs/errors" specifies where the files will be stored.

2. Handling Incremental Writes:

• By default, Spark Streaming might overwrite files if not configured properly. To prevent this, we use outputMode("append"), which ensures that logs are added incrementally rather than overwriting existing files.
• This configuration guarantees that each batch of error logs is appended to the output files without deleting or replacing previous data.

3. How Data is Stored:

• Spark Streaming creates multiple text files within the directory based on the batch intervals defined in the StreamingContext.
• Each batch generates a new file containing only error logs from that interval.

4. Checkpointing: checkpointLocation is specified to store the state of the stream in case of failures, which is important for fault tolerance and recovery.

5. File Naming Convention:

The output files are automatically named like:

/logs/errors-00000
/logs/errors-00001

Each file contains error logs captured within that batch.

Example (Saving Results to Cassandra or MongoDB): In addition to saving data to HDFS, Spark can also write to NoSQL databases like Cassandra or MongoDB. 

• Saving to Cassandra:

error_logs.writeStream \
    .format("org.apache.spark.sql.cassandra") \
    .option("keyspace", "logs") \
    .option("table", "error_logs") \
    .outputMode("append") \
    .start()

• Saving to MongoDB:

error_logs.writeStream \
    .format("mongodb") \
    .option("uri", "mongodb://localhost:27017/logs.error_logs") \
    .outputMode("append") \
    .start()

Expected Output in HDFS: 

After running this, you can check the saved error logs in HDFS using:

hdfs dfs -ls /logs/errors

Example output:

Found 2 items
-rw-r--r--   1 spark spark   2048 2025-02-05 12:31 /logs/errors-00000
-rw-r--r--   1 spark spark   3072 2025-02-05 12:36 /logs/errors-00001

To view the stored logs:

hdfs dfs -cat /logs/errors-00000

Example content:

ERROR 2025-02-05 12:30:45 [Server] Connection timeout occurred.
ERROR 2025-02-05 12:30:47 [Database] Query execution failed.

This allows long-term storage of error logs for analysis, debugging, and compliance tracking. 

Also Read: Top 20 HDFS Commands You Should Know About

Step 6: Start and Monitor the Application

After defining the context, sources, transformations, and outputs, start the Spark Streaming job:

ssc.start()  # Start execution  
ssc.awaitTermination()  # Keep application running

Explanation:

1. Start Streaming Execution:

• ssc.start() begins the Spark Streaming job, activating all defined streaming computations.
• Once started, Spark continuously ingests, processes, and outputs data based on the transformations applied (e.g., filtering logs, writing to HDFS).

2. Keep the Application Running:

• ssc.awaitTermination() keeps the streaming job alive until it is manually stopped.
• Without this, the program would execute once and exit immediately.

3. Real-Time Processing:

• Spark continuously processes new data from Kafka, Flume, or other sources based on the batch interval set in StreamingContext.
• The application will keep running indefinitely unless manually stopped or terminated due to an error.

4. Graceful Shutdown: 

• To ensure a clean shutdown, you should use ssc.stop(stopSparkContext=True, stopGraceFully=True), which will stop the Spark Streaming context gracefully and preserve the processed data. This method helps prevent any data loss during termination.
• Implementation in a try-except block:

try:
    ssc.start()  # Start the streaming job
    ssc.awaitTermination()  # Keep the application running
except KeyboardInterrupt:
    print("Streaming job stopped manually.")
finally:
    ssc.stop(stopSparkContext=True, stopGraceFully=True)  # Gracefully stop the job

Monitoring Spark Streaming Jobs:

• Spark UI: You can monitor the performance and progress of your Spark Streaming job via the Spark UI. Navigate to the 'Streaming' tab to see detailed information about the batch processing, including how long each batch took, the number of records processed, and the status of the current batch.
• Prometheus/Grafana: If you have Prometheus and Grafana set up, you can use them to track metrics like throughput, latency, and resource usage for Spark Streaming jobs.

Expected Output (Console Logs):

Once executed, Spark will start processing real-time streaming data, and the console will show logs like:

INFO StreamingContext: StreamingContext started with batch interval 5s
INFO KafkaUtils: Connected to Kafka topic: events
INFO StreamingJob: Processing new batch of streaming data

If new data arrives in the Kafka topic, Spark processes it in 5-second micro-batches and writes results to HDFS or another specified sink.

To manually stop Spark Streaming:

ssc.stop(stopSparkContext=True, stopGraceFully=True)

Or in the terminal:

Ctrl + C  # Stop execution

This ensures the streaming application shuts down gracefully while preserving processed data. 

Monitoring tools like Spark UI, Prometheus, or Grafana can help track performance and debug issues.

Also Read: Apache Spark Dataframes: Features, RDD & Comparison

How Apache Spark Streaming Works: Architecture and Components?

Spark Streaming follows a structured pipeline to process real-time data efficiently. Below are its key components:

Input Sources in Spark Streaming

These are the data ingestion points where Spark Streaming collects real-time data. Examples include:

• Kafka (Event Processing): Used for handling high-throughput data streams like user interactions in web applications.
• Flume (Log Streaming): Commonly used for ingesting system logs from distributed infrastructure.
• Amazon Kinesis (Cloud Streaming): Streams real-time data for fraud detection and AI-driven applications.

Spark Streaming Engine

The Spark Streaming Engine converts raw streaming data into Discretized Streams (DStreams), which are then processed using Spark’s distributed computing model. Key features include:

• Fault Tolerance: If a node fails, Spark recovers lost data from the last checkpoint.
• Scalability: Automatically scales across a distributed cluster for large workloads.
• Micro-Batching: Processes data in fixed intervals (e.g., every 5 seconds).

Sinks in Spark Streaming

Sinks are output destinations where processed data is stored. Common sinks include:

• HDFS, AWS S3, Azure Blob Storage – For persisting processed data.
• Elasticsearch – For real-time log analysis and search-based applications.
• PostgreSQL, MySQL, NoSQL (Cassandra, MongoDB) – For storing structured real-time data.

To learn  advanced SQL techniques and elevate your data analysis capabilities, enroll in upGrad’s free course on SQL Functions and Formulas. 

Complete Apache Spark Streaming Code (Kafka Integration)

This Spark Streaming application reads real-time data from Kafka, processes it by filtering error logs, and writes the results to HDFS for further analysis.

Before running this code, ensure:

• Apache Spark 3.x+ is installed.
• Apache Kafka is set up with a topic named "logs".
• Hadoop HDFS is running for storing processed logs.

Here’s the complete Spark Streaming Code (PySpark):

# Import necessary libraries
from pyspark.sql import SparkSession
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils

# Step 1: Initialize Spark Session
spark = SparkSession.builder \
    .appName("SparkStreamingKafka") \
    .config("spark.sql.streaming.checkpointLocation", "/tmp/checkpoints") \
    .getOrCreate()

# Step 2: Create Spark Streaming Context with batch interval of 5 seconds
ssc = StreamingContext(spark.sparkContext, 5)

# Step 3: Define Kafka Data Source (Topic: "logs")
kafka_stream = KafkaUtils.createDirectStream(ssc, ["logs"], {"metadata.broker.list": "localhost:9092"})

# Step 4: Extract log messages from Kafka stream (assuming logs are JSON-formatted strings)
parsed_logs = kafka_stream.map(lambda x: x[1])  # Extract log message from Kafka tuple

# Step 5: Filter only error logs from the stream
error_logs = parsed_logs.filter(lambda log: "ERROR" in log)

# Step 6: Save filtered error logs to HDFS for further analysis
error_logs.saveAsTextFiles("hdfs://namenode:8020/logs/errors")

# Step 7: Start the Spark Streaming job
ssc.start()
ssc.awaitTermination()  # Keeps the streaming process running

Explanation:

1. Initialize Spark Session: Creates a SparkSession with a checkpoint directory for fault tolerance.

2. Set up Streaming Context: Defines a batch interval of 5 seconds, meaning Spark processes data every 5 seconds.

3. Connect to Kafka Topic: Reads real-time log messages from Kafka’s "logs" topic.

4. Extract Log Messages: Kafka produces messages as (key, value) tuples, and we extract only the log text.

5. Filter Error Logs: Identifies logs containing the keyword "ERROR".

6. Write to HDFS: Saves processed logs to HDFS for further analysis or visualization.

7. Start the Streaming Job: Begins the real-time data pipeline and continuously processes new logs.

In Spark 3.x, KafkaUtils.createDirectStream() is deprecated, so users should consider using Structured Streaming with Kafka via spark.readStream() for more efficient and flexible stream processing.

Expected Output (HDFS Output File):

After running the script, you will find filtered error logs in HDFS:

hdfs dfs -ls /logs/errors

Sample log output in HDFS:

ERROR 2025-02-05 12:30:45 [Server] Connection timeout occurred.
ERROR 2025-02-05 12:30:47 [Database] Query execution failed.
ERROR 2025-02-05 12:30:52 [Auth] Unauthorized access detected.

This real-world Apache Spark Streaming tutorial showcases how big data processing pipelines handle real-time log analysis, making it a must-have skill for data engineers and AI professionals. 

Also Read: How to Become a Big Data Engineer: An Ultimate Guide

Now that you’ve learned how to set up and run Spark Streaming, it's important to understand why it's preferred over traditional batch processing. Let’s explore its key benefits and the challenges of real-world implementation.

Why Choose Apache Spark Streaming: Key Benefits and Challenges

As businesses increasingly rely on real-time analytics, the need for a scalable, fault-tolerant, and high-performance streaming framework has become critical. 

Traditional batch processing systems, such as Hadoop MapReduce, cannot process data in real time, making them unsuitable for real-world applications.

Apache Spark Streaming bridges this gap by providing a distributed, micro-batch processing engine that enables real-time data ingestion, transformation, and analysis at scale. 

Below are some key benefits of using Spark Streaming:

1. Real-Time Processing Capabilities

• Processes continuous data streams in near real-time with low latency.
• Ideal for financial transactions, social media monitoring, and IoT event processing.

2. Seamless Integration with the Spark Ecosystem

• Works alongside Spark SQL, MLlib (Machine Learning), and GraphX for advanced analytics.
• Supports structured streaming, making it easy to integrate with existing batch workflows.

3. Scalability and Fault Tolerance

• Can scale horizontally across hundreds or thousands of nodes for high-throughput workloads.
• Uses RDD lineage and checkpointing for automatic fault recovery.

4. Wide Range of Input and Output Sources

• Reads from Kafka, Flume, Kinesis, HDFS, S3, and NoSQL databases.
• Outputs to HDFS, Cassandra, Elasticsearch, JDBC databases, and real-time dashboards.

5. Open Source and Strong Community Support

• Backed by a large open-source community with frequent updates.
• Supported by major cloud providers (AWS, Azure, GCP) for enterprise-level streaming applications.

Despite its advantages, deploying Spark Streaming in production presents certain challenges. Below is a comparison of key challenges and their solutions:

By addressing these challenges proactively, organizations can build robust, real-time analytics pipelines that drive data-driven decision-making.

Also Read: Future of Data Analytics in India: Trends & Career Options 2025

While Spark Streaming offers scalability, real-time processing, and fault tolerance, how does it apply in real-world scenarios? Let’s look at how industries leverage Spark Streaming for real-time analytics.

Applications and Use Cases of Apache Spark Streaming

Apache Spark Streaming is widely used in real-world applications where low-latency, high-throughput data processing is critical. It enables businesses to analyze, detect patterns, and respond to events in real-time, making it a powerful tool for fraud detection, monitoring, and predictive analytics.

Below are some of the most impactful use cases of Apache Spark Streaming across industries:

1. Real-Time Log Monitoring

• Continuously processes server logs, application logs, and network activity to detect anomalies.
• Used in cybersecurity for threat detection and intrusion prevention.
• Companies like Netflix and LinkedIn use Spark Streaming to monitor system health and prevent failures.

2. Fraud Detection

• Analyzes banking transactions in real-time to detect suspicious activities.
• Financial institutions use machine learning models with Spark Streaming to flag anomalous credit card transactions.
• Helps reduce identity theft, money laundering, and online payment fraud.

Also Read: Fraud Detection in Machine Learning: What You Need To Know

3. Social Media Analytics

• Processes real-time data from platforms like Twitter, Facebook, and Instagram, with Twitter’s Firehose API and Kafka seamlessly working together to stream massive volumes of data for real-time analytics and trend detection.
• Identifies trending topics, customer sentiment, and brand perception.
• Used in digital marketing to track campaign performance in real-time.

Also Read: How to Make Social Media Marketing Strategy in Just 8 Steps

4. IoT Data Processing

• Ingests sensor data from IoT devices in smart cities, healthcare, and manufacturing.
• Enables predictive maintenance in industries by detecting failures before they happen.
• Used in self-driving cars to process sensor inputs and make instant decisions.

Also Read: Top 50 IoT Projects For all Levels in 2025 [With Source Code]

5. Stock Market Analysis

• Processes high-frequency trading data to identify market trends.
• Hedge funds and investment firms use Spark Streaming to predict stock movements and execute trades in milliseconds.
• Helps in risk assessment and portfolio management by analyzing real-time market fluctuations.

Its ability to ingest, process, and analyze massive streams of data with low latency is shaping the future of AI-driven decision-making. 

Also Read: Top 10 Apache Spark Use Cases Across Industries and Their Impact in 2025

With Spark Streaming transforming industries like finance, IoT, and social media analytics, mastering it is a valuable skill. upGrad’s courses provide hands-on training to help you build industry-ready expertise in real-time data processing.

How Can upGrad Help You Learn Apache Spark Streaming?

upGrad, South Asia’s leading Higher EdTech platform, equips 10M+ learners with big data and real-time analytics skills. Their courses offer hands-on training in stream processing, Kafka integration, and distributed computing.

You'll master fault tolerance, scalability, and performance tuning, gaining industry-ready expertise to build real-time data pipelines.

Here are some relevant courses you can check out:

• Introduction to Data Analysis using Excel
• Analyzing Patterns in Data and Storytelling
• Executive Diploma in Data Science & AI
• Post Graduate Certificate in Machine Learning and Deep Learning (Executive)
• Post Graduate Certificate in Machine Learning & NLP (Executive)

You can also get personalized career counseling with upGrad to guide your career path, or visit your nearest upGrad center and start hands-on training today!

Unlock the power of data with our popular Data Science courses, designed to make you proficient in analytics, machine learning, and big data!

Explore our Popular Data Science Courses

Elevate your career by learning essential Data Science skills such as statistical modeling, big data processing, predictive analytics, and SQL!

Top Data Science Skills to Learn

Stay informed and inspired  with our popular Data Science articles, offering expert insights, trends, and practical tips for aspiring data professionals!

Read our popular Data Science Articles