Apache Kafka: Architecture, Concepts, Features & Applications

Kafka was launched in 2011, all thanks to LinkedIn. Since then, it has witnessed incredible growth to the point that most companies listed in Fortune 500 now use it. It is a highly scalable, durable and high-throughput product that can handle large amounts of streaming data. But is that the only reason behind its tremendous popularity? Well, no. We haven’t even got started on its features, the quality it produces, and the ease it provides to users.

We will dive into that later. Let’s first understand what Kafka is and where it is used. 

What is Apache Kafka?

Apache Kafka is a open-source stream-processing software that aims to deliver high-throughput and low-latency while managing real-time data. Written in Java and Scala, Kafka provides durability via in-memory microservices and has an integral role to play in maintaining supply events to Complex Event Streaming Services, otherwise known as CEP or Automation Systems. 

It is an exceptionally versatile, fault-proof distributed system, which enables companies like Uber to manage passenger and driver matching. It also provides real-time data and proactive maintenance for British Gas’ smart home products apart from helping LinkedIn in tracking multiple real-time services. 

Often employed in real-time streaming data architecture to deliver real-time analytics, Kafka is a swift, sturdy, scalable, and publish-subscribe messaging system. Apache Kafka can be used as a substitute for traditional MOM because of its excellent compatibility and flexible architecture that allows it to track service calls or IoT sensor data. 

Kafka works brilliantly with Apache Flume/Flafka, Apache Spark Streaming, Apache Storm, HBase, Apache Flink, and Apache Spark for real-time ingestion, research, analysis, and processing streaming data. Kafka intermediaries also facilitate low-latency follow-up reports in Hadoop or Spark. Kafka also has a subsidiary project named Kafka Stream that works as an effective tool for real-time analysis. 

Kafka Architecture and Components

Kafka is used for streaming real-time data to multiple recipient systems. Kafka works as a central layer for decoupling real-time data pipelines. It doesn’t find much use in direct computations. It is most compatible with fast lane feeding systems, real-time or operational data-based, to stream a significant amount of data for batch data analysis.

Storm, Flink, Spark, and CEP frameworks are a few data systems that Kafka works with to accomplish real-time analytics, creating backups, audits, and more. It can also be integrated with big data platforms or database systems like RDBMS, and Cassandra, Spark, etc, for data science crunching, reporting, etc. 

The diagram below illustrates the Kafka Ecosystem:

Source

Here are the various components of the Kafka ecosystem as illustrated in the Kafka architecture diagram:

1. Kafka Broker

Kafka emulates a cluster that comprises multiple servers, each known as a “broker.” Any communication among clients and servers adheres to a high-performance TCP protocol. It comprises more than one stateless broker to handle heavy loading. A single Kafka broker is capable of managing several lacs of reads and writes every second without compromising on the performance. They use ZooKeeper to maintain clusters and elect the broker leader. 

2. Kafka ZooKeeper

As mentioned above, ZooKeeper is in charge of managing Kafka brokers. Any new addition or failure of a broker in the Kafka ecosystem is brought to a producer or consumer’s notice via the ZooKeeper. 

3. Kafka Producers

They are responsible for sending data to brokers. Producers do not rely on brokers to acknowledge the receipt of a message. Instead, they determine how much a broker can handle and send messages accordingly.

4. Kafka Consumers

It is the responsibility of Kafka consumers to keep a record of the number of messages consumed by the partition offset. Acknowledging a message indicates that the messages sent before they have been consumed. To ensure that the broker has a buffer of bytes ready to send to the consumer, the consumer initiates an asynchronous pull request. The ZooKeeper has a role to play in maintaining the offset value of skipping or rewinding a message. 

Kafka’s mechanism involves sending messages between applications in distributed systems. Kafka employs a commit log, which when subscribed to publishes the data present to a variety of streaming applications. The sender sends messages to Kafka, while the recipient receives messages from the stream distributed by Kafka. 

Messages are assembled into topics — an effective deliberation by Kafka. A given topic represents organized steam of data based on a specific type or classification. The producer writes messages for consumers to read which are based on a topic.

Every topic is given a unique name. Any message from a given topic sent by a sender is received by all users who are tuning in to that topic. Once published, the data in a topic cannot be updated or modified. 

Features of Kafka

1. Kafka consists of a perpetual commit log that allows you to subscribe to it, and subsequently publish data to multiple systems or real-time applications. 
2. It gives applications the ability to control that data as it comes. The Streams API in Apache Kafka is a powerful, light-weight library that facilitates on-the-fly batch data processing. 
3. It is a Java application that allows you to regulate your workflow and significantly reduces any requirement of maintenance. 
4. Kafka functions as a “storage of truth” distributing data to multiple nodes by enabling data deployment via multiple data systems. 
5. Kafka’s commit log makes it a reliable storage system. Kafka creates replicas/backups of a partition which help prevent data loss (the right configurations can result in zero data loss). This also prevents server failure and enhances the durability of Kafka.
6. Topics in Kafka have thousands of partitions, making it capable of handling an arbitrary amount of data and heavy loading.
7. Kafka depends on the OS kernel to move data around at a fast pace. These clusters of information are end-to-end encrypted, producer to file system to end consumer.
8. Batching in Kafka makes data compression efficiency and decreases I/O latency.

Applications of Kafka

Plenty of companies who deal with large amounts of data daily use Kafka. 

1. LinkedIn uses Kafka to track user activity and performance metrics. Twitter combines it with Storm to enable a stream-processing framework. 
2. Square uses Kafka to facilitate the movement of all system events to other Square data centres. This includes logs, custom events, and metrics.
3. Other popular companies that avail the benefits of Kafka include Netflix, Spotify, Uber, Tumblr, CloudFlare, and PayPal.

Why Should you Learn Apache Kafka?

Kafka is an excellent event streaming platform that can efficiently handle, track and monitor real-time data. Its fault-tolerant and scalable architecture allow low-latency data integration resulting in a high throughput of streaming events. Kafka significantly reduces the “time-to-value” for data.

It works as the foundational system producing information to organizations by eliminating “logs” around data. This allows data scientists and specialists to easily access information at any point in time. 

For these reasons, it is the top streaming platform of choice for many top companies and therefore, candidates with a qualification in Apache Kafka are highly-sought after.

If you are interested to know more about Big Data, check out our Advanced Certificate Programme in Big Data from IIIT Bangalore.

Check our other Software Engineering Courses at upGrad.