Why Use Kafka?

Apache Kafka is a distributed data streaming platform designed for high-throughput, real-time data processing. It is used to handle continuous streams of data from many different sources at the same time.

Key features of Kafka:

Scalability: Kafka distributes data across multiple servers, allowing it to handle massive amounts of information beyond a single machine's capacity.
Speed: With its decoupled write and read data streams, Kafka offers extremely low latency, making it ideal for real-time applications.
Durability: Data is replicated across servers and written to disk, protecting against server failures.
Flexible data retention: Kafka can store data for configurable periods, from seconds to years.
Multiple consumer support: Unlike traditional message queues, Kafka allows multiple applications to read the same data stream independently.

Kafka's architecture combines messaging queue and publish-subscribe models. It organizes data into topics that are partitioned. The messages within a partition are processed in order.

Kafka is used to build real time data pipelines. For example, Uber drivers send their location to the server every few seconds. This location is added to a Kafka queue. From the queue, the location is read by different consumers for applications like matching drivers with riders or showing nearby drivers to the riders on the app.

How to Scale Kafka in your System Design Interview?

Your e-commerce system processes 1,000 orders/second. Black Friday hits - suddenly it's 50,000 orders/second. How do you scale Kafka?

🧩 𝗔𝗱𝗱 𝗽𝗮𝗿𝘁𝗶𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗹 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 : Partitions are independent, ordered logs. Each partition guarantees order within itself. Customer A's orders in partition 7 stay in sequence, while Customer B's orders in partition 23 process in parallel. More partitions = more throughput.

🖥️ 𝗦𝗰𝗮𝗹𝗲 𝗯𝗿𝗼𝗸𝗲𝗿𝘀 𝗵𝗼𝗿𝗶𝘇𝗼𝗻𝘁𝗮𝗹𝗹𝘆 : Brokers are Kafka servers that host partitions. Running 50 partitions on 3 brokers? Add more brokers to spread the load. Kafka automatically rebalances partitions across all brokers.

👥 𝗦𝗰𝗮𝗹𝗲 𝗰𝗼𝗻𝘀𝘂𝗺𝗲𝗿 𝗴𝗿𝗼𝘂𝗽𝘀 : You can have up to one consumer per partition. With 50 partitions, you can run up to 50 parallel consumers. Each sees events in order within their partition. Need more parallelism? Add partitions.

𝗧𝗵𝗲 𝗸𝗲𝘆 𝘁𝗿𝗮𝗱𝗲𝗼𝗳𝗳: More partitions = more throughput BUT ordering only within each partition, not globally.