Design Publish/Subscribe Messaging System

Design a pub/sub system like Kafka with durability, fanout, and ordering.

Full Problem Statement

Design a publish/subscribe messaging system similar to Google Pub/Sub or Apache Kafka.

Is kafka pub-sub or message-queue?

Kafka is both a publish-subscribe (pub-sub) system and a message queue, but it operates with characteristics of both, making it a more versatile streaming platform. It uses the pub-sub model for broadcasting messages to multiple consumers but also provides queue-like functionality because it stores messages in a distributed log that can be retained and replayed, unlike traditional message queues that often delete messages after they are consumed

Core Requirements:

Producers publish messages to topics.
Consumers subscribe and receive messages.
Support message ordering and durability.

Scale Requirements:

1 million messages/sec.
10,000 topics.
High availability and exactly-once semantics.

Additional Considerations:

How would you implement partitions and consumer groups?
What consistency guarantees would you offer?
How would you persist and replicate data?
How would you handle backpressure and dead-letter queues?

Design Publish/Subscribe Messaging System

Full Problem Statement

PlayGround