Design A Distributed Message Queue

Design a durable, scalable message queue service like Apache Kafka or RabbitMQ.

Full Problem Statement

Design a distributed message queue system like Apache Kafka.

Core Requirements:

• Producers can publish messages to a topic.
• Consumers can subscribe to topics and consume messages.
• Messages should be persisted for durability.
• Support for multiple consumers reading from the same topic independently (consumer groups).
• Guarantee message ordering within a partition.

Scale Requirements:

• Handle millions of messages per second.
• Store petabytes of message data.
• High throughput for both producers and consumers.
• High availability and fault tolerance.

Additional Considerations:

• How would you achieve high throughput? (e.g., sequential I/O, batching).
• How would you partition topics to enable horizontal scaling?
• How would you handle consumer offsets to track progress?
• How would you design for data replication and fault tolerance?