Kafka, From Zero
You keep hearing that Kafka is "a message queue," so you go in expecting RabbitMQ with a different logo - and then nothing fits. Messages don't disappear when you read them. Two different services read the same data. You can rewind and read last Tuesday's events again. The "queue" model you brought with you fights you at every turn, and it feels like the tool is broken.
It isn't broken. Kafka is not a queue - it's a durable, append-only log, and almost every confusion you'll have comes from carrying queue intuitions into a log-shaped world. This guide swaps the mental model first. Once you see Kafka as "a giant shared notebook that nobody erases, where each reader keeps their own bookmark," topics, partitions, offsets, and consumer groups stop being trivia and start being obvious.
How to read this
Read the phases in order. Phase 1 replaces "queue" with "log" in your head - the one swap that makes the rest make sense. Phase 2 is the daily work: producing with keys, consuming in groups, committing offsets, and how partitions give you both ordering and parallelism. Phase 3 is production reality: delivery guarantees, why you get duplicates, idempotent consumers, rebalances, and retention. If the broader idea of async messaging is new to you, start with /guides/webhooks-and-message-queues; if you're wondering why a company builds around streams, see /guides/event-driven-architecture.
The phases
- Phase 1: It's a Log, Not a Queue - the mental model: append-only topics, partitions, offsets, and why reading doesn't delete anything.
- Phase 2: Producing and Consuming for Real - the everyday loop: keys and partitioning, consumer groups, committing offsets, and how to scale readers.
- Phase 3: Production Reality - delivery guarantees, duplicates, idempotent consumers, rebalances, retention, and when Kafka is the wrong tool.