# Kafka, From Zero > The distributed log everyone runs in production: topics, partitions, offsets, and consumer groups - append-only streams you can replay, not a traditional queue. --- # Kafka, From Zero You keep hearing that Kafka is "a message queue," so you go in expecting RabbitMQ with a different logo - and then nothing fits. Messages don't disappear when you read them. Two different services read the same data. You can rewind and read last Tuesday's events again. The "queue" model you brought with you fights you at every turn, and it feels like the tool is broken. It isn't broken. Kafka is not a queue - it's a **durable, append-only log**, and almost every confusion you'll have comes from carrying queue intuitions into a log-shaped world. This guide swaps the mental model first. Once you see Kafka as "a giant shared notebook that nobody erases, where each reader keeps their own bookmark," topics, partitions, offsets, and consumer groups stop being trivia and start being obvious. ## How to read this Read the phases in order. Phase 1 replaces "queue" with "log" in your head - the one swap that makes the rest make sense. Phase 2 is the daily work: producing with keys, consuming in groups, committing offsets, and how partitions give you both ordering and parallelism. Phase 3 is production reality: delivery guarantees, why you get duplicates, idempotent consumers, rebalances, and retention. If the broader idea of async messaging is new to you, start with [/guides/webhooks-and-message-queues](/guides/webhooks-and-message-queues); if you're wondering *why* a company builds around streams, see [/guides/event-driven-architecture](/guides/event-driven-architecture). ## The phases 1. [Phase 1: It's a Log, Not a Queue](01-its-a-log-not-a-queue.md) - the mental model: append-only topics, partitions, offsets, and why reading doesn't delete anything. 2. [Phase 2: Producing and Consuming for Real](02-producing-and-consuming.md) - the everyday loop: keys and partitioning, consumer groups, committing offsets, and how to scale readers. 3. [Phase 3: Production Reality](03-production-reality.md) - delivery guarantees, duplicates, idempotent consumers, rebalances, retention, and when Kafka is the wrong tool. --- # It's a Log, Not a Queue Picture the queue you already know: a line of messages, a worker grabs one off the front, processes it, and it's *gone*. One message, one consumer, then it vanishes. That model is burned into most people's heads, and it's exactly the model you have to put down before Kafka makes sense. Kafka doesn't hand out messages and delete them. It keeps a **log** - and the difference changes everything downstream. ## The one analogy: a shared notebook nobody erases Imagine a notebook where every line is an event, written in order, and **no line is ever crossed out**. Writers only ever append to the bottom. Readers each keep their own bookmark - a sticky note saying "I've read up to line 4,812." Two readers can sit at completely different pages. A reader can move their bookmark *backward* and re-read. Nothing a reader does affects the notebook or any other reader. That notebook is a Kafka **topic**. The line numbers are **offsets**. The bookmarks are what each consumer commits. ```text topic "orders" (an append-only log) offset: 0 1 2 3 4 5 ← next write goes here [evt] [evt] [evt] [evt] [evt] ▲ ▲ consumer-A bookmark consumer-B bookmark (reading offset 2) (caught up, at 5) ``` *What just happened:* writers only append on the right; each event gets the next offset and stays put. Consumer-A and consumer-B read the *same* events at their *own* pace, and neither one's progress removes anything. Reading is just "advance my bookmark," not "remove from the list." This is the whole shift. In a queue, consuming is destructive. In Kafka, consuming is **reading a number off a log you don't own.** A message can be read by five different teams, and replayed next month, because it was never anyone's to delete. ## Topics, partitions, offsets - the three nouns These three words carry most of Kafka. Get them and you're past the hard part. **Topic** - a named stream of events, like a table name or a channel: `orders`, `payments`, `clicks`. You produce to a topic and consume from a topic. It's the notebook's title. **Partition** - here's the twist a plain notebook doesn't have. A topic isn't one log; it's split into several parallel logs called **partitions**. Each partition is its own append-only sequence with its own offsets starting at 0. ```text topic "orders" with 3 partitions partition 0: [0][1][2][3][4] partition 1: [0][1][2] partition 2: [0][1][2][3] ``` *What just happened:* "orders" is really three independent logs. Offset 2 in partition 0 has nothing to do with offset 2 in partition 1 - offsets are *per partition*, never global. This is why "what's the offset of this message" only makes sense once you also name the partition. **Offset** - the position of an event within one partition. It's a number that only ever goes up as new events are appended. A consumer's whole job, bookkeeping-wise, is "which offset am I at in each partition?" 💡 **Key point.** Ordering in Kafka is guaranteed **within a partition**, never across the whole topic. The events in partition 0 are strictly in order; but partition 0 and partition 1 are independent races. If two events *must* be processed in order relative to each other, they have to land in the same partition - which is exactly what message keys are for (Phase 2). ## Why split a topic into partitions at all? Two reasons, and they're the reasons Kafka scales when a single queue can't. **Parallelism.** One log can only be appended to and read in one sequence - that caps your throughput. Split it into 12 partitions and you have 12 logs that can be written and read at the same time, across many machines (called **brokers**). Partitions are the unit of parallelism: more partitions, more consumers can work at once. **Ordering where it matters, freedom where it doesn't.** You almost never need *global* order across millions of events. You need order *per customer*, or *per account*, or *per device*. Partitions let you say "all events for customer 7 go to the same partition, so they stay in order" while customers 7 and 8 are processed completely in parallel. ```mermaid flowchart LR P[Producer] -->|key=cust-7| Pa0[partition 0] P -->|key=cust-8| Pa1[partition 1] P -->|key=cust-9| Pa2[partition 2] Pa0 --> C[Consumers read
in parallel] Pa1 --> C Pa2 --> C ``` *A topic fans out into partitions so many consumers work at once, while a key pins related events to one partition to preserve their order.* ## What "durable" buys you: replay Because nothing is deleted on read, Kafka can do something a queue fundamentally can't: **replay**. The log sits on disk (replicated across brokers for safety) and stays there for a configured retention window - days, weeks, or forever. That means: - A brand-new service can start reading from offset 0 and process *all of history* to build up its own view. - A consumer with a bug can fix the bug, reset its bookmark back, and re-process the events it mangled. - Two teams can read the same `payments` topic for totally different purposes without coordinating. > 📝 **Terminology.** A **broker** is one Kafka server. A **cluster** is several brokers working together; partitions are spread across them and **replicated** so a broker dying doesn't lose data. You don't need to manage this by hand to start - but it's why Kafka is called *distributed*: the log lives on many machines, not one. ## For builders If you've only ever used a queue, the instinct is "I read it, so it's handled, so it's gone." Kill that instinct early. In Kafka, *handled* and *removed* are unrelated. A message stays in the log whether you processed it perfectly or crashed mid-way - and that's a feature, because it means a crash never loses data: you resume from your last committed offset. The flip side, which Phase 3 makes concrete, is that "resume from last offset" can mean re-reading a few events, so your processing needs to tolerate seeing the same event twice. ## Recap 1. Kafka is a **durable, append-only log**, not a queue - reading doesn't delete anything. 2. A **topic** is a named stream; it's split into **partitions**, each an independent log with its own **offsets** starting at 0. 3. Ordering is guaranteed **within a partition**, not across a topic. 4. Partitions exist for **parallelism** (many consumers at once) and **per-key ordering** (related events stay together). 5. Because the log persists, consumers keep their *own* position and can **replay** - one stream feeds many readers. ```quiz [ { "q": "In Kafka, what happens to a message after a consumer reads it?", "choices": [ "It is deleted from the partition immediately", "It stays in the log; the consumer just advances its own offset", "It moves to a dead-letter topic", "It is locked so no other consumer can read it" ], "answer": 1, "explain": "Kafka is an append-only log. Reading advances a per-consumer offset; the message remains until retention expires, which is what makes replay possible." }, { "q": "Within which scope does Kafka guarantee message ordering?", "choices": [ "Across the entire cluster", "Across a whole topic", "Within a single partition", "Across all partitions with the same offset" ], "answer": 2, "explain": "Ordering holds only within one partition. Different partitions are independent, so cross-topic or cross-partition order is not guaranteed." }, { "q": "Why is a topic split into multiple partitions?", "choices": [ "To make messages disappear faster after reading", "To enable parallelism and keep related (same-key) events ordered together", "To encrypt each partition separately", "To guarantee a single global order across the topic" ], "answer": 1, "explain": "Partitions are the unit of parallelism - many can be read at once - and keys route related events to the same partition so their order is preserved." } ] ``` --- # Producing and Consuming for Real You've got the model: a partitioned, append-only log with per-consumer bookmarks. Now for the part you'll actually do every day - putting events in and getting them out, at scale, without losing your place. The two questions this phase answers are the two that trip everyone up: **which partition does my message go to?** and **how do I run ten consumers without each one re-processing everything?** ## Producing: a message is a key, a value, and a destination A Kafka record is mostly three things: a **key** (optional), a **value** (your payload, usually JSON or Avro bytes), and the **topic** it's headed to. You hand it to a producer; the producer figures out the partition and sends it. Let's send one with the built-in CLI so there's no client library in the way: ```bash # Produce two messages to the "orders" topic. # Format here is key:value , split on the first colon. kafka-console-producer \ --bootstrap-server localhost:9092 \ --topic orders \ --property "parse.key=true" \ --property "key.separator=:" > cust-7:{"order":"A100","amount":42} > cust-7:{"order":"A101","amount":18} ``` *What just happened:* both records carry the key `cust-7`. Kafka hashes the key to choose a partition, and **the same key always hashes to the same partition** - so both of customer 7's orders land in the same partition, in the order you sent them. That's the mechanism behind "per-key ordering" from Phase 1. **The partitioning rule, plainly:** - **Key present** → partition = `hash(key) % number_of_partitions`. Same key, same partition, order preserved. - **No key** → the producer spreads records across partitions (round-robin-ish) for even load. Fast, but you give up ordering between those records. 💡 **Key point.** The key is not an ID you look records up by - Kafka has no "get message by key." The key exists for *one* job: deciding the partition, and therefore deciding what stays ordered together. Choose it to match your ordering need: `user_id` if per-user order matters, `account_id` for per-account, and so on. ⚠️ **Watch out.** Adding partitions later *changes* `hash(key) % N`, so the same key can start landing in a different partition than before. Existing ordering guarantees for in-flight keys break at that moment. Pick a partition count with growth in mind; changing it is not free. ## Consuming: read the log, track your offset A consumer subscribes to a topic and pulls records in offset order from each partition it's assigned. Reading them is the easy half. The half that matters is **committing offsets** - telling Kafka "I've successfully handled up to here," so that if you restart you resume from the right spot instead of from 0 or from a random place. ```bash # Read "orders" from the very beginning, in a named group. kafka-console-consumer \ --bootstrap-server localhost:9092 \ --topic orders \ --group billing \ --from-beginning {"order":"A100","amount":42} {"order":"A101","amount":18} ``` *What just happened:* the consumer joined the group `billing`, read from offset 0 because that group had no saved position yet, and as it read it committed its progress under the name `billing`. Restart this exact command and it picks up after A101 - not from the beginning - because the *group's* committed offset is remembered by Kafka. > 📝 **Terminology.** A committed offset is stored per **(group, topic, partition)**. It is "the next offset this group will read." Committing is a deliberate act - commit too early and a crash makes you *skip* unprocessed records; commit too late and a crash makes you *re-read* records. Phase 3 is about living with that trade-off in practice. ## Consumer groups: the scaling move Here's the feature that makes Kafka a workhorse. A **consumer group** is a set of consumers that share the work of one topic by **dividing the partitions among themselves**. Each partition is handled by exactly one member of the group at a time - so adding members adds throughput, up to the partition count. ```mermaid flowchart LR subgraph T["topic: orders (4 partitions)"] P0[p0] P1[p1] P2[p2] P3[p3] end P0 --> A[consumer A] P1 --> A P2 --> B[consumer B] P3 --> B ``` *Group "billing" has two consumers; Kafka assigns 2 partitions to each. Add a third consumer and the assignment rebalances to roughly 1–2 partitions each.* Two rules fall straight out of this picture: 1. **Parallelism is capped by partitions.** A group can usefully have at most as many active consumers as the topic has partitions. A 4-partition topic with 6 consumers leaves 2 consumers idle - there's nothing left to assign them. 2. **Different groups are independent.** The `billing` group and an `analytics` group each get *their own* copy of every record and *their own* committed offsets. That's Phase 1's "one stream, many readers" made concrete: each group reads the full topic without stepping on the other. ```bash # Inspect a group: see lag (how far behind it is per partition). kafka-consumer-groups \ --bootstrap-server localhost:9092 \ --describe --group billing # GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID # billing orders 0 812 815 3 consumer-A # billing orders 1 540 540 0 consumer-A ``` *What just happened:* **lag** = `LOG-END-OFFSET − CURRENT-OFFSET`, i.e. how many records have been written that this group hasn't read yet. Partition 0 is 3 behind; partition 1 is caught up. Lag is *the* number you watch in production - steadily climbing lag means your consumers can't keep up with producers. ## In the wild A typical setup: one `orders` topic with, say, 12 partitions, produced to with `customer_id` as the key. The `billing` group runs 4 consumers (each handling 3 partitions) to charge cards in per-customer order. Separately, an `analytics` group of 2 consumers reads the *same* topic to update dashboards, committing its own offsets, completely unaware billing exists. When Black Friday traffic spikes, billing scales from 4 to 8 consumers and Kafka rebalances partitions across them automatically - no producer change, no analytics change. That elasticity, with ordering preserved per customer, is the everyday payoff of the partition model. ## Recap 1. A record is a **key + value + topic**; the key's hash picks the partition, so same key → same partition → preserved order. 2. **No key** spreads records across partitions for even load but drops ordering between them. 3. **Committing offsets** is how a group remembers what it processed; commit timing decides whether a crash makes you skip or re-read. 4. A **consumer group** splits a topic's partitions among its members - the way you scale reading. 5. Parallelism is **capped by partition count**; **different groups** each read the whole topic independently. Watch **lag** to know if you're keeping up. ```quiz [ { "q": "You produce records with key=\"user-42\". What does Kafka do with that key?", "choices": [ "Stores it so you can later fetch the record by key", "Hashes it to choose a partition, keeping all user-42 records in order together", "Uses it to encrypt the message value", "Routes the record to whichever consumer is least busy" ], "answer": 1, "explain": "The key's only job is partition selection via hash(key) % N. Same key lands in the same partition, preserving order for that key. Kafka has no get-by-key." }, { "q": "A topic has 4 partitions and a consumer group has 6 consumers. What happens?", "choices": [ "All 6 consumers split each partition into pieces", "4 consumers each get one partition; 2 sit idle with nothing assigned", "Kafka automatically creates 2 more partitions", "The group fails to start because consumers outnumber partitions" ], "answer": 1, "explain": "Each partition goes to exactly one group member, so parallelism caps at the partition count. The extra 2 consumers have no partition to handle and stay idle." }, { "q": "What does consumer lag tell you?", "choices": [ "How long messages are retained on disk", "How many records have been produced but not yet read by that group", "The network latency between brokers", "How many consumers are in the group" ], "answer": 1, "explain": "Lag is log-end-offset minus the group's current offset - the backlog of unread records. Steadily rising lag means consumers can't keep up with producers." } ] ``` --- # Production Reality Everything works on your laptop with one consumer and ten messages. Production is where the log model earns its keep - and where the sharp edges live. None of these are bugs; they're the plain consequences of "durable distributed log." Know them in advance and they're routine. Meet them at 2am during an incident and they're a very bad night. ## Delivery guarantees: you'll almost always get at-least-once There are three theoretical guarantees. Here's what each actually means and which one you live with. - **At-most-once** - commit the offset *before* processing. If you crash mid-process, you've already moved your bookmark past the record, so it's never retried. You can **lose** records. Rare in practice; nobody wants silent data loss. - **At-least-once** - process the record, *then* commit the offset. If you crash after processing but before committing, you resume from the old offset and **re-process** the record. You can get **duplicates**, never loss. **This is the default and the one you should design for.** - **Exactly-once** - no duplicates, no loss. Kafka supports it for specific Kafka-to-Kafka flows (idempotent producer + transactions), but the moment your consumer touches an outside system - a database, an email, a payment API - you're back to designing for at-least-once at that boundary. ```text at-least-once timeline (the common case): read offset 50 → process (charge card ✅) → CRASH before commit restart → resume from offset 50 → process AGAIN → card charged twice ❌ ``` *What just happened:* the record was handled correctly, but the crash landed in the gap between "did the work" and "saved my place," so the work runs a second time. Kafka did nothing wrong - it just resumed from the last *committed* offset. This is why the next section isn't optional. 💡 **Key point.** "At-least-once" is not a setting you can flip off. It's the physics of "do work, then record that you did it" with a crash possible in between. The fix is never "make Kafka stop sending duplicates" - it's "make processing the same record twice harmless." That property has a name: idempotency. ## Idempotent consumers: the real-world fix An operation is **idempotent** if doing it twice has the same effect as doing it once. If your consumer is idempotent, duplicates stop mattering, and your whole system gets dramatically simpler. Three common ways: **1. Use a natural unique key + upsert.** Instead of `INSERT`, write `INSERT ... ON CONFLICT DO NOTHING` (or an upsert) keyed on something stable in the event, like `order_id`. The second arrival hits the conflict and does nothing. ```sql -- Re-processing the same event is a no-op: the order_id already exists. INSERT INTO orders (order_id, amount, status) VALUES ('A100', 42, 'paid') ON CONFLICT (order_id) DO NOTHING; ``` *What just happened:* the first time, the row is inserted. The duplicate from the at-least-once retry hits the unique `order_id`, the `ON CONFLICT` clause swallows it, and the order isn't double-counted. The duplicate became harmless without Kafka doing anything special. **2. Track processed IDs.** Keep a table (or Redis set) of event IDs you've already handled; skip any you've seen. More work, but handles side effects that aren't a simple row write. **3. Make the side effect itself idempotent.** Many external APIs accept an *idempotency key* - send the same key twice and the second call is ignored. Pass the event's ID as that key when you charge the card or send the email. > 📝 **Terminology.** "Make it idempotent" is the single most useful sentence in event-driven work. It moves the burden from "guarantee each message is delivered exactly once" (very hard, distributed) to "guarantee processing twice is safe" (a local design choice you fully control). The broader pattern is covered in [/guides/event-driven-architecture](/guides/event-driven-architecture). ## Rebalances: when the group reshuffles Whenever group membership changes - a consumer joins, leaves, or is presumed dead because it stopped checking in - Kafka triggers a **rebalance**: it re-divides the partitions among the surviving members. This is the mechanism that lets you scale up and recover from crashes. The catch: - During a rebalance, consumption **pauses** briefly while partitions are reassigned. Frequent rebalances mean choppy throughput. - The usual cause of *surprise* rebalances is a consumer that takes too long between polls. Kafka has a heartbeat and a max-poll interval; if your processing of a batch exceeds it, the broker decides the consumer is dead, kicks it out, and rebalances - even though it was alive and only slow. ⚠️ **Watch out.** A common production spiral: processing is slow → consumer misses its poll deadline → kicked out → rebalance → the records it was working on get reassigned and re-processed by someone else → still slow → repeat. The fixes are to shrink batch sizes, do heavy work asynchronously, or raise the max-poll interval - and to be idempotent so the re-processing during the churn is safe. ## Retention: the log is not infinite by default Kafka keeps records on disk, but not forever unless you say so. **Retention** governs when old records are deleted, and it's set per topic: - **Time-based** - e.g. keep 7 days; anything older is eligible for deletion. The default on many setups. - **Size-based** - e.g. keep the most recent 50 GB per partition. - **Compaction** - a different mode: instead of deleting by age, keep only the *latest* record per key. Useful when a topic represents current state ("the latest profile for each user") rather than a stream of events. ```bash # See a topic's retention settings. kafka-configs --bootstrap-server localhost:9092 \ --entity-type topics --entity-name orders --describe # retention.ms=604800000 (7 days) # cleanup.policy=delete (delete by age, vs "compact") ``` *What just happened:* this topic keeps records for 7 days (`604800000` ms) and uses the `delete` policy. After 7 days, old segments are removed - so "I can always replay from offset 0" is true *only within the retention window*. If you need full history forever, you must configure for it; don't assume it. ## When Kafka is the wrong tool Kafka is heavy. Reach for it when you genuinely have streams: high throughput, multiple independent consumers of the same data, replay, or event history. Don't reach for it when: - **You need per-message routing and complex delivery logic** (priorities, fan-out by routing rules, per-message acks/requeue). That's a classic broker's job - [/guides/webhooks-and-message-queues](/guides/webhooks-and-message-queues) covers that shape, and a tool like RabbitMQ fits it better. - **You have low volume and one consumer.** A database table or a simple queue is less to operate than a Kafka cluster. - **You need a request/response or a task that returns a result to the caller.** Kafka is one-directional event flow, not RPC. 💡 **Key point.** "Kafka vs a queue" is the wrong framing. They're different shapes: a queue is for *work to be done once by someone*; Kafka is for *events to be read by anyone, possibly more than once, possibly again later*. Match the tool to which sentence describes your problem. ## Recap 1. You almost always design for **at-least-once**: process then commit, accept possible **duplicates**, never lose data. 2. **Idempotent consumers** (upsert on a natural key, dedupe tables, idempotency keys) make duplicates harmless - that's the real fix, not chasing exactly-once. 3. **Rebalances** reassign partitions on membership change; slow processing causes surprise rebalances and re-processing. 4. **Retention** deletes old records by time, size, or compaction - replay only works *within* the window you configure. 5. Use Kafka for **streams, replay, and many readers**; use a **queue** for one-time routed work and a **DB/RPC** for low volume or request/response. ```quiz [ { "q": "Why do duplicate messages happen under the common at-least-once setup?", "choices": [ "Kafka intentionally sends every message twice for safety", "A consumer can crash after processing a record but before committing its offset, so it re-reads on restart", "Producers always send each record to two partitions", "Retention causes old messages to be re-delivered" ], "answer": 1, "explain": "At-least-once means process then commit. A crash in the gap between the two makes the consumer resume from the last committed offset and re-process - a duplicate, never a loss." }, { "q": "What is the practical fix for duplicate deliveries?", "choices": [ "Switch Kafka to at-most-once mode", "Make the consumer idempotent so processing the same record twice is harmless", "Reduce the partition count to 1", "Disable retention so messages can't be replayed" ], "answer": 1, "explain": "You can't eliminate at-least-once duplicates across an external side effect. Designing processing to be idempotent (upserts, dedupe, idempotency keys) makes duplicates safe." }, { "q": "A topic has retention.ms set to 7 days. What does that mean for replay?", "choices": [ "You can always replay from offset 0 no matter how old", "Records older than 7 days are eligible for deletion, so replay only works within that window", "Consumers are kicked from the group every 7 days", "Each consumer keeps its offset for only 7 days" ], "answer": 1, "explain": "Retention deletes records past the configured age. Replay from offset 0 is only possible for data still inside the retention window; older data is gone unless you configured longer retention or compaction." } ] ```