← Back to the catalog

Apache Kafka Fundamentals: Partitions & Groups

Learn how Kafka's append-only log differs from a message queue: topics, partitions, offsets, replication and in-sync replicas. Then run a single-broker Kafka in Docker, produce keyed events with the console tools, and watch a consumer group split partitions, rebalance and commit offsets.

Language
English
Lessons
6
Minutes
180

Price: · The subscription includes the whole catalog at no extra cost.

Course outline

  1. How Kafka Stores Events

    1. The Log: Why Kafka Is Not a Queue (30′)
    2. Partitions and Offsets (30′)
    3. Replication: Leaders, Followers and Surviving a Crash (30′)
  2. Running It: Producers, Groups and Offsets

    1. Run Kafka in Docker and Create Your Topic (30′)
    2. Produce Keyed Events and Watch Where They Land (30′)
    3. Consumer Groups, Rebalance and Offsets in Action (30′)

Free first lesson

The Log: Why Kafka Is Not a Queue

An event is a record, not a bare message

In the queues you know, a message is mostly a body you ship to someone. In Kafka, every event is a record with three parts. A key, which says what the event is about: cust-17 for customer 17. A value, the payload itself: {"order": 913, "amount": 42.50}. And a timestamp, when it happened. Sending an event with no key is allowed too; the key is optional.

Why carry a key at all? Kafka uses it to keep events about the same thing together, so their order survives the trip. Lesson 2 shows exactly how the key picks a partition.

A queue deletes on delivery; a log appends

In the message queue you know, the broker hands each message to a consumer and forgets it: delivered means deleted. Kafka does the opposite. Every event is appended to a log — an ordered file where new events go on the end, and reading never removes anything.

The log belongs to a topic, the named home for one stream of events: orders-placed, payments-made. A consumer does not get events pushed at it. It reads the log like a file, keeping an offset — the number of the next event it has not read yet, like a bookmark. It reads a batch, processes it, saves (commits) its offset, reads on.

Why can two different teams read the same topic without getting in each other's way?

Retention: the log is a window, not forever

Because reading never removes anything and each consumer holds its own offset, one team can be an hour behind and another a day behind on the same events — neither affects the other.

So what cleans up? Retention does. Each topic has a retention setting, and Kafka's default is seven days (168 hours): once an event is older than that, the broker deletes it to free disk, read or not. A topic can also cap the log by size. Either way the log is a rolling window of recent history, not permanent storage. Check the Kafka documentation for your version before relying on a specific default.

Why not keep events forever? Disk is finite: retention is the trade-off between how far behind a consumer can fall and still catch up, and how much disk the topic eats.

Your billing service misses a day

Friday evening, a bug takes your billing consumer down. It comes back Sunday. In a deleting queue, Saturday's orders would be gone. In Kafka:

  1. The consumer restarts with the same group id (consumer groups arrive in lesson 6).
  2. It fetches its committed offset — say 48,121, the first event it had not yet read on Friday.
  3. It reads forward from there through everything appended while it was down.

Nothing was lost: reading never deletes, so the events just sat in the log. With the seven-day default, a 24-hour outage is nowhere near the limit.

One edge to know: if there is no committed offset — a brand-new group id — or the offset points to events retention has already deleted, Kafka falls back to auto.offset.reset. Its default, latest, starts at the end with only new events; earliest starts at the log's beginning.

Your turn: the ten-day outage

Same consumer, but this time the outage lasts ten days on a topic whose retention was set to three days. The bug is fixed; the consumer restarts with the committed offset it saved on day zero, and auto.offset.reset is at its default (latest).

Can it catch up completely? What exactly will it read? Write your answer before reading mine.

Mistakes that cost you the day you missed

  • Treating Kafka like a queue: nothing is removed by reading — only retention removes events.
  • Treating the log as permanent storage: past the retention window, events are gone, read or not.
  • Expecting a second consumer to steal events: each keeps its own offset, so both read the same stream.

What to carry forward

  • Reading never deletes; only retention removes events.
  • Each consumer keeps its own offset and resumes from it.
  • Retention — seven days by default — caps how far back it can recover.

Think it through: the Monday restart

Your alerting consumer crashed Friday night and returns Monday morning; the topic uses the default retention. In two sentences: what does it read on restart, and which single setting decides whether it could have caught up at all?

event record
The data Kafka stores: a key (what the event is about, e.g. cust-17, optional), a value (the payload) and a timestamp (when it happened).
log
The ordered, append-only file a topic's events are written to: new events go on the end, and reading never removes anything.
topic
The named home for one stream of events, such as orders-placed; producers write to it, consumers read from it.
retention
The per-topic setting (time or size based, 7 days by default) that decides how long events stay in the log before the broker deletes them.
offset
A consumer's bookmark: the number of the next event it has not read yet. The consumer saves (commits) it as it goes.
consumer group
A set of consumers identified by a shared group id; Kafka stores the group's committed offsets so members can resume where they left off. Details in lesson 6.