Apache Kafka: Topics, Partitions & Event Streaming

Apache Kafka

Apache Kafka is a distributed event-streaming platform: a durable, append-only log that producers write events to and consumers read from at their own pace. It was built at LinkedIn to move trillions of events per day, and it has become the backbone of event-driven architectures, data pipelines, and stream processing.

The mental shift from a message queue is that Kafka keeps the data: events aren't deleted when consumed, they're retained (hours, days, forever) and any number of consumers can replay them independently. The log becomes a shared source of truth that many systems read.

TL;DR

Kafka is a distributed, replicated commit log — not a queue. Messages persist for a retention period regardless of consumption.
Topics are named streams, split into partitions for parallelism; ordering is guaranteed only within a partition.
Messages with the same key always land in the same partition — pick keys to preserve the ordering you need (e.g. per-user).
Consumer groups scale reading: each partition is assigned to exactly one consumer in a group; different groups read independently.
Consumers track position via offsets; commit discipline determines your delivery guarantee (at-least-once is the practical default).
Design consumers to be idempotent — duplicates will happen.

Quick Example

Producing and consuming order events in Node.js (kafkajs):

The billing group and an analytics group can both read orders in full, at different speeds, without affecting each other.

Core Concepts

Topics, Partitions, and Offsets

A topic is a category of events. A partition is an ordered, immutable log within it.
Each message gets a sequential offset in its partition.
More partitions = more parallel consumers, but ordering only holds within one partition.
The message key is hashed to choose a partition — same key, same partition, preserved order.

Consumer Groups

A consumer group divides a topic's partitions among its members:

Different groups are fully independent — this is Kafka's fan-out model: publish once, consume by every interested system.

Delivery Guarantees

The pragmatic rule: build at-least-once + idempotent consumers. Deduplicate by event ID or use naturally idempotent writes (upserts).

Retention and Compaction

Time/size retention: keep events for N days or N GB, then drop the oldest segments — consumers replay history within the window.
Log compaction: keep at least the latest event per key forever — turns a topic into a durable changelog you can rebuild state from.

Kafka vs Message Queues

Rule of thumb: jobs ("resize this image, once") fit a message queue or background job system; facts ("order 1042 was created") fit Kafka, where many systems care and history matters.

Common Patterns

Event-Driven Microservices

Services publish domain events instead of calling each other directly. The order service emits order.created; billing, inventory, and notifications each consume it in their own group. Producers don't know consumers exist — see Event-Driven Architecture.

Change Data Capture (CDC)

Debezium tails a database's write-ahead log and publishes every row change to Kafka — turning PostgreSQL or MySQL into an event source without touching application code. Downstream: search indexing, cache invalidation, warehouse sync.

Stream Processing

Kafka Streams, Flink, or ksqlDB consume topics, transform/aggregate/join them, and produce results to new topics — fraud scoring, sessionization, realtime metrics.

The Outbox Pattern

To publish events atomically with a database write, insert the event into an outbox table in the same transaction, and let a relay (or CDC) publish it to Kafka. This avoids the "DB committed but publish failed" split-brain.

Best Practices

Choose Keys for the Ordering You Need

Ordering guarantees exist per partition only. If a user's events must be processed in order, key by userId. Beware hot keys — one huge customer keyed to one partition becomes a throughput ceiling.

Size Partitions for Target Parallelism

Partition count caps consumer parallelism and is painful to change later (re-keying shifts data). Start with headroom (e.g. 6–12 for a modest topic) rather than 1.

Use a Schema Registry

JSON blobs drift. Avro/Protobuf schemas with a registry (Confluent, Redpanda, Buf) give producers and consumers a checked contract with evolution rules — the same discipline as gRPC protos.

Monitor Consumer Lag Above All

Lag (latest offset minus committed offset) is the health metric: growing lag means consumers can't keep up. Alert on it — see Alerting and Monitoring.

Plan for Poison Messages

A message that always crashes the consumer blocks its whole partition. Catch processing errors, route failures to a dead-letter topic, and keep consuming.

Common Mistakes

Using Kafka as a Job Queue

Kafka has no per-message ack/retry/delay semantics — one slow message holds up its partition. Task distribution with retries and priorities is queue territory (RabbitMQ, SQS, Sidekiq).

Assuming Global Ordering

Consumers of a multi-partition topic see events interleaved across partitions. If you need total order, you need one partition (and its throughput limit) — usually the real fix is keying by the entity that needs order.

Forgetting Idempotency

At-least-once delivery will re-deliver during rebalances and retries. Consumers that blindly INSERT will duplicate rows; upsert or dedupe by event ID.

Unbounded Retention by Accident

Default retention plus high volume fills disks fast. Set explicit retention per topic and monitor broker disk usage.

FAQ

Is Kafka overkill for my app?

If you need "run this task later," yes — use a background job library. Kafka earns its operational cost when multiple systems consume the same events, you need replay/history, or throughput is genuinely high. Managed offerings (Confluent Cloud, AWS MSK, Redpanda Cloud) remove most of the ops burden if you're in between.

Does Kafka still need ZooKeeper?

No — modern Kafka runs in KRaft mode (built-in Raft consensus), and ZooKeeper support has been removed. New deployments are KRaft-only.

How is Kafka different from Redis Pub/Sub?

Redis Pub/Sub is fire-and-forget: subscribers offline at publish time miss the message. Kafka persists events, tracks per-group offsets, and replays. Redis Streams closes some of the gap for lighter workloads.

What throughput can Kafka handle?

Clusters routinely handle millions of messages per second; LinkedIn runs trillions of messages per day. For a single modest broker, tens of thousands of messages per second is unremarkable.

What are Kafka alternatives?

Redpanda (Kafka-protocol-compatible, C++, no JVM), Apache Pulsar (segmented storage, multi-tenancy), and cloud-native options like AWS Kinesis and Google Pub/Sub — the concepts here transfer to all of them.