Messaging & Event Streaming

A request/response API couples two systems in time: the caller waits, and if the callee is slow or down, the caller suffers. Messaging breaks that coupling. A producer writes a message and moves on; a consumer processes it when it can. That one change unlocks retries, load leveling, fan-out to many consumers, and replaying history.

It also introduces new problems — duplicate delivery, ordering, poison messages, and "where did my event go?" debugging. This hub covers the tools and the patterns that keep asynchronous systems correct.

TL;DR

• Queues distribute work; logs distribute history. RabbitMQ-style queues delete a message once it's handled. Kafka-style logs keep it so many consumers can read and replay it.
• Assume at-least-once delivery. Make consumers idempotent.
• Plan for failure paths — retries with backoff, dead-letter queues, and alerting on lag.
• Move slow work off the request path with background jobs.
• Use CDC instead of dual writes to get database changes into streams reliably.
• Reach for durable execution when a workflow spans minutes to months and must survive crashes.

Queue vs Log vs Workflow

Featured Topics

Foundations

• Message Queues & Event Streaming — Delivery semantics, retries, DLQs, and ordering
• Background Jobs & Workers — Moving slow work off the request path

Brokers

• Apache Kafka — Topics, partitions, consumer groups, and retention
• RabbitMQ — Exchanges, bindings, acknowledgments, and routing
• NATS — Lightweight pub/sub, request-reply, and JetStream persistence

Streams & Workflows

• Stream Processing — Windowing, watermarks, and stateful computation
• Change Data Capture — Streaming database changes with Debezium
• Durable Execution — Crash-proof workflows with Temporal-style engines

A Typical Event Pipeline

The application writes only to its database. CDC turns committed changes into events, so there's no dual-write race. Each downstream team reads the same topic with its own consumer group and its own pace.

Common Mistakes

🚫 Assuming exactly-once — Brokers can redeliver after a crash or rebalance. Deduplicate in the consumer.

🚫 Dual writes — Writing to the database and then publishing to Kafka loses events when the second step fails. Use an outbox table or CDC.

🚫 No dead-letter queue — One malformed message blocks the partition forever.

🚫 Ignoring consumer lag — Lag is the leading indicator that you're about to miss an SLA. Alert on it.

🚫 Kafka for a to-do list — A simple job queue is often all you need. Don't run a distributed log for ten jobs a minute.

Learning Path

Beginner

Add a background job queue to a web app and make the job idempotent.

Intermediate

Run RabbitMQ or Kafka locally, build producers and consumer groups, and add retries and a DLQ.

Advanced

Build a CDC pipeline into a stream processor, and model a multi-step business process with durable execution.

Related Topics

• Event-Driven Architecture — Architectural patterns built on messaging
• Saga Pattern — Distributed transactions across services
• APIs & Integration — The synchronous side of system integration
• Data Engineering & Analytics — Where many event streams end up