Tracing Across Queues, Events & Async Jobs

Trace Correlation Across Queues and Async Work

Distributed tracing works well for synchronous request chains: an HTTP client injects a traceparent header, the server extracts it, and spans nest neatly. The moment work crosses a queue, an event stream, or a background job, many traces simply end. The checkout request shows a "publish order" step and nothing more; the fulfillment worker that processes the event an instant later starts a brand-new, unconnected trace. When a customer reports a missing confirmation email, nobody can follow the path from click to failure.

The fix is the same idea applied to messages: carry trace context inside the message, restore it in the consumer, and choose deliberately how asynchronous work relates to the work that caused it — as a child, or through a link.

TL;DR

Inject trace context (W3C traceparent, tracestate, and optionally baggage) into message headers or attributes when publishing.
Extract it in consumers and start a consumer span in that context.
Use parent-child relationships for one message processed promptly; use span links for batches, fan-in, retries, and long delays.
Follow OpenTelemetry messaging semantic conventions for span names, kinds, and attributes.
Put trace and span IDs in every log line so logs and traces correlate in both directions.
Prefer auto-instrumentation for Kafka, RabbitMQ, SQS, and job libraries, and fill gaps manually.

Quick Example

Propagating context through Kafka with OpenTelemetry in Python. The producer injects context into message headers:

The consumer extracts it and continues the same trace:

The trace now shows the HTTP request, the publish span, and the consumer's processing — with the queue wait visible as the gap between them.

Core Concepts

Context Propagation Through Messages

W3C Trace Context defines the traceparent header (trace ID, parent span ID, sampling flag) and tracestate for vendor data. For messaging, the same values travel as message headers (Kafka, RabbitMQ, NATS), message attributes (SQS, SNS, Pub/Sub), or job metadata (Celery, Sidekiq, BullMQ). OpenTelemetry's propagators handle encoding; instrumentation needs only a carrier that can read and write key-value pairs. See OpenTelemetry Context Propagation.

Span Kinds and Names

Attributes such as messaging.system, messaging.destination.name, messaging.operation.type, and messaging.message.id make spans searchable across systems. See OpenTelemetry Semantic Conventions.

Parent-Child Versus Links

Links keep traces a reasonable size and duration while preserving the causal path. A trace that stays open for three days because one email was delayed is hard to query and often dropped by backends.

Baggage and Business Identifiers

Baggage propagates key-value pairs such as tenant ID or experiment variant to every downstream service. It travels in plain headers to all participants, so never put secrets or personal data in it. Business identifiers like order.id usually belong as span attributes where you need to search by them.

Correlating Logs With Traces

Configure logging so every record includes trace_id and span_id; OpenTelemetry logging integrations do this automatically.
In consumers, log inside the processing span so log lines carry the restored context.
Log message IDs and destinations alongside trace IDs, so you can go from a dead-letter message to its trace.

See Logging and Log Aggregation.

Sampling and Partial Traces

Head sampling decisions travel in the traceparent sampled flag, so consumers that respect the parent's decision keep traces complete. Problems arise when producers and consumers use different samplers, when batch processing links to unsampled contexts, or when consumers run in separate pipelines with their own sampling. Tail sampling in a collector, keyed on trace ID, keeps whole traces — including async parts — when they contain errors or slow spans. See OpenTelemetry Sampling.

Best Practices

Start With Auto-Instrumentation

OpenTelemetry instrumentation exists for most Kafka clients, RabbitMQ libraries, cloud queue SDKs, and job frameworks. Enable it first, check which hops lose context, and add manual inject and extract only where needed.

Inject Context Where the Message Is Created

With the outbox pattern, store the trace context in the outbox row when the business transaction runs, and have the relay publish it — otherwise the relay's own trace replaces the original.

Use Links for Anything Not Directly Caused or Not Prompt

Batches, aggregations, retries, and long delays are clearer as linked traces than as enormous ones.

Watch Header Limits

Some systems limit message attributes (for example, SQS allows ten per message). Reserve room for trace context and keep baggage small.

Common Mistakes

Context Dropped at the Relay

One Giant Trace for a Batch

Secrets in Baggage

FAQ

Why do my traces stop at the message queue?

Because trace context is not being carried in the message. The producer must inject the context into message headers or attributes, and the consumer must extract it before starting its span — or use instrumentation libraries that do both.

Should the consumer span be a child of the producer span?

For a single message processed promptly as a direct result of the producer's work, yes. For batches, aggregations, retries, or work that runs much later, start a new span or trace and add links to the originating contexts.

What is a span link?

A reference from one span to another span's context, possibly in a different trace. Links express causality without making one span the parent of another, which suits batches and fan-in.

How do I trace scheduled or cron jobs?

Start a new root trace for each run. If the job processes items created by earlier requests, link to the contexts stored with those items.

Can I see queue wait time in traces?

Yes. The gap between the producer's publish span and the consumer's process span shows time spent in the queue, and some instrumentations record it explicitly as an attribute or metric.