Distributed Tracing
Distributed tracing follows a single request as it travels across all the services that handle it, recording how long each step took and how they connect. In a microservices system, one user action might touch a dozen services — and when it's slow or fails, metrics tell you that something is wrong but not where, and logs are scattered across every service with no easy way to connect them. Tracing stitches the whole journey into one coherent picture, making it the tool that answers "where did this request spend its time, and which service broke?"
The core structure is simple: a trace represents one request end-to-end, composed of spans — timed operations (an API call, a DB query, a function) — linked into a parent-child tree. The magic that makes it work across process boundaries is context propagation: each service passes a trace id along so the spans can be reassembled into one trace. Once you've debugged a cross-service latency problem with a trace, you understand why it's indispensable at scale.
TL;DR
- A trace follows one request across all services; spans are its timed steps.
- Context propagation (passing a trace id) links spans across service boundaries.
- Essential for microservices — finds where a request is slow or failing.
- Use sampling to control volume; instrument with OpenTelemetry.
Quick Example
A trace is a tree of spans across services, each with a duration:
Core Concepts
- Trace — the complete record of one request's path through the system, identified by a trace id.
- Span — a single timed operation within the trace (a service handling the request, a DB query, an external call), with a start/end time, name, and metadata (tags).
- Parent-child relationships — spans nest to form a tree, showing what called what.
- Context propagation — the trace id (and span context) is passed between services, usually via HTTP headers, so spans created in different processes belong to the same trace.
Why Tracing Matters in Microservices
In a monolith, a stack trace and logs from one process often suffice. In a distributed system, a single request fans out across services, each logging independently — and metrics only show aggregate health. When checkout is slow, was it the order service, the payment service, the database, or an external API? Tracing answers this directly by showing the latency of each step in the actual request path, turning "something is slow somewhere" into "the payment service's external call is the bottleneck."
It also reveals the service dependency graph and surfaces problems (retries, fan-out, N+1 calls across services) that are invisible from any single service's vantage point.
Sampling
Tracing every request at high traffic is expensive in storage and overhead, so most systems sample:
- Head-based sampling — decide at the start (e.g. keep 1% of traces) — simple, but may miss rare errors.
- Tail-based sampling — decide after the trace completes, keeping interesting ones (errors, slow requests) — smarter, more complex.
The goal is to keep enough traces to debug problems without storing everything.
Instrumentation
- OpenTelemetry — the open standard for generating traces (and metrics/logs), vendor-neutral, with auto-instrumentation for common frameworks.
- Auto-instrumentation captures spans for HTTP, DB, and RPC calls with little code; manual spans add detail for your own important operations.
- Backends — traces are sent to a system like Jaeger, Tempo, or Datadog to store, search, and visualize.
Best Practices
- Propagate context everywhere — a broken propagation chain fragments the trace; use standard headers (W3C Trace Context).
- Instrument with OpenTelemetry for vendor-neutral, portable tracing.
- Sample intelligently — keep errors and slow requests (tail-based where possible).
- Correlate with logs — include the trace id in log lines so you can jump from a trace to its logs.
- Add meaningful span attributes — but avoid putting secrets/PII in spans.
Common Mistakes
Broken context propagation
Tracing everything at full volume
FAQ
What's the difference between a trace and a span?
A trace is the complete journey of one request through your system — everything that happened to handle it, identified by a single trace id. A span is one timed operation within that trace: a service processing the request, a database query, an external API call. Spans nest into a parent-child tree (the gateway span contains the order-service span, which contains the DB span), and together they form the trace. So a trace is the whole picture; spans are its building blocks, each showing how long one step took and how steps relate.
Why do I need distributed tracing if I already have logs and metrics?
Because they answer different questions, and neither handles cross-service requests well alone. Metrics show aggregate health ("error rate is up") but not where. Logs capture detailed events but are scattered across each service independently, with no built-in way to connect the lines belonging to one request. Distributed tracing stitches a single request's path across all services into one view, pinpointing exactly which service or call is slow or failing. In a microservices architecture, it's the only practical way to debug "this request is slow somewhere in the system."
What is context propagation?
Context propagation is how a trace stays connected as a request crosses service (and process) boundaries. When service A calls service B, A passes along the trace context — primarily the trace id and current span id — typically as HTTP headers (the W3C Trace Context standard). Service B reads that context and creates its spans as children of A's, so all the spans across all services can be reassembled into one trace. Without propagation, each service would create disconnected, orphan traces, and you'd lose the end-to-end picture — which is why a broken propagation chain is a common tracing bug.
Should I trace every request?
Usually not at scale — capturing 100% of traces in a high-traffic system is expensive in storage, network, and overhead. Instead, sample: head-based sampling keeps a fixed fraction (e.g. 1%) decided upfront, while tail-based sampling decides after a request finishes, preferentially keeping the interesting ones (errors, unusually slow requests). Tail-based is smarter because it retains the traces you actually need for debugging while discarding routine ones, but it's more complex to operate. The aim is enough trace coverage to diagnose problems without the cost of storing everything.
Related Topics
- Observability — The three pillars together
- Metrics & Time Series — Aggregate health signals
- Logging — Correlate logs with traces
- OpenTelemetry — The instrumentation standard
- Microservices — Why tracing is essential