OpenTelemetry
OpenTelemetry (OTel) is the vendor-neutral standard for instrumenting applications with telemetry — traces, metrics, and logs. You instrument your code once against OTel's APIs, then export the data to whatever backend you choose (Prometheus, Datadog, Jaeger, Grafana) without rewriting instrumentation.
That portability is the whole point: it ends vendor lock-in at the instrumentation layer. Its standout capability is distributed tracing — following a single request across services to see exactly where time and failures occur.
TL;DR
- Vendor-neutral instrumentation: write it once, export anywhere.
- Covers traces, metrics, and logs; distributed traces are the headline.
- Context propagation (trace IDs flowing across hops) is essential — and easy to break.
- The Collector receives, processes, and exports telemetry reliably.
Quick Example
Wrap an operation in a span; OTel propagates the trace context to downstream calls automatically:
Core Concepts
- Traces / spans — a span is one timed operation; spans link into a trace of a request's full path.
- Metrics — counters, gauges, histograms.
- Logs — structured logs correlated with traces (support varies by stack).
- Context propagation — passing trace/span IDs across service boundaries (HTTP headers, message metadata).
The Collector
The OpenTelemetry Collector is a separate process (sidecar, daemonset, or central gateway) that decouples your apps from backends. It can:
- Receive telemetry from services (OTLP and other formats).
- Process — batch, retry, add resource attributes, sample.
- Export to one or more backends, swappable without touching app code.
Best Practices
- Standardize service names and environment attributes across services.
- Propagate context through HTTP and messaging so traces stay connected across hops.
- Add spans at I/O boundaries (DB queries, external API calls) — that's where time goes.
- Choose a sampling strategy that keeps cost down without hiding failures (e.g. tail-based sampling of errors).
- Never put secrets/PII in spans or attributes.
Common Mistakes
Broken trace context
Sampling that hides failures
FAQ
What problem does OpenTelemetry solve?
It standardizes how you produce telemetry, so you instrument once and aren't locked to a vendor's SDK. Switch or add backends (Datadog, Jaeger, Grafana) by reconfiguring exporters — no re-instrumentation. It also unifies traces, metrics, and logs under one project.
What is context propagation and why does it matter?
It's passing the trace and span IDs from one service to the next (via HTTP headers or message metadata) so all the work for one request joins a single trace. If any hop drops the context, the trace fragments and you lose the end-to-end picture — the main value of tracing.
What is the OpenTelemetry Collector?
A standalone agent that receives telemetry from your services, processes it (batching, sampling, enrichment), and exports it to backends. It decouples apps from vendors, so you can change or fan out destinations centrally without redeploying instrumented code.
How should I sample traces?
Enough to control cost while keeping the traces you need. Head-based sampling decides up front (cheap but can drop errors); tail-based sampling decides after seeing the whole trace, so you can keep all errors and slow requests. Prefer tail-based for debugging value at scale.
Related Topics
- Monitoring & Alerting — Where traces fit
- Prometheus — Metrics backend
- Grafana — Visualizing traces and metrics
- Logging — Correlated logs
- Microservices — Why distributed tracing matters