Grafana: Dashboards, Data Sources & Alerting

Grafana

Grafana is the open-source visualization and dashboarding layer of modern observability. It connects to many data sources — Prometheus for metrics, Loki for logs, tracing backends — and renders them as dashboards, with built-in alerting on top.

It's the "single pane of glass" that ties metrics, logs, and traces together. Its value comes less from the tool itself than from the discipline of building focused, owned dashboards that answer real questions about service health.

TL;DR

Build dashboards and alerts over Prometheus, Loki, and other data sources.
Default a service dashboard to the golden signals (latency, traffic, errors, saturation).
Use templating variables for service/env, and version dashboards as code.
Keep alerts actionable and dashboards owned — ownerless ones rot.

Quick Example

A panel query for p95 latency (PromQL), templated by a $service variable:

Core Concepts

Dashboard / panel — a collection of visualizations; each panel runs a query.
Data source — where data comes from (Prometheus, Loki, Tempo, SQL, cloud APMs).
Templating variables — dropdowns ($service, $env) that make one dashboard reusable.
Alerting — rules evaluated against queries, routed to notification channels.

A Good Service Dashboard

Default to the golden signals:

Latency — p50/p95/p99.
Traffic — requests/sec, queue depth.
Errors — rate and top error types.
Saturation — CPU/memory, DB connections, worker utilization.

If your team uses SLOs, put the SLO burn rate front and center.

Best Practices

Version dashboards as code (JSON/Terraform/Grafonnet) or keep a single source of truth.
Use templating so one dashboard serves every service/environment.
Assign each dashboard an owner — unowned dashboards drift and mislead.
Keep metric labels/tags consistent across services so panels compose.

Common Mistakes

Ownerless dashboards

Inconsistent labels across services

FAQ

What's the difference between Grafana and Prometheus?

Prometheus collects, stores, and queries metrics; Grafana visualizes them (and data from many other sources). Prometheus is the database and query engine; Grafana is the dashboard and alerting UI on top. They're commonly used together but are separate tools.

What should I put on a service dashboard?

Start with the four golden signals — latency, traffic, errors, saturation — plus your SLO burn rate if you have one. Resist the urge to add every available metric; a focused dashboard that answers "is this service healthy?" beats a wall of graphs.

Can I manage dashboards as code?

Yes, and you should for anything important. Export dashboards as JSON and store them in version control, or define them with tools like Terraform or Grafonnet. This makes them reviewable, reproducible, and resistant to accidental UI changes.

What data sources does Grafana support?

Many: Prometheus and other metrics stores, Loki and Elasticsearch for logs, Tempo/Jaeger for traces, SQL databases, and cloud provider monitoring. This is what lets Grafana unify metrics, logs, and traces in one place.