Vector: Observability Pipelines for Logs and Metrics

Q: How does Vector avoid losing data?

Disk buffers persist events across restarts and outages, block policies apply back pressure instead of dropping, sinks retry failed requests, and end-to-end acknowledgements confirm delivery for supporting sources.

Vector Observability Pipelines

Between the applications that emit telemetry and the systems that store it sits a pipeline: something must tail log files and container output, parse formats, add context, drop noise, remove secrets and personal data, and deliver events to one or more backends without losing them when a backend is slow or down. Vector is an open source, high-performance observability data pipeline, written in Rust and maintained by Datadog, that does exactly this for logs and metrics.

Vector's design centers on three ideas: a simple source → transform → sink model, a purpose-built transformation language (VRL) that fails safely at configuration time rather than in production, and explicit control over buffering and delivery guarantees. Because the pipeline sits in front of expensive storage, it is also where most teams control observability cost.

TL;DR

Pipelines connect sources (files, Kubernetes logs, syslog, OTLP, Kafka), transforms, and sinks (Loki, Elasticsearch, S3, Datadog, Kafka, and many more).
Deploy as agents on every node, aggregators that centralize processing, or both.
Use VRL (remap) to parse, enrich, and normalize events; its errors are caught when the configuration loads.
Redact secrets and personal data before events leave the node; sample, filter, and route to control cost.
Choose memory or disk buffers and a full-buffer policy (block or drop); enable end-to-end acknowledgements for stronger delivery.
Unit-test configuration with vector test, validate in CI, and monitor Vector's own metrics.

Quick Example

A Kubernetes agent configuration: collect pod logs, redact sensitive values, parse JSON, and send to Loki and an S3 archive with durable buffering:

A unit test in the same configuration, run with vector test vector.yaml:

Core Concepts

The Pipeline Model

Components connect through inputs, forming a directed graph. One event stream can fan out to several sinks — for example, recent logs to a searchable store and everything to cheap object storage.

Deployment Topologies

In Kubernetes, agents run as a DaemonSet and aggregators as a StatefulSet with persistent volumes for disk buffers; the official Helm chart supports both roles.

VRL: Vector Remap Language

VRL is an expression language for transforming events. Functions that can fail — parsing, type conversion — must be handled explicitly (with err values, ?? defaults, or ! to abort), and Vector checks this when it loads the configuration, so many mistakes surface before deployment instead of as dropped events. VRL includes parsers for JSON, logfmt, syslog, common log formats, key-value pairs, and grok patterns, plus functions for redaction, hashing, IP and GeoIP enrichment, and timestamps.

Redaction and Privacy

Remove secrets and personal data as early as possible, ideally in the agent: redact with built-in filters and custom patterns, del() for known sensitive fields, and hashing for identifiers you need to correlate but not read. Test redaction with unit tests, and remember that application-side logging hygiene is the first defense. See PII Handling.

Cost Control

See Metric Cardinality and Telemetry Costs.

Buffering, Back Pressure, and Delivery

Each sink has a buffer: in memory (fast, lost on restart) or on disk (survives restarts, limited by disk space). When a buffer fills because the backend is slow or down, when_full: block applies back pressure upstream — sources slow down, and file-based sources simply stop reading and resume later — while drop_newest sheds events to protect the pipeline. Sinks retry failed requests with backoff, and adaptive request concurrency adjusts parallelism to the backend's response times.

With end-to-end acknowledgements enabled, sources that support them (such as Kafka or HTTP-based sources) only acknowledge events after sinks confirm delivery, providing at-least-once delivery across the pipeline. Duplicates are therefore possible after failures; backends or downstream processing should tolerate them.

Operating Vector

Validate and test configurations in CI with vector validate and vector test.
Watch the pipeline: Vector's internal_metrics source exposes events in and out, errors, buffer sizes, and dropped events; vector top and vector tap show live throughput and sample events.
Alert on buffer growth, sink errors, and discarded events — a silent pipeline failure means silent blind spots.
Roll out configuration changes gradually, like application deployments.

Vector and the OpenTelemetry Collector overlap: the Collector is the standard for traces and OTLP-native pipelines, while Vector is strong for logs, transformation, and routing; many teams run both.

Best Practices

Redact at the Edge

Remove sensitive data in the agent, before it crosses the network or reaches shared storage.

Use Disk Buffers for Data You Cannot Lose

Persist buffers for audit and security logs, and size them for realistic backend outages.

Test Transforms Like Code

Write unit tests for parsing, redaction, and routing rules, and run them in CI.

Monitor the Monitor

Alert on Vector's own errors, drops, and buffer usage.

Common Mistakes

Memory Buffers for Critical Logs

High-Cardinality Labels

Redacting Only the Raw Message

FAQ

What is Vector used for?

Collecting, transforming, and routing logs and metrics from hosts, containers, and applications to one or more observability backends, with control over parsing, redaction, sampling, buffering, and delivery.

What is VRL?

The Vector Remap Language, an expression language for transforming events that requires explicit handling of fallible operations, so many errors are caught when the configuration is loaded.

Should Vector run as an agent or an aggregator?

Agents on every node collect data; aggregators centralize heavier processing and routing. Large deployments commonly use both: lightweight agents forwarding to an aggregator tier.

How does Vector avoid losing data?

Disk buffers persist events across restarts and outages, block policies apply back pressure instead of dropping, sinks retry failed requests, and end-to-end acknowledgements confirm delivery for supporting sources.

How is Vector different from the OpenTelemetry Collector?

Both are telemetry pipelines. The OpenTelemetry Collector is the vendor-neutral standard focused on OTLP and traces; Vector focuses on high-performance log and metric processing with VRL. They are often used together.