Compression Formats & CPU Tradeoffs: gzip to zstd

Compression Formats and CPU Tradeoffs

Compression trades CPU time for fewer bytes. Fewer bytes mean cheaper storage, lower egress bills, faster transfers over slow links, and more data per cache or memory page. More CPU means higher latency on the compressing and decompressing side, more servers, and more energy. The right choice depends on where the bytes go, how often data is written versus read, and what the hardware and network actually cost — which is why the answer is "measure on your data," not "always use the maximum level."

Modern codecs make the trade-off unusually flexible. Zstandard spans from LZ4-like speed to xz-like ratios with one format and fast decompression throughout; Brotli compresses web text very well, especially when assets are compressed once at build time; LZ4 and Snappy minimize CPU for hot paths; gzip remains the universal baseline. Choosing among them, and choosing levels, is a performance engineering decision like any other.

TL;DR

Measure ratio, compression speed, and decompression speed on representative data; costs are asymmetric.
Compress once, read many (static assets, archives, data lakes): use high levels — Brotli 11, zstd 19, xz.
Compress per request or per message (APIs, logs, RPC): use fast levels — zstd 1–3, gzip 1–6, LZ4.
Negotiate HTTP compression with Accept-Encoding, serve precompressed static assets, and skip tiny or already-compressed payloads.
Use dictionaries for many small similar messages and columnar encodings before generic codecs in analytics formats.
Cap decompressed sizes against bombs, and never compress secrets alongside attacker-controlled data over TLS (BREACH).

Quick Example

Benchmarking codecs on 100,000 JSON log lines (about 10 MB) with Python:

Output from one desktop machine (single-threaded; numbers vary with hardware and data):

Read it by workload: shipping logs from busy services favors zstd-1 (fast, good ratio); archiving them once for years favors zstd-19 (best ratio and fast decompression); xz matches the ratio but decompresses five times slower. Also notice that a higher level does not always win — zstd-3 was slightly worse than zstd-1 on this data.

Core Concepts

Codec Families

See Compression.

Asymmetric Workloads

Compression cost is paid by the writer; decompression cost by every reader. Static assets compressed at build time and served millions of times justify the slowest, best levels. Dynamic API responses compressed on every request need fast levels to keep latency and CPU low. Data lakes written once and scanned repeatedly benefit from codecs with fast decompression.

HTTP Compression

Clients advertise support with Accept-Encoding (for example gzip, br, zstd); servers choose one and set Content-Encoding and Vary: Accept-Encoding so caches store variants correctly. Serve precompressed .br and .gz files for static assets from the web server or CDN; use moderate levels for dynamic responses; skip compression for small responses (a few hundred bytes) and for already-compressed types such as JPEG, PNG, MP4, and zip. See HTTP and HTTP Caching, Cookies, and Proxies.

Small Messages and Dictionaries

Generic compressors need history to find repetition, so tiny messages compress poorly. Zstd dictionaries trained on sample messages (zstd --train) can multiply ratios for small JSON events, RPC payloads, or cache entries — at the cost of distributing and versioning the dictionary with the data.

Columnar Data and Pipelines

Analytics formats such as Parquet apply encodings first — dictionary, run-length, delta — then a codec (Snappy, zstd, gzip) per column chunk. Choosing zstd for cold tables and Snappy or LZ4 for hot, frequently scanned data is common. In Kafka, producers compress whole record batches; larger batches compress better, trading latency.

Memory, Streaming, and Parallelism

Codecs use windows and buffers: zstd's long-range mode and xz's large dictionaries need more memory on both sides. Stream large data rather than loading it fully, and use multi-threaded compressors (zstd -T0, pigz) for big batch jobs.

Security and Safety

Best Practices

Benchmark on Real Data and Hardware

Use samples of production payloads and the target CPUs; synthetic text gives misleading ratios.

Pick Levels by Workload, Not Habit

High levels for compress-once data; fast levels for per-request and per-message compression.

Precompress Static Assets

Generate Brotli and gzip variants in the build and let the server or CDN select them. See Benchmarking.

Price Both Sides

Compare CPU cost against storage and egress savings, including the cost of decompressing on every read.

Common Mistakes

Maximum Level on Dynamic Responses

Compressing Already-Compressed Data

Unbounded Decompression

FAQ

Which compression algorithm is best?

There is no single best. Zstandard is a strong default for general data, Brotli for precompressed web assets, LZ4 when CPU must be minimal, gzip for maximum compatibility, and xz when ratio matters more than speed.

What compression level should I use?

Low levels for data compressed frequently or on the request path; high levels for data compressed once and read many times. Verify with benchmarks, since higher levels can give diminishing or even negative returns.

Should I compress API responses?

Usually yes for responses larger than a few hundred bytes of text, using negotiated gzip, Brotli, or zstd at moderate levels — but not for tiny or already-compressed payloads, and not when secrets and attacker input share a response.

What is a compression dictionary?

Pre-trained shared content that both compressor and decompressor use as history, greatly improving ratios for small, similar messages.

Is decompression always fast?

No. Zstd and LZ4 decompress very quickly at every level, but xz and high-ratio formats can be slow to decompress, which matters for data read often.