Node.js Streams & Backpressure: Bounded Memory

Q: Should I use pipe() or pipeline()?

pipeline(). Both handle backpressure, but only pipeline() propagates errors and destroys all streams when something fails, preventing resource leaks.

Q: Can I use async/await with streams?

Yes. Readable streams are async iterables, pipeline() from node:stream/promises returns a promise, and async generators can act as transform stages with backpressure preserved.

Q: How do I handle a client disconnecting during a download?

Use pipeline() so the destroyed response destroys the rest of the chain, and cancel related work with an AbortSignal tied to the request's 'close' event.

Node.js Streams and Backpressure

Reading a 2 GB export into memory before sending it crashes a Node.js process; streaming it sends the same data using a few megabytes. Streams process data in chunks as it flows from a source — a file, a database cursor, an HTTP request — through transformations to a destination. The crucial part is backpressure: when the destination is slower than the source, the stream machinery tells the source to pause, so memory stays bounded no matter how large the data or how slow the client.

Backpressure only works if code respects it. Ignoring the return value of write(), using pipe() without error handling, or wrapping a stream in code that buffers everything quietly brings back the memory problems streams were meant to solve.

TL;DR

Four stream types: Readable, Writable, Duplex, and Transform.
Each stream buffers up to its highWaterMark; beyond that, it signals backpressure.
writable.write() returns false when the buffer is full — wait for 'drain' before writing more.
Use pipeline() from node:stream/promises: it handles backpressure, propagates errors, and destroys every stream on failure.
Async iteration (for await) and async generators make readable streams easy to consume and produce.
Handle aborted connections so database cursors and file handles close, and test slow consumers explicitly.

Quick Example

Stream a gzip-compressed CSV export from a database cursor to an HTTP client with bounded memory:

If the client downloads slowly, res fills its buffer, gzip pauses, the transform pauses, and the database stream stops fetching rows until the client catches up.

Core Concepts

Stream Types

Streams carry Buffer or string chunks by default; object mode carries arbitrary JavaScript values such as database rows.

Buffers and highWaterMark

Every stream has an internal buffer limited by highWaterMark — 64 KiB for byte streams in recent Node.js versions, or 16 objects in object mode. The limit is a threshold, not a hard cap: it decides when the stream asks producers to slow down.

Backpressure Signals

A Readable stops calling its source (_read) when its buffer is full and resumes as consumers read.
A Writable's write() returns false when its buffer exceeds highWaterMark; the writer should stop until 'drain' fires.
pipeline() and pipe() connect these signals automatically between streams.

Writing manually requires respecting the signal:

pipeline() Versus pipe()

readable.pipe(writable) handles backpressure but not errors: if one stream fails, the others are not destroyed, which leaks file descriptors and database connections. pipeline() propagates the first error, destroys all streams, and returns a promise (from node:stream/promises) that resolves when everything finishes. Use pipeline() everywhere.

Async Iteration and Generators

Readable streams are async iterables, and async generators can be used as pipeline stages:

Backpressure still applies: the generator only resumes when the next stage can accept data.

Web Streams Interop

Node.js also implements the WHATWG Web Streams API used by fetch. Convert with Readable.fromWeb(), Readable.toWeb(), and Writable.toWeb() when combining fetch responses with Node streams.

Aborted Connections and Cleanup

Clients disconnect mid-download, uploads are canceled, and proxies time out. With pipeline(), a destroyed response or request destroys the whole chain, so sources stop producing and release resources. For work outside streams — database queries started for a request — pass an AbortSignal (pipeline(..., { signal }) accepts one) and listen for the request's 'close' event to cancel promptly.

Testing Slow Consumers

Create a deliberately slow writable to prove that memory stays bounded and the source pauses:

Assert that the producer's buffered length stays near the high water mark, that abort mid-stream closes the source, and that errors from any stage reject the pipeline.

Best Practices

Stream Large Payloads End to End

Uploads, exports, file proxies, and log processing should never buffer whole payloads. See File Upload.

Always Use pipeline()

It is the simplest way to get backpressure, error propagation, and cleanup right together.

Keep Transforms Small and Synchronous Where Possible

Heavy CPU work inside a transform blocks the event loop for every connection; batch it or move it to worker threads.

Limit Concurrency in Stages That Call Services

When a stage makes network calls per chunk, bound the number in flight so backpressure reflects downstream capacity.

Common Mistakes

Ignoring write()'s Return Value

pipe() Without Error Handling

Buffering Inside a "Streaming" Endpoint

FAQ

What is backpressure in Node.js streams?

The mechanism by which a slow destination tells upstream streams to pause. When a writable's buffer exceeds its high water mark, write() returns false and producers wait for 'drain', keeping memory use bounded.

Should I use pipe() or pipeline()?

pipeline(). Both handle backpressure, but only pipeline() propagates errors and destroys all streams when something fails, preventing resource leaks.

What is highWaterMark?

The buffer size threshold at which a stream signals backpressure: 64 KiB by default for byte streams in recent Node.js versions, and 16 objects in object mode.

Can I use async/await with streams?

Yes. Readable streams are async iterables, pipeline() from node:stream/promises returns a promise, and async generators can act as transform stages with backpressure preserved.

How do I handle a client disconnecting during a download?

Use pipeline() so the destroyed response destroys the rest of the chain, and cancel related work with an AbortSignal tied to the request's 'close' event.