Streams and buffers

Why they exist

A stream is data that arrives over time: a file, an HTTP request body, a response, a socket, the output of another process. It is the difference between a server that survives a large upload and one that is killed by the operating system.

the same job, two memory profiles

// 2 GB file → 2 GB of RAM, and the thread is busy building the string
const text = await fsp.readFile("access.log", "utf8");

// a few kilobytes at a time, whatever the file size
const stream = fs.createReadStream("access.log", { encoding: "utf8" });
for await (const chunk of stream) { … }

A Buffer is bytes

text is an interpretation

const buf = Buffer.from("Qapı", "utf8");
buf.length          // 5 — five BYTES for four characters
"Qapı".length       // 4

buf.toString("utf8");
buf.toString("base64");
Buffer.concat([a, b]);

A chunk can split a character

A multi-byte character can land half in one chunk and half in the next. Give the stream an encoding, or use StringDecoder, instead of calling toString() on each chunk yourself.

The four kinds

Kind	Example
Readable	`fs.createReadStream`, an HTTP request, `process.stdin`
Writable	`fs.createWriteStream`, an HTTP response, `process.stdout`
Duplex	A TCP socket — both at once
Transform	gzip, a cipher, your own line splitter

Reading one

async iteration is the readable version

let lines = 0;
for await (const chunk of fs.createReadStream("access.log", { encoding: "utf8" })) {
  lines += chunk.split("\n").length - 1;      // careful — see below
}

// the older event form, still everywhere
stream.on("data", (chunk) => { … });
stream.on("end", () => { … });
stream.on("error", (err) => { … });     // never omit this one

The boundary problem

A chunk is an arbitrary slice of bytes, not a record, a line or a message. Every protocol built on a stream has to reassemble its own units — and this pattern, keeping the remainder, is how.

the mistake every line parser makes once

// chunk 1: "id,name\nA,Pain"
// chunk 2: "ted\nB,Veneer\n"
// splitting each chunk on its own invents a door called "Pain"

let rest = "";
for await (const chunk of stream) {
  const parts = (rest + chunk).split("\n");
  rest = parts.pop();                  // the last piece may be incomplete
  for (const line of parts) handle(line);
}
if (rest) handle(rest);                // whatever was left at the end

Piping and backpressure

write() returns false when the destination is full. Piping respects that and pauses the source — that is backpressure.
Ignoring it (reading as fast as you can into a slow socket) is how a process’s memory climbs until it is killed.
Use pipeline, not .pipe(): it propagates errors and cleans up every stream in the chain.

let the slow end set the pace

import { pipeline } from "node:stream/promises";
import { createGzip } from "node:zlib";

await pipeline(
  fs.createReadStream("access.log"),
  createGzip(),
  fs.createWriteStream("access.log.gz")
);

Web streams as well

the same idea, the browser API

const res = await fetch(url);
for await (const chunk of res.body) { … }        // a web ReadableStream

// converting between the two worlds
import { Readable } from "node:stream";
Readable.fromWeb(res.body);

When not to bother

A 40 KB JSON file does not need a stream. The complexity is worth it when the data is large, unbounded, or arriving from somewhere you do not control — an upload, a log, a database export, another process’s output.