Streaming

Parse a server-sent event stream properly, handle partial output, and cancel without leaking a request.

21of 24
≈ 13Min read
5Parts

What streaming actually changes

Nothing about the total time. A 500-token answer takes the same ten seconds to generate whether you stream it or not — what changes is that the user sees the first word after 300 milliseconds instead of after ten seconds.

That is a perceived-performance win, and by the thresholds in the performance course it is a large one: ten seconds of nothing is past the point where attention goes, and ten seconds of text arriving is not.

The wire format

Server-sent events: field: value lines, a blank line between events. The rules that catch people out are all in the specification and all small — a leading space after the colon is stripped, several data: lines in one event are joined with newlines, and a line starting with : is a comment, used as a keep-alive.

The other one is not in the format at all: a chunk from the network is not an event. TCP gives you bytes, and one read can contain half an event or three and a half. You must buffer until you see a blank line, which is why the parser in the lab takes a whole string and why a real one keeps a buffer between reads.

text
event: content_block_delta
data: {"type":"text_delta","text":"Made to"}

event: content_block_delta
data: {"type":"text_delta","text":" measure doors"}

event: message_stop
data: {}

Consuming it

buffer = blocks.pop() is the whole trick: everything before the last blank line is complete, and whatever follows it is held over for the next read. Get that line wrong and you will lose an event every few hundred requests, which is the kind of bug that only shows up under load.

js
const response = await fetch(url, { method: "POST", body, signal: controller.signal });
const reader = response.body.pipeThrough(new TextDecoderStream()).getReader();

let buffer = "";
let text = "";

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += value;

  const blocks = buffer.split("\n\n");
  buffer = blocks.pop();                       // the last one may be incomplete

  for (const block of blocks) {
    const event = parseEvent(block);
    if (event.data === "[DONE]") return text;
    text += JSON.parse(event.data).text ?? "";
    onToken(text);
  }
}

Partial structured output

Streaming JSON is half-formed for most of its life: {"category": "dam does not parse. Three honest options, in order of preference:

  • Do not stream structured output. If the consumer is code, it has to wait for the whole object anyway. Stream prose; return JSON whole.
  • Stream a field at a time. Ask for the user-visible text first in the object, and show that field as it arrives while the rest completes.
  • Use a streaming parser that emits completed values as they close. Real, and more machinery than most applications need.

And whatever you do, validate the finished object, not the fragments. A partial object that happens to parse is a trap: {"items": [1, 2 is invalid, but {"items": [1, 2] with the closing brace added by a repair library is a plausible, wrong answer.

Cancelling, and the rest of the plumbing

Aborting matters for money as much as for tidiness: generation that nobody is reading is still being billed. A chat interface where the user can ask a second question should cancel the first request, and a server that streams to a client which has disconnected should notice and stop.

Two more, both easy to forget. Buffer the render: updating the DOM on every token is a layout thrash at fifty frames a second, so accumulate and paint on an animation frame. And handle an error mid-stream — the connection can fail after two hundred tokens, and you have to decide whether to keep the partial text or discard it.

js
const controller = new AbortController();
input.oninput = () => controller.abort();            // the user typed again

/* and always, however it ends */
finally { reader.releaseLock(); }

Practice

Next →Reliability