The context window

One number, shared by everything

The window covers the input and the output. A 200 000-token window with a 190 000-token prompt leaves you 10 000 tokens of answer, and many APIs will simply refuse rather than truncate. max_tokens is reserved out of the same total.

So context is not "how much can I send". It is an allocation problem between five claimants, and the interesting engineering is deciding what gives way when they do not all fit.

text

┌──────────────── context window ────────────────┐
│ system │ tools │ retrieved │ history │ question │ ← reply │
└────────────────────────────────────────────────┘
        everything above is input, and the reply comes out of the same budget

The five claims, and what each is worth

History is the one that grows on its own, which is why every chat application needs a strategy for it before it needs anything else.

Claim	Typical size	Drop it?
System prompt	100–2 000	Never — it is the behaviour
Tool definitions	50–200 per tool	Yes: send only the tools this step could need
Retrieved context	Usually the largest	Yes: fewer, better chunks — the retrieval lessons
Conversation history	Grows without limit	Yes: the main thing to manage
The current question	Small	Never

The four strategies for history

Pin and slide is the pragmatic default: the first exchange usually contains the task, and the last few contain the state. What sits between them is mostly working out. Summarise only when a conversation is genuinely long, and remember that a summary of a summary loses more each time.

Strategy	How	Loses
Sliding window	Keep the last N messages	Anything said early — including the user’s actual goal
Summarise	Replace old turns with a summary	Detail, and costs a model call; the summary can drift
Retrieve	Store the history and fetch the relevant parts	Nothing in principle; adds a retrieval step and its failure modes
Pin and slide	Keep the first turns and the last, drop the middle	The middle — usually the cheapest thing to lose

Longer windows did not end this

Which is the real lesson of this course’s retrieval half: the goal is not to fit everything in. It is to put the right few thousand tokens in, and leave the rest out.

Cost is linear in input. A million-token prompt on every turn is a bill, not a design.
Latency is linear in input. Time to first token grows with the prompt, on every single call.
Attention is not uniform. Models reliably use the beginning and the end of a long context better than the middle — the "lost in the middle" result, and it has held across model generations.
Irrelevant context actively hurts. Filling the window with plausible-but-wrong material measurably lowers accuracy. More context is not more information.

Budget it explicitly

Write the allocation down, count against it before the call, and fail loudly when something exceeds its share. An overflow discovered by a 400 from the provider, in production, is the same bug found in the worst possible place.

const BUDGET = {
  total: 128000,
  reserveForReply: 4000,
  system: 800,
  tools: 600,
  retrieved: 6000,        // a ceiling, not a target
};

const forHistory = BUDGET.total - BUDGET.reserveForReply
  - BUDGET.system - BUDGET.tools - BUDGET.retrieved;

One number, shared by everything

The five claims, and what each is worth

The four strategies for history

Longer windows did not end this

Budget it explicitly

Practice