Cost and latency

The arithmetic

Two facts do most of the work. Output costs several times more than input — commonly four or five times — so a verbose reply is the expensive one. And cached input is a fraction of the price, typically a tenth, which makes prompt structure a cost decision.

Work in cost per request and then per thousand requests. A tenth of a penny is nothing; a tenth of a penny times four million is a conversation with finance.

cost = (input / 1e6) × inputRate
     + (cached / 1e6) × cachedRate
     + (output / 1e6) × outputRate

Where the seconds go

That split is the whole latency story: input size sets the delay before anything happens, output length sets everything after it. A 500-token answer at 50 tokens per second is ten seconds of generation, and no amount of infrastructure will change that.

So the first question about a slow endpoint is not "which model" but "how long is the reply, and does it need to be".

Part	Depends on	Typical
Network and queueing	Region, load	50–200 ms
Time to first token	The size of the input	200 ms – 2 s
Generation	The number of output tokens, one at a time	20–100 tokens/s

Prompt caching

Providers cache a prefix of the prompt. The rule that follows is the one to remember: put everything stable at the front, and everything that varies at the back. Move one character in the system prompt and the whole cache is invalidated.

It is also a latency win, not only a cost one — a cached prefix does not have to be processed again, so time to first token falls sharply. On a long system prompt with examples, it is frequently the single biggest improvement available.

text

stable    │ system prompt · tool definitions · few-shot examples · documents
          │ ← everything up to here can be cached, and is charged at a tenth
varying   │ the conversation, the question

The four changes that actually move the numbers

Routing is the one people leave on the table. Most traffic in a real application is easy: classify it cheaply, send the easy cases to a small model, escalate the rest. A 90/10 split against a model a tenth of the price cuts the bill by more than 80 % — and the evaluation harness is what tells you whether the small model is good enough on that 90 %.

Change	Cost	Latency
Ask for shorter output	Large — output is the expensive side	Large — it is generated serially
Cache the prefix	Large	Large
Use a smaller model for the easy cases	Very large	Large
Trim the context	Moderate	Moderate (time to first token)
Stream	None	Perceived — the next lesson
Batch offline work	Large (often half price)	Worse, deliberately

Measure it per request

Without this you will be guessing. With it you can answer the four questions that come up: what does a request cost on average, which endpoint is the expensive one, what fraction of prompts are hitting the cache, and where the p95 latency is coming from.

And watch the p95, not the mean: one request in twenty taking fifteen seconds is what users remember, and the mean will not show it — exactly the argument from the performance course, applied here.

log.info({
  requestId, model,
  inputTokens, cachedTokens, outputTokens,
  cost: price(usage, RATES[model]),
  msToFirstToken, msTotal,
  cacheHit: cachedTokens > 0,
});

The arithmetic

Where the seconds go

Prompt caching

The four changes that actually move the numbers

Measure it per request

Practice