Byte-pair encoding, in one paragraph
A tokeniser is built by taking a corpus, finding the most frequent adjacent pair of symbols, merging it into one new symbol, and repeating tens of thousands of times. Common words end up as a single token; rare ones end up as several pieces; and an unknown string falls back to characters or bytes, so nothing is ever unrepresentable.
The consequence is the rule of thumb everyone quotes and few people check: about 0.75 words per token in English, and much worse in most other languages, in code, and in anything with unusual capitalisation.
What surprises people
The space rule is the one that causes real bugs: "the" and " the" are different tokens, so a prompt assembled with a stray leading space is genuinely a different input. Splitting a sentence and rejoining it carelessly changes what the model reads.
| Text | Why it costs what it costs |
|---|---|
the | One token — among the most common strings in the corpus |
the | A different token from the. The leading space belongs to the word |
The | Different again. Case is not normalised |
unbelievability | Several pieces — rare words are assembled from common ones |
x7f9Qz | Nearly one token per character. Identifiers and hashes are expensive |
مرحبا | Several tokens per character in scripts under-represented in the corpus |
{"a": 1} | JSON punctuation is tokens too — the structure is not free |
Where it changes the answer, not just the cost
The engineering answer to all of these is the same: do not ask the model to do it. Count the letters in code, do the arithmetic in a tool, and let the model decide what to compute rather than compute it.
- Counting characters or letters. "How many r’s in strawberry?" is hard because the model never sees the letters — it sees two or three tokens. This is not a reasoning failure; the information is not in the input.
- Arithmetic on long numbers. Digits group into tokens in ways that do not line up with place value, so column arithmetic is fighting the representation.
- Reversing or manipulating strings — same reason.
- Rhyme and wordplay, where sub-word structure matters and the model sees only whole pieces.
- Stop sequences and structured output, where a boundary you care about may sit in the middle of a token.
Counting, properly
Every provider has a different vocabulary, so a count from one model is an estimate for another. If you are enforcing a budget, count with the tokeniser that belongs to the model you are calling — and count the whole request, including the system prompt, the tool definitions and the message scaffolding, which are all tokens you pay for.
/* the wrong way, and the error is not small */
const rough = text.length / 4; // ±30 %, and much worse on code
/* the right way: the tokeniser for the model you are actually calling */
import { encoding_for_model } from "tiktoken";
const encoder = encoding_for_model("gpt-4o");
const tokens = encoder.encode(text).length;
/* or the provider's endpoint, which is exact and costs a round trip */
await client.messages.countTokens({ model, messages });The encoder in this course
encode(text) in these labs is a real byte-pair encoder over a small merge table built into the page. It behaves the way a tokeniser behaves — round-trips exactly, merges common pairs, keeps the leading space with a word, charges more for rare strings — but its vocabulary is a few dozen merges rather than a hundred thousand, so the counts are not any provider’s counts.
That is enough for everything this course does with it: budgets, truncation, chunking and cost arithmetic are all about ratios and limits, and they work identically whatever the vocabulary is.