One operation
A large language model takes a sequence of tokens and produces a probability distribution over what the next token might be. That is the entire operation. Everything else — chat, tools, reasoning, agents — is a loop around it and some text formatting.
Generating a reply means doing this repeatedly: predict a distribution, pick one token from it, append it, predict again. The model does not plan the sentence and then write it; it commits one token at a time, and every token it has already emitted is part of the input for the next one.
What follows from that
These are not limitations to be engineered away. They are what the thing is, and the job of everything in this course is to build something dependable on top of them rather than to pretend they are absent.
| Consequence | Because |
|---|---|
| It has no memory between calls | Every request is a fresh sequence. A conversation exists only because you resend it |
| It cannot revise | A token already produced is input now. A wrong start is committed to, and often defended |
| It is not looking anything up | It produces likely continuations, so a plausible-sounding wrong fact costs it nothing |
| It is not deterministic by default | Picking from a distribution involves randomness — see the sampling lesson |
| Output is slower than input | The prompt is processed in parallel; the reply is produced one token at a time |
Hallucination is not a bug in the usual sense
A model that has never seen your internal API will still produce a confident, well-formed call to it, because "a plausible continuation of documentation" is exactly what it was trained to produce. There is no separate step at which it checks whether something is true, and no signal in the output that distinguishes a remembered fact from an invented one.
The engineering response is not a better prompt. It is grounding: put the facts in the context (retrieval), make the model cite where they came from, and verify anything consequential against a real system. That is the whole argument for the RAG half of this course.
The parts of a request
The roles are a convention the model was trained on, not a security boundary: the whole thing is concatenated into one sequence. A system message is influential, not binding, which is the root of every prompt-injection problem in the safety lesson.
And the conversation is resent in full on every turn. A long chat is not stored anywhere; it is re-uploaded, re-processed and re-paid for each time.
const request = {
model: "…",
messages: [
{ role: "system", content: "Who the assistant is, and the rules" },
{ role: "user", content: "What the person asked" },
{ role: "assistant", content: "What it said last time" },
{ role: "user", content: "…and the next thing they asked" },
],
tools: [ /* what it may call — the tool-calling lesson */ ],
temperature: 0, // how randomly to pick — the sampling lesson
max_tokens: 1000, // a ceiling on the reply
};What this course is, and is not
It is not about training models or about the mathematics inside them. It is about the layer you own: what you put in the context, how you retrieve it, how you validate what comes back, how you measure whether a change helped, and what it costs.
Every lab here runs real code with no API key — tokenisation, sampling arithmetic, embeddings, BM25, schema validation, event-stream parsing, cost and evaluation are all exact and checkable. Two things are labelled stand-ins and say so wherever they appear: a small byte-pair encoder (the behaviour is real, the vocabulary is not a provider’s) and a MockModel that replies by rule, so a loop can be driven and its control flow tested.