Injection and data handling

Treat model output and retrieved text as untrusted input

The security course had one rule that carries over without modification: data from outside your trust boundary is validated at the boundary, and never reaches a privileged operation unchecked. In an LLM application, two things sit outside that boundary — the text you retrieved, and the text the model produced.

That is the whole mental model. Everything below is the ordinary engineering that follows from it, and none of it depends on the model behaving well.

Least privilege for tools

Scope the tool set to the task in front of you rather than to everything the application can do. A step that reads documents does not need the tool that sends mail, and a tool that is not in the list cannot be called.

For anything irreversible — sending, refunding, deleting, publishing — the model proposes and a person confirms. That control is independent of how the request arose, which is exactly why it holds.

/* the tool set available to a summariser that reads public web pages */
const TOOLS = { fetch_page: … };           // and nothing else

/* the checks that run on every call, from the tool-calling lesson */
validate(tool.schema, args);               // the shape
authorise(currentUser, tool, args);        // this user, this record
applyLimits(args);                         // bounded work

Keep untrusted text separable

These reduce confusion between instructions and content. They are worth doing, and they are not a security boundary on their own — the controls that actually hold are the ones on the other side of the model, in the next two sections.

• One clear delimiter around retrieved or user-supplied content, and strip that delimiter from the content itself before inserting it.
• Label the provenance of each chunk — which document, which source — so the model and your logs both know what came from where.
• State the job in the system prompt, so the task does not have to be inferred from the content being read.
• Do not put secrets in the context. An API key in a system prompt is one paraphrase away from an output filter having to catch it.

Validate the output before acting on it

Allow-list, never deny-list. A list of the four actions this endpoint may perform is complete and checkable; a list of things to forbid is a guess about what you thought of.

And apply the same authorisation you would apply to a request from an anonymous client. The model asking for something is not a reason to do it — that decision belongs to your code, with the acting user’s permissions, every time.

const parsed = parseJson(reply);
if (!parsed.ok) return reject("unparseable");

if (validate(SCHEMA, parsed.value).length) return reject("schema");

/* an allow-list, not a deny-list */
if (!ALLOWED_ACTIONS.has(parsed.value.action)) return reject("unknown action");
if (!isInternal(parsed.value.url)) return reject("external url");

/* and the same authorisation you would apply to any request */
if (!currentUser.canDo(parsed.value.action, parsed.value.target)) return reject("not permitted");

Personal data, and what leaves the building

Redaction is mechanical and cheap — a few patterns, applied to the text before it is assembled into the prompt — and it removes a whole class of incident that is otherwise entirely dependent on the model never repeating what it read.