Structured output

Three ways to ask, in order of reliability

Use the strongest one the provider offers. Constrained decoding removes the entire category of parse failures by construction, which is worth more than any amount of prompt engineering — but it does not make the values correct, only the shape. A schema cannot tell you the extracted price was wrong.

What still goes wrong

Truncation is the one worth designing around: check finish_reason before you check the JSON. A reply that stopped because it ran out of room is a different failure from a reply that is malformed, and it needs a different response — more tokens, or a smaller request, not a repair.

• Prose around the JSON. "Here is the JSON you asked for:" followed by a fenced block.
• Markdown fences. ``` `json ``` — extremely common, and trivial to strip.
• Truncation. Hitting max_tokens mid-object. The JSON is not wrong, it is unfinished, and no repair can invent what was cut.
• Plausible but wrong types. "42" instead of 42, "true" instead of true, "null" instead of null.
• Invented fields, or a missing optional one you assumed was always present.
• Trailing commas and single quotes, especially in smaller models.

Parse defensively

Find the outermost braces and parse what is between them. That handles the prose and the fences, which together are most of the failures. What it deliberately does not do is guess at broken structure — a JSON repair library will happily turn a truncated object into a valid one that is missing half its data, and silent data loss is worse than a thrown error.

function parseJson(text) {
  const cleaned = String(text)
    .replace(/^[\s\S]*?```(?:json)?/i, "")   // drop anything before a fence
    .replace(/```[\s\S]*$/, "")               // and anything after it
    .trim();

  const start = cleaned.search(/[{[]/);
  const end = Math.max(cleaned.lastIndexOf("}"), cleaned.lastIndexOf("]"));
  const body = start === -1 || end === -1 ? cleaned : cleaned.slice(start, end + 1);

  try {
    return { ok: true, value: JSON.parse(body) };
  } catch (error) {
    return { ok: false, error: String(error.message) };
  }
}

Validate at the boundary

Model output is untrusted input, exactly like a request body. The same rule from the security course applies without modification: validate at the boundary, and never let an unvalidated value reach a database, a template or a shell.

And feed the specific errors back on a retry. "Invalid JSON" gives the model nothing to act on; "confidence must be at most 1" gets it fixed on the next attempt.

const SCHEMA = {
  type: "object",
  required: ["category", "confidence"],
  properties: {
    category: { enum: ["delivery", "damage", "billing", "other"] },
    confidence: { type: "number", minimum: 0, maximum: 1 },
    order_id: { type: "string" },
  },
};

const parsed = parseJson(reply);
if (!parsed.ok) return retry("that was not valid JSON");

const errors = validate(SCHEMA, parsed.value);
if (errors.length) return retry(errors.map((e) => e.path + " " + e.message).join("; "));

The retry ladder

Two retries, then stop. A model that has failed the schema twice with the errors in front of it is not going to succeed on the fifth attempt, and each try costs a full request. Log the failures — a rising schema-failure rate is one of the clearest signals that a model or a prompt has drifted.

1. Parse. If it failed, retry once with the parse error quoted.
2. Validate. If it failed, retry once with the field errors quoted.
3. Still failing? Fall back — a simpler schema, a different model, or an honest error.
4. Never loop without a ceiling, and never retry at temperature > 0 expecting a different kind of answer.