Reliability

What will fail, and how often

The line is simple: retry what is transient, never what is your fault. A retry loop on a 400 is a way of turning one bug into a rate-limit incident.

Exponential backoff, with jitter

Doubling gives an overloaded service room to recover. The jitter is not optional: without it, every client that failed at the same moment retries at the same moment, and the thundering herd keeps the service down. Multiplying by a random factor between 0.5 and 1 is enough.

Three or four attempts, then give up. And cap the total: a request that has already taken twenty seconds should fail rather than start a fourth attempt nobody is waiting for.

const delay = Math.min(cap, base * 2 ** attempt) * (0.5 + Math.random() * 0.5);

/* base 500 ms, cap 30 s
   attempt 0 →  250–500 ms
   attempt 1 →  500 ms–1 s
   attempt 2 →  1–2 s
   attempt 3 →  2–4 s */

Timeouts, and the two you need

Without a per-attempt timeout a hung connection holds a slot for ever. Without an overall one, three retries of a twenty-second timeout is a minute, and the user left forty seconds ago.

Set them from the work: a classification should answer in two seconds, a long generation might need sixty. A single global timeout is always wrong for one of them.

const perAttempt = AbortSignal.timeout(20000);          // one call
const overall = AbortSignal.timeout(45000);             // the whole operation, retries included

fetch(url, { signal: AbortSignal.any([perAttempt, overall]) });

Idempotency

A retry is a second request, and the first one may have succeeded — the response was lost, not the work. For anything that has an effect, send an idempotency key so the server can recognise the repeat, and make your own tools idempotent where you can.

This is the agent lesson’s point arriving from the other direction: a retried get_order is nothing, a retried send_email is an incident. Decide which of your tools may be retried automatically, and mark the rest as manual.

Falling back, and failing well

A degraded answer beats no answer, and no answer beats a wrong one. The bottom of that ladder is an error message, not a fabricated reply — a system that invents an answer when retrieval is down has converted an outage into misinformation, which is much worse and much harder to notice.

Two more pieces worth having: a circuit breaker, so that after a run of failures you stop calling for thirty seconds instead of hammering a service that is down; and a queue for anything that does not need an answer now. The most reliable request is the one that can wait.

primary model
  ↓ 529, twice
smaller / different model            ← usually good enough
  ↓ still failing
cached or default answer             ← if one exists
  ↓
an honest error                      ← and never an invented answer