Timeouts, retries, backoff, and jitter

A request is a finite budget of time and load. Every attempt consumes part of both. Retry only an operation that is safe to repeat and plausibly failed for a transient reason, keep every attempt and every delay inside one bounded completion budget, and make retries less synchronized than the failure that caused them — otherwise a brief dependency slowdown becomes a self-inflicted outage.

Given an uncertain dependency outcome, the layer holding the retry has to answer one question: retry, reconcile, return pending, fail, or hand the work to an asynchronous path? Status code alone never answers it — that decision needs the prior outcome, repeat safety, plausible transience, remaining budget, and who owns the retry, considered separately.

A useful way to hold the model in your head: total attempts is the initial call plus every retry, not "the number of retries." Saying "three retries" without saying whether that means three calls or four is how load estimates go wrong.

• Attempt — one invocation of a dependency. The first call is an attempt, not a "zeroth" retry.
• Retry eligibility — never decided by status code alone. It is six separate questions, walked in order below.
• Backoff — intentional delay before a retry, usually increasing after each failure and always capped, so it cannot grow to consume the remaining budget on its own.
• Jitter — randomized variation added to that delay so many callers retrying after the same failure don't arrive back at the dependency in a synchronized wave.

The retry decision, in order

Answer these in sequence. Any "no" or "unsafe" ends the sequence at a non-retry outcome — it does not fall through to the next question.

1. Is there enough completion budget remaining to wait, attempt, and handle the response? If not, stop now — retrying anyway spends the remaining budget on a call that will also fail from lack of time.
2. What is known about the prior outcome? Success returns success. A permanent failure (validation, auth, an explicit do-not-retry response) is surfaced and never retried. Everything else is either plausibly transient or outcome-unknown, and continues to the next question.
3. Is another execution repeat-safe under the dependency's documented contract — naturally idempotent, or covered by a receiver-side idempotency mechanism that this caller has actually confirmed? If not, stop: reconcile, return pending, or fail rather than guess.
4. Is the failure plausibly transient — a connection reset, a timeout, a documented retryable status — rather than a stable condition another identical attempt cannot change?
5. Does this layer own the retry budget for this operation, or might a client, SDK, gateway, mesh, queue consumer, or workflow engine downstream already be retrying it? See Nested retry ownership below.
6. If every prior check passed: wait a capped, jittered backoff while budget remains, then retry. If any check failed: choose the safer non-retry outcome — fail, return pending, reconcile/status-check, or move to an asynchronous path.

Why timeouts trigger overload: longer dependency latency raises the chance an attempt times out; a timeout triggers a retry; retries add to the dependency's concurrency and queueing; more queueing raises latency further. This positive feedback loop is a retry storm, and it is the main reason retry policy is a capacity decision, not just a client-side convenience.

A small timing-budget example, as a conservative planning model: a caller has an 800 ms deadline and reserves 150 ms for its own validation and response handling, leaving 650 ms for the dependency. A 300 ms per-attempt timeout and a single retry with up to 100 ms of backoff fit inside that 650 ms (300 + 100 + 300 = 700 ms — already tight). A second retry would not fit; the policy has to stop at one. Treat this as a floor, not an exact guarantee: a real budget may also need to absorb connection-pool wait or connection establishment, DNS or service discovery, network transit, dependency queueing, response decoding, and cancellation cleanup — and a per-attempt timeout expiring does not necessarily stop downstream processing that the dependency already accepted.

Bounded retries are the right tool when all of the following hold:

• The failure is plausibly transient — a connection reset, a timeout, or a dependency-reported retryable status (a documented 5xx class, or 429 with Retry-After) — not a validation, authentication, or authorization error.
• The operation is safe to repeat: it is naturally idempotent (a read, a PUT of full desired state), or the receiver documents and correctly implements a durable idempotency mechanism the caller has verified applies here.
• A known, remaining completion budget exists that can absorb both the delay and a genuinely useful additional attempt.
• The failure semantics are clear enough to say, in advance, which responses are retryable and which are not.
• This layer is the documented retry owner for the operation — see Nested retry ownership below.

Typical fits: service-to-service calls inside a request path, and external APIs that document retryable status codes and rate-limit behavior.

Retries can happen independently at several layers of one call chain: an application client, an SDK, an API gateway, a service mesh sidecar, a queue consumer, or a workflow engine. Each layer typically has no visibility into whether another layer already retried. A local "maximum three attempts" setting does not protect the dependency if the SDK underneath it also retries — the two policies compose, they don't share a budget.

Example (an upper bound, not a universal formula): an application configured for three attempts, calling through an SDK also configured for three attempts, can produce up to nine physical calls to the dependency for one logical request — each application-level attempt triggering a full SDK-level retry cycle underneath it.

There should be exactly one named retry owner per logical operation — the layer responsible for deciding whether, when, and how many times to retry — or, where multiple layers must each retry, a documented shared retry budget across them. Everywhere else, retries should be disabled or reduced to a single attempt.

Retries are never free, even when they work:

• Tail latency grows. A caller that retries once roughly doubles its worst-case latency for that operation.
• Load multiplies, at this layer alone. If 1% of calls need one retry, this layer sends roughly 1% more traffic to the dependency — before any other layer in the chain adds its own retries on top; see Nested retry ownership above for how quickly that compounds.
• Duplicate-risk exposure grows with every additional attempt on a mutation, even a nominally idempotent one, if the idempotency mechanism itself has a gap — see Idempotency keys are conditional, not absolute below.
• Diagnosis gets harder. An aggregate success-rate metric can look healthy while retries silently absorb a real degradation; the retry-to-initial-call ratio has to be tracked as its own signal.

Do not apply a default bounded-retry policy when:

• The mutation has no confirmed receiver-side deduplication and no safe reconciliation path, and a duplicate side effect (a second charge, a second email) would be harmful.
• The remaining completion budget cannot fit both a delay and a genuinely useful additional attempt — retrying anyway just spends the budget on a call that will also fail from lack of time.
• The dependency has already reported a permanent failure: validation, authentication, authorization, or an explicit documented do-not-retry response.
• The caller has already cancelled — the result no longer has value to anyone, though see Cancellation and deadline propagation below for why the downstream work may still be running.
• The work is known to be long-running. Repeatedly extending timeouts to simulate asynchronous processing is a sign the operation belongs behind a queue instead; that is the next topic in this path, not something to improvise here with retry configuration.
• Another layer already owns the retry for this operation — see Nested retry ownership above.

An outcome-unknown failure with no remaining attempt budget is not a failure to paper over with a longer timeout — it is a distinct result. Return a pending outcome, or reconcile against the dependency's own status record, and let the caller resolve it through the product's existing status path, rather than asserting the operation did not happen.

A caller's deadline or timeout only tells the caller when to stop waiting — it does not prove that the downstream work stopped. Cancellation propagation (passing the cancellation signal into every downstream call and dependency) can reduce abandoned work, but only where every layer and dependency in the chain actually supports and honors it. Treat cancellation as advisory, not guaranteed: many transports, protocols, and third-party providers accept a cancel request but do not enforce it, and some cannot be cancelled once accepted. A cancelled or timed-out mutation can therefore still need a status lookup or reconciliation afterward, because its outcome remains unknown regardless of what the local caller decided to do.

A checkout service calls an external payment provider to create a payment intent. The mobile client needs an answer inside a 2.5-second end-to-end completion budget. The provider typically responds in 250 ms but has occasional short 5xx bursts, and returns 429 with Retry-After when its quota is exhausted.

Budget allocation:

• 2.5 s total completion budget, inherited from the client's deadline.
• 300 ms reserved for local validation, mapping, and response handling.
• 700 ms per-attempt timeout for the provider call, subject to remaining budget.
• Maximum two total attempts — the initial call plus one retry, not "one retry" left ambiguous.
• Capped exponential backoff with full jitter between attempts: delay = random(0, min(100ms × 2^(retryIndex-1), 400ms)).

Retry eligibility for this call: retry only on connection failures, a timeout, selected documented 5xx statuses, and 429 — and for 429, only when the greater of the policy delay and the provider's Retry-After still leaves enough budget for the attempt to complete; otherwise stop rather than wait past the deadline anyway. Never retry a 400 validation error or a 401 credential error — those are deterministic, and another attempt would fail identically while spending budget and load for nothing.

On Retry-After specifically: treat it as a policy input and often a minimum delay, not a guarantee that the next request will succeed — it must still fit inside the caller's useful completion budget, and retrying still requires the repeat-safety and ownership checks above. Rate-limit scope varies by provider and can apply per user, per account, per credential, per endpoint, or globally, so one caller's Retry-After does not necessarily say anything about a different caller's chances. If many concurrent requests are being rate-limited at once, fail-fast, admission control, deferral, or moving the work to an asynchronous path is often better than every caller independently retrying against the same limit. Never wait past the useful deadline simply to honor Retry-After.

Reconciliation as a first-class alternative: when a mutation's outcome is unknown and the provider exposes a status lookup or supports safe replay under its documented contract, checking status is often safer than retrying blind — it can resolve to success, a known failure, or a genuine "still pending" without risking a second side effect. It fits best exactly where blind retry is riskiest: outcome-unknown mutations after a timeout or cancellation. It is not free, either — status-checking or reconciling without a bounded budget of its own becomes the same kind of unbounded loop retries can become, so give it the same deadline discipline. Expected outcomes are: return pending, query status now, schedule reconciliation for later, or safely replay once the provider's contract confirms it is safe to do so.

Before shipping a retry policy, confirm:

• The owner of the bounded completion budget is identified, and it is propagated to every attempt.
• Maximum total attempts is stated as a number, not "a couple of retries," and one layer is named as the retry owner for this operation.
• The timing budget — timeout × attempts + backoff — has been calculated as a conservative plan and fits inside the budget, with room for connection setup, queueing, and response handling.
• Failures are classified into success, permanent, plausibly transient, and outcome-unknown, not treated as one bucket.
• The operation's repeat safety has been confirmed against the receiver's documented contract, not assumed from the presence of an idempotency key alone.
• Backoff is capped and jitter is applied so retries from many callers don't synchronize.
• Retry-After is treated as a policy input and minimum delay, honored only if it still fits the remaining budget, with rate-limit scope (per user, account, credential, endpoint, or global) understood.
• Cancellation is propagated where supported, and reconciliation or status-check exists as the fallback where it is not.
• Long-running or budget-exceeding work has an asynchronous path, rather than an ever-longer request timeout.
• Observability distinguishes the logical request from its individual attempts — see below.

Observability note: track attempts as children of one logical request, not as independent events. At minimum, record: a logical request/correlation ID; attempt number and the configured cap; the retry owner/layer; the retry reason; the backoff/jitter delay applied; remaining completion budget at decision time; the deadline/timeout and cancellation result; the dependency's response category; the final logical outcome; a separate count of outcome-unknown results; a separate count of reconciliation/status-checks; and the retry-to-initial-request ratio as a monitored threshold. Keep per-request identifiers out of metric labels — high-cardinality values belong in logs or as attempt spans/events nested under one logical-operation trace, not as metric dimensions.