Idempotency

Idempotency is not "add an idempotency key to every POST." It is the decision to give a specific operation a stable identity and coordinated duplicate handling, because that operation may be attempted more than once and a duplicate effect would cost something real. It assumes the retry behaviour covered in reliable-workflows.timeouts-retries-backoff-jitter — this guide is about what happens when one of those retries lands on a server that already processed the first attempt, not about deciding whether or how to retry.

When not to reach for a key by default

• Read-only operations. Nothing to coordinate; re-execution has no effect.
• Naturally idempotent state-setting. PUT /users/123 { "email": "…" } already converges to the same state no matter how many times it lands — see Operation identity for why "same request" still needs care.
• Low-risk, low-cost duplicates. A duplicate "mark notification as read" is not worth a deduplication store.
• Operations already protected by a business invariant — e.g., a unique constraint on (user_id, plan) that makes a second "subscribe" a no-op at the database layer. This is real protection, but it is narrower than it looks: it stops a second *row*, not a second *side effect* (a second welcome email can still fire from the same insert-conflict path) and it says nothing about a request still mid-flight — see Lifecycle and concurrent claims.

"Is this idempotent?" is ambiguous until you say idempotent *with respect to what*. These are not the same thing:

A workable contract states, in advance:

• The key identifies one intended business operation, not a transport attempt — a retry of the same call must reuse the same key; a genuinely new order must not.
• The server binds a safely-comparable representation of the request to the key at first use. Comparing full payloads (canonicalizing JSON, ignoring field order, deciding which fields are significant) is a real design problem, not a one-line hash — don't reach for full payload hashing unless you're prepared to keep the canonicalization rules deliberate and versioned.
• If the same key arrives with a materially different request, that is a conflict, not a replay — return a defined error, don't silently execute the new payload or replay the old response.
• Key scope must include every identity boundary that could otherwise collide: tenant, account, endpoint/operation type, and anything else that makes "abc-123" from customer A different from "abc-123" from customer B.
• For workflows built around a resource the caller already names — "the payout for invoice_9f2" — a stable business identifier is often a better key than a client-generated random one: it can't be forgotten, regenerated, or reused incorrectly across retries.

A unique index prevents two idempotency *records* from existing for the same key. It does not, by itself, tell you whether the *operation* those records refer to is still running, finished, or safe to retry — that requires an explicit state.

The pseudocode sketch below is one way to write the same states down, not a second, different model:

absent
  -> processing   (the claim: one attempt now owns this key)
  -> completed    (safe to replay: return the recorded outcome)

processing
  -> completed
  -> retryable / failed / recovery-required   (the claimant died or errored)

The right shape of this state model — whether failed is retryable, whether there's a distinct unknown state, how a stuck processing row gets recovered — depends on the operation and the chosen recovery strategy. There is no universal version of this table; a payment claim and a notification claim recover differently.

Four separate mechanisms get collapsed into "use a unique index" far too often. They solve different problems and are often combined:

A duplicate that arrives while the first attempt is still processing is not the same problem as a duplicate that arrives after completed — see Replay policy by state. And a claim that never resolves because its owner crashed needs an explicit recovery path (a lease timeout, a reconciliation job) or it silently blocks every future retry forever.

A local claim and a local transaction only cover what's inside your own transactional boundary. They cannot tell you, with certainty, what a remote system did.

What actually closes this gap:

• Propagate the key downstream. If the provider supports idempotent requests, pass it the same stable key (or a derived, still-stable one) so a retried call at the provider is also deduplicated there.
• Transactional outbox is the pattern for coordinating a local state change with a message or event that must be published — see reliable-workflows.transactional-outbox. It does not, by itself, resolve *this* problem (a remote synchronous call whose result is unknown); it solves the narrower one of "local write and publish must not diverge."
• Reconciliation is the pattern for the case above: periodically checking the provider's own record of what happened and repairing local state to match — because "unknown" is not a state you can leave a payout in indefinitely.

Idempotency is one piece of a workflow's reliability design, not a replacement for outbox, sagas, observability, or reconciliation — see reliable-workflows.exactly-once-claims-reality and reliable-workflows.sagas-compensating-transactions for the surrounding pieces.

"Return the stored response" is the right answer for exactly one of these states. Replay policy is part of the contract, decided per state:

A stored response is not automatically safe to replay years later, or to replay for a caller you haven't re-authorized. If the response can go stale (a price, a status) or contains anything sensitive, replaying it must still pass the same authorization check as the original request — a matching key is not a substitute for checking who's asking. In many designs, storing a compact outcome or reference (a status, a transaction ID) rather than the full original response sidesteps both problems at once.

Don't standardize on a single HTTP status code for these cases across every API — which code fits which row above is a choice each API's contract makes explicitly, not a universal convention.

Three different retention questions get merged into "TTL" far too often:

• Correctness retention — how long a late duplicate must still be caught. Driven by the realistic tail of retry and delayed-delivery behaviour for *this* operation, not a round number. A background job fed by a queue with hours of possible redelivery delay needs a much longer window than a synchronous HTTP retry loop.
• Operational/debug retention — how long the claim record is useful for investigating an incident, independent of correctness.
• Audit/compliance retention — where regulation or finance requires keeping a record regardless of the other two.

These can have different lifetimes for the same key, which is a reason to think about what you're storing (a full response vs. a compact outcome) and where, rather than one blanket expiry.

Any numbers below are illustrative, not a standard to copy:

Whatever the retention window, remember that request or response bodies stored for replay may contain sensitive data — that's a data-minimisation decision (store a reference, not the payload) independent of how long the entry lives.

"Exactly-once" is precise only when it names a boundary and a failure model. Four different claims get flattened into that one phrase:

• At-most-one committed effect inside a named local transactional boundary — real, and provable, for writes that share one transactional datastore.
• Duplicate suppression within a service — what an idempotency key actually buys you: the service won't knowingly re-execute for a key it has already completed.
• Provider-level idempotency — a guarantee the *downstream* system makes about its own handling of a repeated key, which your own service can rely on only to the extent that provider documents and tests it.
• End-to-end exactly-once across independently failing systems — the broad claim, and the one that is almost never actually justified, because it requires every hop in the chain to have made one of the narrower guarantees above and for those guarantees to compose without a gap.

Broad end-to-end exactly-once claims are usually unjustified — not because exactly-once is impossible in every sense, but because most systems making the claim haven't actually closed every failure window in the chain (see the example above). Bounded guarantees, stated precisely, are meaningful and worth stating. In practice, at-least-once delivery, plus idempotent processing at each hop, plus reconciliation for what idempotency can't cover, is the design that actually ships.

A transaction is powerful and narrow:

• It can atomically create the local business mutation and its local idempotency claim together, when both live in the same transactional datastore — that's the guarantee behind the lifecycle table above.
• It does not extend to a remote API call, a message broker publish, an email send, or a second, independently-committed database — none of those roll back if your transaction does.
• Publish after commit (write locally, then publish) leaves a gap where the write succeeded but the publish can still fail — the case a transactional outbox exists to close; see reliable-workflows.transactional-outbox rather than re-deriving it here.
• Call before commit (call the remote system, then commit) leaves the opposite gap: the call can succeed and the local commit can still fail, leaving an effect with no local record of it.
• A workflow that runs long — waiting on a person, a batch process, a slow provider — should not be modelled as one long-held transaction. It needs explicit state (the processing/completed shape above) and an explicit recovery path, not a lock held open across the wait.

A merchant-payout API pays a merchant's linked bank account for a period's sales. POST /payouts accepts an idempotency key from the caller (typically the merchant's own back-office system, itself retrying after a timeout).

Without a key, or with the concurrency gap in Lifecycle and concurrent claims left unhandled: the back-office system times out waiting for a response, retries, and both requests reach the payout service before either has claimed the key. Both execute. The merchant is paid twice for the same period — a real, refund-requiring, reconciliation-triggering mistake, not a cosmetic duplicate.

With the design above: the second request's atomic claim fails (the first already owns processing or completed), so it never reaches the bank call. If the first request is still processing when the second arrives, the second gets a defined "still processing, check status at /payouts/{id}" response rather than a guess. If the bank call itself is in the ambiguous state from the failure-window example above, reconciliation — not another retry — is what resolves it.

• Retries or duplicate deliveries are arriving after the chosen correctness-retention window — the window was sized for the wrong tail.
• processing claims are accumulating without resolving — a sign recovery/lease logic is missing, not that duplicates are rare.
• The operation now crosses a service or provider boundary it didn't before, and the existing local-only claim no longer covers the new failure window.
• Idempotency storage itself becomes a bottleneck or a single point of failure for the request path.
• Reconciliation volume is growing — a sign the "unknown" state from an external call is being hit often enough to need first-class handling, not ad hoc fixes.
• The operation has become asynchronous or long-running, and the original synchronous request/response idempotency model no longer fits — see reliable-workflows.queues-async-processing.
• A downstream provider ships its own idempotency feature — that's usually a signal to propagate your key to it rather than only deduplicating locally.

• The business operation identity is explicit — what exactly is being deduplicated, and at what scope (tenant, account, endpoint)?
• The conflict policy is defined: what happens when the same key arrives with a materially different payload?
• Claiming is atomic and race-safe — two simultaneous arrivals cannot both execute.
• processing state exists, and abandoned claims (crashed owner, expired lease) have a recovery path.
• Completion and replay policy is defined per state (completed / processing / conflicting / failed / expired), not just "return the cache."
• Retention window is chosen from realistic retry/delayed-delivery tails, not a round default — and correctness, debug, and audit retention are considered separately where they differ.
• Storage decision addresses sensitive data — full response vs. compact outcome, and who's authorized to see a replay.
• The idempotency key is propagated to any downstream provider that supports its own deduplication.
• There's an explicit plan — reconciliation, provider status check, or similar — for the case where an external call's outcome is genuinely unknown.
• Duplicates, conflicts, stuck claims, expiries, and recoveries are all observable, not just the happy path — see reliable-workflows.observability-reliability for what "observable" should mean beyond a log line.