Queues and asynchronous processing

Move work off the request path when delayed completion is acceptable and the benefits of decoupling — load smoothing, failure isolation — exceed the costs of eventual completion, duplication, and operations. Do not queue work when the caller needs a definitive result now: a queue trades an immediate outcome for a durable, delayed one, and that trade is not free.

A queue decouples producer availability and rate from consumer availability and rate — it does not make the underlying work disappear. It turns immediate dependency and latency risk into backlog, delayed completion, duplicate delivery, and operational responsibility that someone has to own.

• Producer — publishes a message describing work to be performed, including a stable identifier and enough metadata to route, trace, and safely retry it.
• Broker — durably accepts, stores, delivers, and redelivers messages according to queue semantics.
• Consumer — receives a message, performs the work, records its result, and acknowledges successful handling.
• Visibility timeout — the interval during which a received, unacknowledged message is hidden from other consumers; expiry makes it eligible for redelivery.

Why arrival and service rate matter: call the rate messages arrive λ and the rate consumers successfully complete them μ. Backlog grows while λ > μ and only drains while μ > λ. If that imbalance persists, latency rises with depth even when the broker itself is perfectly healthy — the fix is capacity or admission control, not a bigger queue.

Why duplicates are normal, not a bug: at-least-once delivery means a message can be delivered more than once — most commonly because a consumer crashes or its visibility timeout expires after it performed the side effect but before it acknowledged. The queue cannot know the work already happened; only the consumer, by recording state before acknowledging, can make a repeat delivery harmless.

Ordering is local, not global. A queue, partition, or key can preserve order among the messages that share it. Adding consumers or partitions to increase throughput commonly weakens or removes that guarantee outside the key it was scoped to — treating a FIFO queue as proof of business-wide ordering is a frequent source of bugs once concurrency increases.

Move work off the request path when several of the following hold:

• The work is slow relative to the caller's acceptable response time.
• Demand is bursty, and a queue can smooth it into steady consumer throughput instead of provisioning for peak in the request path.
• The work is independently retryable without the caller waiting for that retry.
• The work depends on a service that is temporarily rate-limited or unreliable, and delaying completion is acceptable.
• The product can honestly tell the user "accepted," not "done," and has a real status path for what happens next.

A queue also gives the producer consumer-controlled load: consumers pull work at a rate they can sustain, rather than the producer pushing at whatever rate requests arrive.

A queue is not free reliability — it replaces one set of problems with another:

• Completion becomes eventual, with explicit pending / sent / failed states the product has to expose rather than a single synchronous response.
• Duplicate-safe side effects become mandatory. At-least-once delivery is the default; the consumer must make repeated processing of the same message safe.
• Poison messages need a policy. A malformed or permanently invalid message that a consumer cannot process will otherwise be retried forever, consuming capacity that healthy messages need.
• Monitoring and capacity become someone's job. Depth, oldest-message age, consumer utilization, and dependency quota all need an owner and alerting, not just the broker's default dashboard.
• Failures become less visible immediately. A production incident downstream of a queue can take minutes to show up as user-facing symptoms, because the queue is absorbing it in the meantime.

Do not introduce a queue when:

• The caller needs an immediate, authoritative result to continue — an authorization decision or a validation the next step depends on.
• The work is small, fast, and reliable enough that synchronous handling is simpler and just as good.
• Delayed or stale effects would be actively harmful to the interaction, and there is no credible pending-state or cancellation model to fall back on.
• There is no plan for how the system will answer "what happened?" after the fact — a queue with no durable application state behind it is not a substitute for one.

Queuing a database write does not fix an undefined consistency contract; it just moves the same undefined contract one hop later, with less visibility into when it breaks.

A subscription billing API charges an invoice synchronously through a payment provider, then must send a receipt email through a third-party email provider and record its delivery status for support. The payment response must return within two seconds; receipt delivery is expected within five minutes, not instantly, and a customer must never receive a duplicate receipt for the same paid invoice. The email provider allows 100 requests per second; typical load is 20 paid invoices per second, with marketing-driven peaks to 250 per second for ten minutes.

Flow:

1. The API charges the invoice, then stores payment success plus a receipt-notification state of pending, keyed by a stable notification ID receipt:{invoice_id}.
2. The API returns 200 Payment confirmed with receiptStatus: pending, decoupling payment success from email delivery.
3. The producer publishes a message carrying the notification ID, invoice ID, correlation ID, and schema version — not the rendered email body.
4. A consumer receives one message under a visibility timeout and checks whether the notification is already sent.

1. If already sent, the consumer acknowledges without resending. Otherwise it sends, stores sent plus the provider message ID, then acknowledges.
2. For 429, 5xx, or timeout responses, the consumer does not acknowledge — the broker redelivers using bounded backoff. For an invalid recipient address, the consumer records a terminal failure and routes the message to a dead-letter queue rather than retrying.

Configuration choices: consumer concurrency starts at 50 — below the provider's 100 requests/second limit — and is tuned from measured send latency and headroom, not from queue depth alone. Visibility timeout starts conservatively above the 99th-percentile provider call plus persistence and acknowledgement time (for example, 60 seconds against a typical 10-second send). Retryable failures (timeouts, 5xx, 429) get capped exponential backoff with jitter, up to five attempts; a malformed message, missing invoice, or invalid recipient is terminal and is never retried.

Before shipping a queue-backed workflow, confirm:

• The async acceptance contract is explicit: the caller is told "accepted," never "done."
• Every message carries a stable identity used for deduplication and idempotency, not a fresh key per attempt.
• The consumer records outcome state before acknowledging, never after.
• Visibility timeout is derived from measured processing-time percentiles, not guessed.
• Retryable and terminal failures are classified explicitly, with a capped attempt count for the former.
• A dead-letter queue exists, has an owner, and has a documented repair-and-replay path.
• Consumer concurrency is bounded by downstream capacity, not scaled on queue depth alone.
• Depth and age of oldest message are both monitored, with alerts on age — the more direct latency signal.
• Ordering requirements (none, per-key, or global) are stated explicitly, not assumed from FIFO delivery.
• A durable, user-visible status exists for pending / sent / failed outcomes, independent of message retention.