Retries and idempotency — telling safe retries apart

"Retry on failure" is the right principle, but it isn't enough on its own. Some failures must not be retried, and getting the retry count and interval wrong makes incidents worse.

Two questions decide it. Is this failure safe to retry? If so, how many times and at what interval?

What you'll learn in this chapter

• Splitting failures into determined, indeterminate, and overload to see which are safe to retry
• Deriving when a retry is safe from idempotency
• Why exponential backoff needs jitter
• Why the ceiling is set by two values — attempt count and total elapsed time

Prerequisites

Builds on safety and idempotency from HTTP, and the three-way classification from Error handling.

Split failures into three

Retry decisions start from classifying the failure.

Kind	What happened	Retry
Determined failure	Server processed and answered "can't" (4xx other than 429)	No
Indeterminate	No response (timeout, connection cut), or a response that doesn't tell you whether it completed (500, 502, 504)	Conditionally
Overload	Peer answered "not now" (429, 503)	After waiting

Retrying a determined failure changes nothing. If the request content is invalid, it stays invalid no matter how many times you send. Beyond that, you add load, pollute logs, and hide real incidents.

Overload is retryable, but sending immediately makes it worse. The peer is saying they can't cope; you need to space out.

Overload isn't only a caller-side concern. How the receiver prepares for it (add capacity, don't route to failing nodes, drop some features) is covered by Availability and redundancy.

Indeterminate is the trickiest

On timeout, you can't tell whether the request was processed.

Assuming "timed out so not processed" is wrong. If the response was cut in transit, processing completed. Retrying here creates two orders.

Conditions for safe retry

Only two cases are safe to retry.

Condition	Basis
Idempotent method	Same final state regardless of send count (HTTP definition)
Idempotency key is in play	Server detects duplicates and processes only once

GET / PUT / DELETE are idempotent, so retrying them on indeterminate failure is safe. POST isn't idempotent, so without an idempotency key it must not be retried.

What the idempotency key guarantees

The client creates a unique value per request and sends it in a header. On retry, send the same value.

POST /api/orders HTTP/1.1
Idempotency-Key: 9f2b1c4e-6a3d-4f8b-9c1e-2d5a7b0f3e4c

The server uses this to decide "is this a retry". Three design points.

1. The client creates the key. Server-issued keys make the initial request-to-get-a-key indeterminate too — you shift the problem one step.

2. The second call returns the first call's result. Not an error. The retrying client wants "did it get created", and returning 409 leaves them unsure.

3. Push the duplicate check to a database constraint. Checking "was this already processed" in code and then saving leaves a window between check and save. Same structure as the contention in Locks and isolation levels.

Concrete table design and implementation are covered by Laravel API Development Guide — Transactions and concurrency.

Designing the interval

Fixed intervals clog

"Send every 3 seconds up to 5 times" is simple but doesn't work when the peer is down. Giving up at 15 seconds means you can't outwait recovery.

Exponential backoff

Wait times double each round.

Attempt	Wait
After 1st failure	1 second
2nd	2 seconds
3rd	4 seconds
4th	8 seconds

Retry quickly early to absorb transient blips; stretch if it continues, giving the peer time to recover. It's also for the peer's sake. If a recovering server gets a flood of retries, it goes down again.

Without jitter, it becomes a wave

Backoff alone isn't enough. Clients that failed at the same time retry at the same time.

100 clients fail at the same instant
  → All retry 1 second later (a wave of 100 requests)
  → All fail
  → All retry 2 seconds later (another wave of 100)

Doubling the interval doesn't change the wave's shape. It just widens the gap between peaks. The phenomenon has a name — thundering herd.

Breaking it is jitter — mixing randomness into the wait. Instead of "1 second later", "somewhere between 0 and 1 second later" spreads retries along the time axis. The recovering server sees a thin, sustained flow instead of a lump.

For retries in a distributed system, backoff and jitter come as a pair. Backoff alone leaves clients still lockstep, so widening intervals doesn't erase the waves.

When the peer specifies the wait

HTTP/1.1 429 Too Many Requests
Retry-After: 30

If Retry-After is present, it takes priority over your calculation. The peer is saying "30 seconds is when I can accept", and sending earlier is refused.

The ceiling is two values

Count alone isn't enough.

Ceiling	Prevents
Attempt count	Infinite retrying
Total elapsed time	Making the caller wait forever

With exponential backoff, wait times grow quickly, so count alone leaves total time hard to read. Starting at 1 second and trying 8 times exceeds 2 minutes of waiting.

Back out from the timeout budget

Retries that outlast the upstream timeout are pointless.

Browser → API server (timeout 10 s)
              ↓
           API server → Payment service

If the browser gives up at 10 seconds, retrying the payment service for 30 seconds is wasted. The browser has already disconnected; there's no one to receive the result.

So make the inner total elapsed time shorter than the outer timeout. The deeper the call stack, the less time each inner layer can allocate.

This mirrors the "where to catch" thinking from Error handling. The time budget is also allocated across layers.

Applying to database contention

In Locks and isolation levels, we noted that serialization failures and deadlocks are practically caught by retries. There's a caveat.

On contended rows, increasing retry counts makes things worse.

Situation	Increasing count
Occasional collisions	Recovers. Increasing the count helps
Constantly concentrated on the same row	Wait queue lengthens; overall wait time worsens

For a hot-item stock row with concentrated updates, retries don't fix the root cause. "No matter how many tries, I can't get it" — while consuming server resources continuously.

What's needed here isn't tuning retries but design that reduces the contention itself. Split stock across multiple rows, funnel updates through an asynchronous queue that serializes them, and so on.

Retries absorb transient failures. They don't solve structural contention.

Decision procedure

When a POST without an idempotency key times out, you can't choose to retry. You can query state. Ask via GET whether "this order was created". Designing with an idempotency key from the start removes this branch.

Layers that stop double processing

For irreversible operations like payment, layer defenses.

Layer	Stops	Enough alone?
UI double-click prevention	User operation mistake	Not enough (invalid on connection loss)
Idempotency key	Client retries	Nearly enough
Database unique constraint	Everything that slipped through	Last defense

The top two work only on well-behaved clients. The last line to depend on is the database constraint. Constraints aren't affected by race conditions and don't leave a gap between check and write.

Common misconceptions

"Timed out so not processed." — Processing may have completed. The correct state to hold is "unknown".

"More retries are kinder." — They add load to the peer and make the caller wait. Always cap it.

"Adding exponential backoff is enough." — Without jitter, retries synchronize into waves.

"The server issues idempotency keys." — Clients do. Server-issued makes the key-fetch request itself indeterminate.

"Duplicate idempotency-key requests return an error." — Return the first call's result. Errors leave the retryer without the outcome.

"Retries solve contention." — They handle transient failures; they don't handle concentrated same-row contention. Increasing counts worsens it.

Exercises

Q1. A payment API call timed out. There's no idempotency key. What do you do?

Answer and explanation

Answer: Don't retry. Query the payment state.

POST isn't idempotent, so retrying risks double-charging. Timeout means "unknown outcome", not "not processed", so first pin the state.

Steps:

1. Check via GET on the payment service's lookup API whether payment for the order settled
2. If settled, align your record to it
3. If not settled, execute again

If there's no lookup API, there's no safe automatic recovery. Fall back to manual confirmation.

To avoid this situation, add an idempotency key. With one, resending with the same key returns the first call's result — no lookup, no manual step. For irreversible operations, add it at design time.

Q2. You set "5 retries with exponential backoff starting from 1 second" on an external API call. During an incident, user-facing timeouts increased. What's happening?

Answer and explanation

Answer: The retry total elapsed time exceeds the caller's timeout.

Doubling from 1 second gives 1 + 2 + 4 + 8 = 15 seconds of waiting. Add the time each attempt itself takes. If the external API takes 10 seconds to time out, 5 attempts total 50 seconds — 65 seconds combined.

If the browser or upstream server gives up at 30 seconds, from that point retries are wasted. Your side keeps going, occupying connections and threads. Under incident load, this stacks up and starves other, non-retrying requests.

Two things to decide:

Ceiling	How to decide
Total elapsed time	Shorter than the upstream timeout
Attempt count	Back out from the total elapsed time

Don't decide count first — derive count from the time budget. If the budget is 5 seconds, waits of 1s + 2s put the ceiling at 3 attempts.

Q3. You increased deadlock retries on stock updates from 3 to 10. Errors dropped, but overall response time worsened.

Answer and explanation

Answer: Contention itself didn't decrease; the wait queue lengthened.

Retries didn't turn failures into successes — they turned failures into wait time.

Change	Content
Error rate	Falls (eventually succeeds)
Response time	Rises (many rounds of waiting)
Server resources	Occupation stretches (connections and transactions held)

And it doesn't only affect stock updates. Holding transactions and locks longer makes other processes touching the same table wait too.

If updates concentrate on the same row, what's needed is contention-reducing design.

• Fix lock acquisition order to reduce deadlocks themselves (Locks and isolation levels)
• Split stock across multiple rows to remove the hot single row
• Push updates to a queue and serialize

Retry counts should be set only after doing that — to absorb transient failures.

Summary

• Split failures into determined (4xx other than 429), indeterminate (no response, 500/502/504), overload (429/503). Retry eligibility changes
• Timeout is "unknown", not "not processed"
• Retrying is safe only on idempotent methods, or when an idempotency key is in play
• Idempotency keys are created by the client and, on the second call, return the first's result. Push detection to a database constraint
• Backoff and jitter are a pair. Without jitter, retries synchronize into waves
• Cap by both attempt count and total elapsed time. Total time is shorter than the upstream timeout
• Retry-After takes priority over your calculation
• Retries handle transient failures; they don't handle structural contention. Increasing counts worsens it

Related references

• Laravel API Development Guide — Transactions and concurrency — Table design for Idempotency-Key and implementation of duplicate detection via unique constraint
• Laravel API Development Guide — Queues in production — Setting job attempt counts and waits, handling failed jobs
• RFC 9110 — HTTP Semantics — Definition of idempotent methods and Retry-After
• RFC 6585 — Additional HTTP Status Codes — 429 Too Many Requests
• AWS Architecture Blog — Exponential Backoff And Jitter — Verification that backoff alone leaves retries synchronized (thundering herd), and that jitter breaks the lockstep

What to read next

• Origins and CORS — Moving into the browser boundaries topic from here
• Locks and isolation levels — How serialization failures and deadlocks arise
• HTTP — How idempotency is determined per method