The retry storm: when the fix for a failure keeps it going
This is the first post in a series about how large systems fail. Each one takes a single pattern and walks through the mechanism, what it looks like when it starts, and the fix, including the one people reach for first that doesn’t work. They’re patterns, so they show up in every system, and none of these posts is about one particular incident.
Retries come first because they’re the most reasonable code in any system. A request failed, the failure was probably transient, so you try again. Each retry on its own is correct. The trouble starts when they all fire together, and no single one of them can see that happening.
The arithmetic
The clearest version of this I know is a toy example from Metastable Failures in Distributed Systems, a 2021 paper by Bronson, Aghayev, Charapko and Zhu and the best short thing written on the subject.
A web application sends one query to a database per user request. Below 300 queries a second the database answers in under 100 ms. Above that, latency gets an order of magnitude worse. The application retries a query once if it hasn’t come back within a second.
It’s running at 280 queries a second, healthy with some headroom, when the network between application and database drops out for ten seconds. Once it’s back, every request and retry that queued up during the outage arrives at once, the database overloads, and its latency blows past the one-second timeout. Now every query gets retried. The application is sending 560 queries a second to a database that can handle 300.
Someone has fixed the network by now. The system is still down. Every query times out, so every query is sent twice, so the database stays overloaded, so every query times out. The paper says it stays that way until the load falls below 150 queries a second or the retries fall below 20 a second, and nothing in the system will push either number down.
Two numbers in there deserve a closer look. The database can do 300 queries a second, but with one retry per query the system is only safe below 150, which makes the real capacity your capacity divided by your worst-case amplification. Between 150 and 300 it looks healthy and has spare capacity for everything except its own retries. The paper calls this the vulnerable state. Most systems spend most of their lives there without knowing.
The other thing to notice is that the trigger was a ten-second network blip, and the retries are what kept the failure alive afterwards. Fix the trigger and wait, and nothing changes.
Layers multiply
That example has one layer retrying once. Real systems have several, each written by someone who reasonably decided their layer should retry.
The Google SRE book is blunt about it: one request at the top can produce a number of attempts equal to the product of the attempts at each layer. Its example has a JavaScript client, a frontend and a backend each making up to four attempts (one try plus three retries), so one user action becomes 64 attempts on the database.
On AWS the layers stack up without anyone noticing:
- The browser’s fetch wrapper retries.
- The API route calls a service through an HTTP client with its own retry policy.
- That service calls DynamoDB through the AWS SDK, which makes up to three attempts by default.
- If any of that runs in a Lambda invoked asynchronously, Lambda itself retries a failed invocation on top.
Each layer picked its own number. Multiply them and you get a total nobody ever agreed to.
A real one
The best-documented retry storm I know of is AWS’s own: the DynamoDB disruption in US-East-1 on September 20, 2015.
DynamoDB’s storage servers periodically ask an internal metadata service for their membership, meaning the partitions they’re supposed to hold. Global secondary indexes, launched shortly before, had made those membership lists much bigger. Responses slowed down enough that some servers didn’t get theirs within the time they allowed, so those servers stopped taking requests and retried the membership request. The retries kept the metadata service overloaded, healthy servers’ routine requests started failing as well, and they joined in retrying. Customer requests settled at an error rate of around 55%.
Nothing got repaired to end it. The operators paused requests to the metadata service, which removed the retry traffic and let it recover enough to accept administrative commands. Then they added significant capacity, and only after that did they turn membership requests back on. That’s the paper’s recovery rule in practice. Take the load away first.
Fixes that don’t work
Exponential backoff without jitter. Backing off exponentially is right. But when every client uses the same formula, clients that failed together retry together, and the traffic arrives in synchronised waves with gaps between them. Marc Brooker’s 2015 post on backoff and jitter simulates exactly this: plain exponential backoff still produces clusters of calls, and randomness is what breaks them up.
More capacity, mid-storm. Scaling up during a retry storm helps only if you add enough to cover the amplified load. In the toy example that means doubling the database, and in the SRE book’s example a lot more. Autoscaling rarely moves that far that fast, and the overloaded component is often one that doesn’t scale at all.
Waiting. This failure keeps itself going. Waiting just extends it.
What works
Retry at one layer. For each dependency, decide which layer owns retries and have the others fail fast. The best owner is usually the layer closest to whatever failed, since it knows best whether the error is worth retrying. Everywhere else, pass the failure up.
Always add jitter. Here’s the formula the AWS SDKs use:
delay = random(0, 1) × min(cap, base × 2^attempt)Brooker calls this full jitter. The randomness spreads retries across the whole backoff window, where plain backoff bunches them at its end.
Give retries a budget as well as a per-request count. A per-request limit caps how bad one request can get, and a thousand requests failing in the same second sail right past it. A budget asks something different: across everything this client is doing right now, how much of its traffic is retries? The SRE book suggests a per-process cap, 60 retries a minute for example. A token bucket does the same job more smoothly. Each retry spends tokens, each success earns some back, and once the bucket is empty the client stops retrying and returns the error.
For AWS calls the SDKs already do this for you (more on that below). For your own calls between services, the whole idea fits in a few lines:
const budget = { tokens: 500 }
async function withRetry<T>(call: () => Promise<T>, attempts = 3): Promise<T> {
for (let i = 0; ; i++) {
try {
const result = await call()
budget.tokens = Math.min(500, budget.tokens + (i === 0 ? 1 : 14))
return result
} catch (err) {
const giveUp = i + 1 >= attempts || !isRetryable(err) || budget.tokens < 14
if (giveUp) throw err
budget.tokens -= 14
await sleep(Math.random() * Math.min(20_000, 50 * 2 ** i))
}
}
}(isRetryable and sleep are yours to fill in, and isRetryable matters. A validation error or an access denied will fail the same way every time.)
Let the server say “don’t”. An overloaded service should answer with a status that tells clients to back off. The SRE book recommends a dedicated “overloaded, don’t retry” response. Some AWS services send a version of this in an x-amz-retry-after header, which the updated SDKs honour.
Deprioritise retries. The paper points out that if retried requests go to the back of the queue, new requests get through, succeed, and generate no retries of their own. That breaks the loop from the inside.
Build the off switch before you need it. Removing load, like the DynamoDB operators did, is the one recovery move that always works. A flag that disables retries or sheds a fraction of traffic makes it a toggle. Without one, it’s a code change in the middle of an incident.
What AWS SDKs change in November
The AWS SDKs announced updated retry behavior on May 20, 2026, and it becomes the default in November 2026. You can opt in early with AWS_NEW_RETRIES_2026=true. Most of what’s above is now built into the client:
- Different waits for different failures. Transient errors (connection resets, 500s, 503s) back off from a 50 ms base. Throttling errors, where the service is explicitly asking you to slow down, back off from 1,000 ms. Both use full jitter, capped at 20 seconds.
- A retry quota in every SDK. Each client has a 500-token bucket. A retry after a transient error costs 14 tokens (up from 5), a retry after throttling costs 5, and a request that succeeds on its first try earns one back. With the default three attempts, the bucket starts draining once roughly 22% of requests fail with sustained transient errors. Once it’s empty, the SDK stops retrying and returns the error immediately.
- Standard mode by default. SDKs that defaulted to legacy mode, which had no standardised retry quota, move to standard mode, so some clients get a quota for the first time.
I think it’s a good change, and you will see it. During a disruption your AWS calls will give up sooner and fail faster, so dashboards will show more errors at shorter latency where they used to show fewer errors at longer latency. Run a load test with the flag set before November, so you meet the new shape in a test and not in the middle of an incident.
There are two gaps the SDK leaves open. Its quota belongs to one client instance in one process, and knows nothing about the few hundred other Lambda instances hammering the same service. It also covers only the SDK’s own layer, so if your code wraps SDK calls in a retry loop of its own, you’re back to multiplying.
The checklist
- For each dependency, write down how many attempts each layer makes and multiply them. That’s your worst-case amplification.
- Divide your capacity by that number. That’s the load at which you become vulnerable, and it’s probably lower than you think.
- Retry at one layer per dependency, and fail fast everywhere else.
- Full jitter, always.
- A retry budget on top of the per-request limit.
- Never retry errors that can’t succeed on a second try.
- A tested way to shed load or turn retries off, built before the incident.
- Before November 2026, a load test with
AWS_NEW_RETRIES_2026=true.