Timeouts, retries and pools
Retries, jitter, and the storm
Why backoff without jitter makes a brief outage into a long one.
Exponential backoff spaces out one client's retries. It does nothing at all about a thousand clients, because they all failed at the same instant and they all wait the same interval — so they all come back at the same instant, and the service that was recovering is knocked over again by its own clients.
Jitter is the fix, and it is one line. Draw the delay from a range instead of using the whole of it: the herd spreads across the interval instead of arriving as a wall.
The other half is knowing when not to retry. A retryable failure and a retryable request are two different questions, and a client that only asks the first will happily send a payment twice.
Predict
A service is down for ten seconds. A thousand clients retry with exponential backoff and no jitter. What does the service see when it comes back?
Comes after Connect, read, total: three different clocks.