Timeouts, retries and pools
Connect, read, total: three different clocks
Three timeouts that sound like the same setting and are not.
A connect timeout covers getting a connection — the handshake, and the TLS handshake if there is one. A read timeout covers idle time waiting for bytes to arrive, and it restarts every time some do. A total budget covers the whole thing including every retry and every backoff in between.
Only the third is a promise to the caller. The first two are per-attempt, so a client with a five-second connect timeout, a ten-second read timeout and three attempts can take forty-five seconds to fail while every individual setting looks reasonable.
A total budget is armed once and never re-armed. When it expires the request is over, whatever it happened to be doing — waiting on a socket, or sitting in a backoff.
Predict
Connect 5 s, read 10 s, three attempts, backoff 1 s and 2 s, no total budget. What is the worst case?