Timeouts, retries and pools
The pool, and the connection that was already closed
Keep-alive, reuse, and the three different problems a pool has.
Keep-alive lets a connection serve more than one request, which removes a round trip and a TIME_WAIT from every request after the first. A client with keep-alive off does not have a pool; it has a counter of connections it is about to close.
A pool that is too small is the quietest of the three failures: requests queue for a slot while the network sits idle, and no server-side metric shows it. The waiting is entirely on the client.
The third is a connection the origin has already closed. Handing it out trades a handshake for a reset, and a client that does not notice keeps doing it.
Predict
p99 latency is 900 ms; the server reports p99 of 40 ms and is not busy. Where is the time?
Comes after Retries, jitter, and the storm.