Concepts
Tail latency amplification (fan-out)
Intuition
Think of a relay race in which the team's final time is the time of the last runner to cross the line, not the average of the four. If you have only one runner, a bad day for them is rare. But if your 'team' has a hundred runners running at the same time and you only finish when the last of them arrives, it takes just ONE of the hundred having a bad day for the whole team to look slow — and with a hundred runners, someone having a bad day stops being rare and becomes almost a certainty in every race.
How it works
Dean and Barroso show that, when a user request needs to query dozens or hundreds of servers in parallel to assemble the response, the final latency is determined by the slowest component among all of them, not by the average latency of each one. In the paper's numerical example: if each individual server has a p99 of 1 second (even though it normally responds in 10ms), and a request waits for 100 of those servers in parallel, about 63% of user requests end up taking more than 1 second, because a single slow server among the hundred is enough to delay the entire response. In a real Google system, a p99 of 10ms per leaf request becomes ~140ms of aggregate p99, and the slowest 5% of requests account for half of all the latency observed at the p99. This latency variability does not come from faulty hardware: the paper attributes it to normal components of any shared machine — contention for CPU/cache/disk/network with other processes, background maintenance, garbage collection pauses, power management lowering and then restoring the processor frequency, and queueing effects when bursts of requests pile up in internal queues. Since these sources of variation are practically unavoidable, the small, constant fraction of slow responses from each individual server becomes, via fan-out, a common event at the level of the whole request.
In practice
This effect is why operations teams of high fan-out systems (search, microservice aggregators, partitioned databases) monitor p99 and p999 at the points where many parallel calls converge, instead of looking only at the average latency of each isolated service — the average hides exactly the component that dominates the user experience when the number of dependencies per request grows.
Use it in a design
Practice the cases that rely on this concept in the System Design school.