Concepts
Commodity hardware and cost per query
Intuition
Think of a delivery company that has to decide between buying a few giant, very expensive trucks that almost never break down, or many ordinary, cheap vans that break down every now and then. What matters to it is not how much each vehicle costs or which is the fastest, but how much it costs to deliver each package. If there are plenty of packages for all the vans and the company knows how to handle a stopped van (by sending the package on another one), the ordinary vans come out much cheaper per package delivered.
How it works
Barroso, Dean, and Hölzle point out that an average Google query reads hundreds of megabytes and consumes tens of billions of CPU cycles, and that peak load reaches thousands of queries per second, which requires a cluster of more than 15,000 commodity-class PCs. The final criterion for choosing hardware is cost per query: capital expenses (with depreciation) plus operating costs, divided by performance. Depreciation weighs heavily because a server rarely lasts more than two or three years. Because the workload is easily parallelizable (different queries run on different processors), the peak performance of each processor matters less than its price/performance, so you buy the CPU generation with the best performance per unit of price. In the paper's example, a rack of PCs costing about US$278,000 (176 2 GHz Xeon CPUs, 176 GB of RAM, 7 TB of disk) is compared with a multiprocessor server costing about US$758,000 (8 CPUs, 64 GB of RAM, 8 TB of disk): the multiprocessor costs about three times more, with 22 times fewer CPUs and three times less RAM, a difference that comes largely from the interconnect and the reliability of high-end machines. To make up for the lack of fault-tolerant hardware (redundant power supplies, RAID, high-quality components), reliability is delivered in software: the service is already replicated to achieve throughput, so fault tolerance comes for free, and a machine that goes down only reduces capacity in proportion to the slice it represented. There is also a physical ceiling: a mid-range server draws about 120 W of alternating current, which works out to about 10 kW per rack and ~400 W/ft², versus 70 to 150 W/ft² in a typical commercial data center. Power and cooling cost about US$1,500 per month per rack, versus US$7,700 of depreciation, so what counts is watts per unit of performance, without paying much more for low-power servers.
In practice
This reasoning is the basis of cost engineering: compare alternatives by cost per unit of work (per query, per request), adding up depreciated capital and operation, and not by the price or peak performance of each machine. It works when the workload has request-level parallelism and servers without private state, as the paper observes for high-volume web servers, and when the application is homogeneous enough that managing thousands of identical machines is feasible. The authors also note that cheaper computation allows spending more per query, with more expensive ranking or a larger index. The limits are real, such as the cooling capacity of commercial data centers.
Use it in a design
Practice the cases that rely on this concept in the System Design school.