Concepts
Load shedding and backpressure
Intuition
Imagine a small restaurant on a night when far more people show up than the kitchen can handle. If the host accepts everyone, the tables wait so long that customers leave before the dish arrives, and the kitchen wastes effort on food nobody will eat. If the host says right at the door that there is no room, those already seated are served on time, and those turned away find out immediately instead of waiting in vain. Rejecting early and explicitly protects those who were already accepted.
How it works
According to the Amazon Builders' Library article on load shedding, latency grows with utilization; once it exceeds the client timeout, requests expire, all the work the server already did on them is wasted, and retries multiply the load, so the overload sustains itself. That is why the article distinguishes throughput (everything that arrives per second) from goodput (the subset handled without error and fast enough for the client to use), and goodput is what the protection tries to preserve. In load shedding, a server close to overload rejects the excess to keep latency low for the requests it accepted, and the ideal result in a load test is a goodput plateau, formed when the service is near full utilization, that stays flat even as more load is applied. Rejecting also costs resources, so goodput eventually drops at some point. The article advocates testing until it breaks, and well beyond, because anyone who has not tested should assume the service will fail in the worst possible way; it treats rejection when there was still capacity as a false positive; and it calls for instrumenting who was rejected and on which operation, keeping fast failures out of the latency metrics, because they lower the median and hide the bad latency of the successful requests, as in the example of a service that rejects 60% of traffic. Queues hide the problem: bounding their size and wait time and discarding what has become too old (or using LIFO, when the protocol allows) frees the server for new requests, which have a better chance of succeeding; at the load balancer, the article recommends spillover, which fails fast instead of queueing the excess (unlike the surge queue); outside the article, pushing saturation back to the sender like this is usually called backpressure. To avoid wasting work, the client can send a timeout hint, and the server abandons a request that has already expired; in addition, it prioritizes what completes work already started and the load balancer's ping, because if the server does not respond in time, the load balancer stops sending it requests for a period; the article also recommends configuring the maximum connections of the load balancer and the proxy high, without exceeding the server's number of threads, processes, and file descriptors, so that resources remain for health checks, and letting the server do the more precise shedding using local information. In a service that depends on many servers in parallel, The Tail at Scale paper points out that several layers of queues in intermediate servers and switches amplify latency variability, and that the final latency is dictated by the slowest component; the paper warns that indiscriminately resending adds unacceptable load, and so one of the hedged request variations only resends to a second replica after the first exceeds the expected 95th-percentile latency for that class of requests, which limits the extra load to about 5%.
In practice
In a service with a load balancer in front, a safe configuration is to let the load balancer fail fast (spillover), set its maximum connections high without exceeding the server's threads and descriptors, so that resources are left over for health checks, and let the server apply precise load shedding using local information. Use this whenever clients have timeouts and retry, also capping hedging at a ceiling of extra load, and validate with overload tests taken all the way to breaking. Measure rejections by client and operation and do not let fast error responses count toward latency, or the median will look great even while the latency of successful requests is terrible.
Use it in a design
Practice the cases that rely on this concept in the System Design school.