Concepts
Sampling strategies for observability
Intuition
Think of a quality inspector in a giant factory: he does not open every box that comes off the production line to inspect it, because that would stop the whole factory — he inspects a sample, but when he chooses to examine a box, he examines the whole product inside, not just half of it. If a manufacturing defect exists, it will show up again in many other boxes, so there is no need to look at every one to discover the problem.
How it works
Dapper was designed with three explicit goals: application transparency, ubiquitous deployment at very large scale, and very low overhead — and it is precisely this third goal that makes it infeasible to record every request in detail, given the CPU, memory, and network cost that would imply when running continuously across Google's entire fleet. The solution is to sample: early in the request lifecycle, the system decides whether that entire trace will be kept or discarded, and this decision is made for the complete tree, not span by span — either the whole trace is recorded, with all its spans, or none of its spans are recorded, preserving the coherence of the sample and avoiding partial trees that are useless for diagnosis. The first versions of Dapper used a fixed sampling rate (a small, constant fraction of requests); later versions adopted an adaptive scheme, parameterized by a desired rate of sampled traces per unit of time instead of a uniform probability, which avoids oversampling very-high-traffic services and undersampling low-traffic ones. The paper's argument for justifying this deliberate loss of completeness is that, in systems at Google's scale, a problematic execution pattern that occurs once tends to repeat thousands of times, so there is no need to capture every occurrence to detect it.
In practice
This trade-off — fixed rate versus adaptive rate, and an all-or-nothing decision per whole trace instead of per span — is the same dilemma that modern observability systems such as Jaeger, Zipkin and OpenTelemetry face when scaling: in services with traffic that varies a lot between peaks and troughs, adaptive sampling by target rate captures rare events more consistently than a fixed probability, even though more sophisticated strategies (such as error-driven or latency-driven sampling) exist today than those described in the original Dapper.
Use it in a design
Practice the cases that rely on this concept in the System Design school.