Concepts
Distributed tracing / spans
Intuition
Imagine ordering an item at a restaurant that is actually a hub for several suppliers: the waiter writes down your order, passes it to the kitchen, the kitchen sends for an ingredient from a third-party warehouse, and the warehouse calls a courier. If the dish takes a long time, nobody alone knows where it got stuck — each person only saw their own little piece. A trace is as if each step stamped a receipt saying 'I started here, I was called by that receipt over there', and in the end you join all the receipts into a single tree, seeing exactly who called whom and how long each one took.
How it works
Dapper represents the complete path of a user request as a causality tree called a trace, identified by a unique ID shared by the whole tree. Each node of this tree is a span: a unit of work with a readable name, its own identifier, and the identifier of its parent span — it is this pair (span id, parent id) that reconstructs the 'who called whom' edges. Each span records the time interval in which that work occurred (for example, a specific RPC call) and can carry additional annotations that the application voluntarily inserts with business information. The central point of the design is that this tree is not assembled by each team writing manual instrumentation code: Dapper instruments a small number of common libraries (RPC layer, threading, flow control) used by practically every Google service, and it is this shared instrumentation, not application code, that creates and links the spans automatically. As a result, most of a trace's tree is assembled transparently; manual annotations are an optional complement, and the paper reports that most spans and the vast majority of traces contained at least one application-specific annotation, showing that teams used this voluntary mechanism when useful, even without being required to.
In practice
This model of a tree of spans with a name, span id, and parent id is exactly the vocabulary that Zipkin, Jaeger, and the OpenTelemetry standard use today to visualize the path of a request through microservices; Dapper's practical lesson is that tracing only scales in a large organization if the instrumentation comes from shared libraries (SDKs, RPC middleware), and not from the discipline of each team in manually annotating its own service.
Use it in a design
Practice the cases that rely on this concept in the System Design school.