Concepts
Canary, blue/green, and rollback
Intuition
Before serving a new dish to all the tables in the restaurant, the cook sends a single portion to just a few tables and watches to see whether anyone gets sick. If there is a problem, only a few customers were affected and the dish goes back to the kitchen; if it works out, he starts serving everyone. Releasing a new version of software works the same way: testing with a small share of users limits the damage of a hidden defect.
How it works
The chapter starts from the observation that, in Google's experience, most incidents come from binary or configuration pushes, and that tests do not cover everything, so some defects only show up with real production traffic. Canarying is a partial, time-limited deployment of a change, together with its evaluation: the part that receives the change is the canary, the rest is the control, and comparing the two resembles an A/B test. The impact on the error budget is proportional to the traffic exposed to the defect, assuming detection and rollback take similar time with and without a canary; in the chapter's example, a failure in 20% of requests becomes about 1% overall errors with a 5% canary (a simple model that assumes uniform load and 100% failure in the worst case). Canary size and duration must be sufficient to represent the deployment as a whole: it depends on traffic homogeneity and volume, on time of day (performance defects tend to appear only under high load), and on release cadence, and the chapter recommends one canary at a time. Metrics should reflect user-perceived problems, starting from the SLIs, be few (perhaps a dozen at most), and be attributable to the change; a criterion that is too strict produces false positives and one that is too loose increases the chance that a bad canary passes undetected; metrics without these properties can still be useful in a gradual, staged canary, with a small first stage evaluated by the clearest metrics, such as crashes and request failures, and larger later stages. The evaluation must be tied to the release process: if the canary's error rate strays too far from the control's, the rollout pauses and rolls back or calls a person; if they are similar, it proceeds. This requires monitoring broken down by version, since a 1% error in the aggregate can get lost among other sources, and metric windows equal to or shorter than the canary duration; and, since canary and control can share backends, a bad result does not necessarily blame the canary, and a bad canary can affect the control, so it is also advisable to use absolute measures, such as SLOs. The chapter also describes blue/green as two instances where rollback means reverting the change in the router, at the cost of twice the resources, equivalent to a before/after canary and close to an A/B canary if traffic is split between the two.
In practice
In practice, the canary becomes a step built into the release pipeline: a small subset receives the new version, the comparison with the control decides between continuing the rollout and pausing and rolling back or calling a person, and a gradual staged canary (a small first stage, with the clearest metrics, such as crashes and request failures, and then larger ones) also allows the use of less ideal metrics. The chapter treats blue/green as a close concept: rollback is just reverting the change in the router, at the cost of keeping double the resources, and, with traffic split between the two instances, it approaches an A/B canary. In the canary, since canary and control may share backends, it is worth also checking absolute measures, such as SLOs.
Use it in a design
Practice the cases that rely on this concept in the System Design school.