Papers
Canarying Releases (Site Reliability Engineering Workbook)
Alec Warner, Štěpán Davidovič, Alex Hidalgo, Betsy Beyer, Kyle Smith, Matt Duftler · Google SRE Workbook (O'Reilly), cap. 16, 2018
The problem: in Google's experience, most incidents come from pushes, and tests do not catch everything
The chapter defines release engineering as the set of processes and artifacts that take code from a repository to a running production system, and lists principles such as reproducible and automated builds, automated tests, deploys done by machines, and small, self-contained changes. Automating releases makes it possible to release more often; in software with a nontrivial rate of change, each artifact ends up carrying fewer changes, and it becomes cheaper and easier to roll one back when there is a bug. The authors treat delivery speed and reliability as goals that are usually seen as opposed, but say you can ship as fast as possible while meeting the product's reliability goals, using SLOs and error budgets to measure the impact of releases. The starting point is an observation from Google's experience: most incidents are triggered by binary or configuration pushes. Even with a fully automated pipeline, the test environment is not identical to production and the tests probably do not cover every scenario, so some defects only show up with real traffic. If the release goes everywhere instantly, the defect goes everywhere too. A safer alternative is the canary: initially expose only part of production traffic to the new version, to detect defects as early as possible with the least possible impact on the service.
The definition: canary, control, and the cost measured in error budget
The chapter defines canarying as a partial, time-limited deployment of a change to a service, together with its evaluation, which helps decide whether the rollout should proceed. The part of the service that receives the change is the canary; the rest is the control. In general the canary affects a much smaller subset of production than the control, and the canary's behavior is compared with the control's; for this reason the process is described as effectively an A/B test. The authors list three requirements: a method to deploy the change to a subset of the service's population, an evaluation process that tells whether the change is good or bad, and the integration of these evaluations into the release process. The value of the process shows when it detects bad releases with high confidence and identifies good ones without false positives. In the chapter's example, an application simulates failure on 20% of requests. With a global deployment, the service risks a 20% error rate; with a canary of 5% of the population, the overall error rate is 1%. The impact on the error budget is proportional to the traffic exposed to the defect, and the authors assume that detection and rollback take similar time in both cases. They warn that it is a simple model, which assumes uniform load and 100% failure in the worst case, and recommend the simplest model that meets the technical and business goals.
Choosing the canary population and duration
According to the chapter, you don't need a huge canary population to detect basic critical conditions, but a representative canary requires decisions along several dimensions. Size and duration: the canary must be large enough and last long enough to represent the deploy as a whole; ending after a few requests gives no useful signal when queries are diverse, and the higher the processing rate, the less time you need to get a representative sample. Traffic volume: in general, the more homogeneous the requests, the less traffic is needed. Time of day: performance defects tend to show up only under high load, so an off-peak deploy would probably not reveal them. Metrics: query success can be evaluated quickly, but others, such as queue depth, may need more time or more population. Duration also depends on cadence: a team that releases daily cannot leave a canary up for a week if it runs one canary at a time, a team that deploys weekly has time for very long canaries, and a team that deploys continuously needs much shorter canaries. The authors strongly recommend one canary at a time, because simultaneous canaries increase mental effort and, if they overlap, the risk of signal contamination. They say these requirements can conflict, and that, with a history of past canaries, you can choose parameters based on the typical rate of evaluation failures instead of hypothetical worst-case scenarios.
Metrics, false positives, and false negatives
The chapter says that a single metric, such as success rate, is not enough: ten times higher latency or ten times higher memory usage can also indicate a problem. A good metric should indicate real problems; the authors suggest starting from the SLIs, since good SLIs tend to have a strong relationship with service health and may already be measured for the SLOs. The criterion for acceptable behavior needs to be calibrated: if it is too strict, false positives appear, that is, the canary is deemed bad without being bad; if it is too loose, the chance of a bad canary passing undetected increases. Choosing this criterion correctly can be expensive, takes time and requires analysis, and it should be reevaluated as the service evolves. Noisy metrics, such as CPU usage in the example, can lead operators to ignore or turn off the process. The recommendation is to rank metrics by how well they reflect user-perceived problems and to use few, perhaps a dozen at most. Metrics should also be attributable to the change and not to external factors, such as overloaded machines, which is why, for example, CPU time spent serving the request is preferred over the CPU of the whole machine. Comparing before and after on the same system is risky, because time itself changes the metrics (weekday versus weekend). Finally, metrics that do not have the ideal properties can still be useful if introduced through a gradual, staged canary: a small first stage, to minimize impact while there is no confidence in the release, with the clearest metrics, such as crashes and request failures, and larger later stages.
Release integration, rollback, and related concepts
The chapter shows that the canary only works if the evaluation is tied to the release process: if the canary's error rate drifts too far from the control's, the deploy is considered bad and the response is to pause and roll back, or to call a person to investigate; if the rates are similar, the rollout proceeds normally. This requires monitoring that separates canary from control: in the example, the 1% error in the aggregate may be indistinguishable from other sources of error, while the per-version breakdown clearly shows the candidate's errors. Care is also needed with aggregation windows: using an errors-per-hour metric to evaluate a 30-minute canary produces a confusing signal, and metric intervals must be equal to or shorter than the canary duration. Because, for practical reasons, canary and control may share backends and other dependencies, a bad result does not necessarily mean the canary is to blame, and a bad canary may affect the control; so it is also advisable to use absolute measures, such as SLOs. On related concepts, the text describes blue/green as two instances, where rollback is simply reverting the change in the router, at the cost of double the resources, and which amounts to a before/after canary; by splitting traffic between the two, it approaches an A/B canary. Artificial load and traffic teeing are similar to canarying and can be used as such, but they are not strictly canaries by the chapter's definition; they remain viable approaches for finding defects, with caveats.
See the ideas at work
Practice the designs that use this paper in the System Design school.