Netflix Kayenta (automated canary)

01 · CONTEXT

Reducing the risk of a change in production

Kayenta is an automated canary analysis (ACA) platform, open-sourced by Netflix in partnership with Google. The launch post describes it as a crucial component of delivery at the company, because it reduces the risk of changes in the production environment.

Um canary is a new version of the software deployed to a small subset of users, alongside the stable version. Traffic is split so that part of the requests goes to the canary, which makes it possible to discover problems quickly without affecting most users.

Canary quality is evaluated by comparing key metrics of the old and new versions. If there is significant degradation, the canary is aborted and all traffic returns to the stable version.

1. Starting point: users reach the load balancer, which forwards requests to the production service.

2. The service publishes metrics, and continuous delivery deploys new versions. What is missing is deciding how a new version would be compared with the current one.

The rest of the case shows how Netflix answers that question: which clusters it creates, what it compares, and who decides.

02 · THE PROBLEM

When the canary was a manual analysis

Canary analysis started as a manual process. A developer or release engineer looked at graphs and logs from the baseline and canary servers to see how close the metrics were, such as HTTP status codes, response times, and exception counts. If the data looked reasonable, the person manually decided to move forward or roll back.

Netflix writes that the approach did not scale and was not reliable. Each canary meant several hours staring at graphs and digging through logs, which made it hard to deploy more than once or twice a week, and comparing graphs visually made it hard to see subtle differences between canary and baseline.

The first attempt at automation was a script specific to the application being measured. The company then tried to generalize the process and created the first version of automated canary analysis, of which Kayenta is an evolution.

03 · THREE CLUSTERS

Production, baseline, and canary

In Netflix's process, three clusters serve the same traffic in different amounts. A typical setup uses a load balancer to include the baseline and canary instances in the regular instance pool.

1. Users reach the load balancer, which splits the traffic among the clusters.

2. Production receives most of the traffic and runs the current version, unchanged.

3. The baseline runs the same code and configuration version as production and receives a small share of the traffic.

4. The canary runs the proposed changes and receives a small share of the traffic.

5. Spinnaker manages the lifecycle of the baseline and canary clusters.

6. Metrics are stored in a time-series database, with tags that identify whether the data came from the canary or the baseline.

It would also be possible to use the production cluster instead of creating a baseline, but comparing a freshly created canary with a long-running production could give unreliable results. Creating a fresh baseline guarantees metrics free of the effects of long-running processes.

Spinnaker, the company's continuous delivery platform, runs one or more iterations of the analysis and decides to continue, roll back, or, in some cases, ask for manual intervention. If the version is considered safe, the deploy proceeds; if not, Spinnaker aborts the canary and all traffic goes to production.

04 · KAYENTA

Metric retrieval and judgment

Kayenta is responsible for assessing the risk of a canary and checking whether there is significant degradation between baseline and canary. It is split into two main stages: metric retrieval and judgment.

1. The Spinnaker pipeline triggers the analysis through Kayenta's REST endpoint.

2. Kayenta reads the query configuration and combines it with a scope to query a metrics source.

3. The retrieval results are passed to the judge for analysis.

4. The judge produces a decision: the canary passed or failed.

5. Spinnaker uses the score to decide whether the canary continues or rolls back.

6. The output of retrieval and judgment is archived, which makes it possible to run new judges over data already collected.

During retrieval, Kayenta reads a configuration file that defines the metric queries and combines each one with a scope, that is, a cluster and a time range. Metrics are typically stored in a time-series database, with tags that identify whether the data came from the canary or the baseline. Among the sources supported at the time of the post were Prometheus, Stackdriver, Datadog, and Netflix's Atlas, and different sources can be combined in a single analysis.

The main comparison algorithm uses confidence intervals, computed with the Mann-Whitney U test, to classify whether a significant difference exists. The High classification indicates that the canary's metric is significantly higher than the baseline's.

The final score is the ratio of metrics classified as Pass to the total number of metrics: for example, 9 of 10 metrics at Pass result in a score of 90%. Netflix says more complex methodologies exist, but that it prefers techniques that are simple to understand.

The canary report details the result per metric and shows the input data. In one example from the post, the score is 58%, with several metrics classified as High. For Netflix, having a detailed view of why a canary failed is crucial to building confidence in the system.

05 · RESULTS

Flexibility, adoption, and what to take into the interview

Because the output of retrieval and judgment is archived, it is possible to run new comparison algorithms and new judges over data already collected. Metrics sources, judges, configuration storage, and results storage are pluggable.

Kayenta offers a REST endpoint for CRUD operations on configurations and for fetching results. Despite its tight integration with Spinnaker, it can run without the other Spinnaker components, with Redis as its only dependency.

According to Netflix, Kayenta is much more flexible than the previous solution and easier to configure. The company removed much of the complexity of defining thresholds and manual adjustments, relying on better algorithms. At the time of the post, migration from the legacy system was in progress, and Kayenta was running about 30% of canary judgments in production, an average of 200 per day.