Code deployment system

01 · CONTEXT AND REQUIREMENTS

What a deploy system needs to guarantee

A deploy system takes an already built version to the machines that serve users. The design defines how each machine receives the change and how the system goes back if something goes wrong. It is a topic of availability patterns.

In Google's experience, most incidents are caused by binary or configuration pushes. If a release is deployed everywhere instantly, the defects are too.

The release engineering principles in the SRE workbook include deploys done by computers, not by people, and artifacts with small, self-contained changes. Smaller artifacts make rollback cheaper and easier.

02 · FROM COMMIT TO PACKAGE

The build pipeline and the package repository

According to Google's SRE book, builds are hermetic: they do not depend on the libraries and other software installed on the build machine. The expectation is that two people building the same revision on different machines get identical results.

1. The change enters the pipeline, which creates a release branch from a chosen integration revision, often taken from the continuous testing system.

2. The build compiles the binaries and runs the unit tests, often in parallel.

3. The result becomes a versioned package with a unique, signed hash, to guarantee authenticity.

4. Labels on the package indicate where it is in the release process, such as dev, canary, or production.

5. A report of all changes since the last release is also generated.

6. The rollout process can deploy a new binary to successive clusters over several hours.

In Google's SRE book, a rollout is a logical unit of work made up of one or more individual tasks. A typical canary deploy, again according to the book, starts a few jobs in production after the system tests.

03 · ROLLOUT STRATEGIES

Rolling, blue/green, and canary

Google's goal is to fit the deploy process to the service's risk profile. For large user-facing services, you can start with one cluster and expand exponentially until all are updated. For sensitive infrastructure, the rollout can stretch over several days, interleaving instances from different geographic regions.

1. The deploy service provisions instances for the replacement environment.

2. The latest application revision is installed on the replacement instances.

3. There is an optional wait time for activities such as application testing and system verification.

4. The replacement instances are registered with the load balancer, which redirects traffic to them; those in the original environment are deregistered and can be terminated or kept.

5. If something goes wrong, CodeDeploy rolls back by redeploying a previously deployed revision; technically, the rollback is a new deploy.

In Kubernetes, a Deployment in RollingUpdate mode gradually scales down the old ReplicaSets and scales up the new one, and maxUnavailable and maxSurge control the process. The controller halts a bad rollout on its own, which depends on maxUnavailable, whose default is 25%.

In ECS, CodeDeploy lets you shift traffic with a canary, linear, or all-at-once configuration. One of the predefined ones shifts 10 percent of traffic in the first increment and the remaining 90 percent five minutes later.

04 · CANARY IN PRACTICE

How to set up and evaluate a canary

A canary requires three capabilities: deploying the change to a subset of the service's population, evaluating whether the change is good or bad, and integrating that evaluation into the release process.

1. The orchestrator deploys the candidate to only a subset of the service's population.

2. The traffic splitter sends a small share of traffic to the candidate and compares it with the control; the split can use weights on a load balancer, proxy, or DNS round-robin.

3. In aggregate, the defect barely shows. The key metric (in the book's example, HTTP response codes) must be read per version.

4. In the example, the candidate serves the vast majority of the errors. With 20% errors on 5% of traffic, the overall error rate is 1%.

5. If the canary's error rate is too far from the control's, that signals a bad canary, and the response is to pause and roll back the deploy, or to call a person to investigate.

6. If the error rates are similar, the deploy continues normally, and the process gains confidence as it exposes the change to more traffic.

To choose what to measure, the workbook recommends starting from the SLIs. In the example service, the best metrics are probably HTTP codes and response latency. If the definition of acceptable behavior is too strict, there will be many false positives; if it is too loose, a bad canary is more likely to slip through undetected.

A representative canary requires decisions along many dimensions. Anyone who deploys continuously, say 20 times a day, needs a much shorter canary duration, and performance defects usually only show up under high load.

05 · ROLLBACK AND CONFIGURATION

Going back without creating a second problem

CodeDeploy rolls back by redeploying a previously deployed revision. Technically, rolled-back deploys are new deploys. A deploy or group can be configured to roll back automatically when the deploy fails or a monitoring threshold is reached, and then the last known good version is deployed.

In Kubernetes, 10 old ReplicaSets are kept by default. Once an old ReplicaSet is deleted, the ability to roll back to that revision is lost.

Feature flag frameworks let you separate feature launches from the binary release, and a binary's features can be enabled one by one through the experiment configuration. At Google, generating one package for the binary and another for the configuration lets you change each one independently.