Netflix: 7 years to leave the data center (2008–2016)

01 · CONTEXT

What led Netflix to leave the data center

Netflix's journey to the cloud began in August 2008, when a severe database corruption left the company unable to ship DVDs to members for three days.

The conclusion was that it needed to abandon vertically scaled single points of failure, such as the data center's relational databases, and move toward distributed, highly reliable, horizontally scalable systems in the cloud.

1. In the earlier design, the member's device talks to a monolithic application, and the relational database sits in the data center.

2. In August 2008, a severe database corruption left the company three days without shipping DVDs to members.

3. The database, vertically scaled and single, is the single point of failure the company decided to leave behind.

02 · THE CHOICE

Take everything as is or rebuild

The easiest path would be the forklift: take all the systems out of the data center, unchanged, and put them on AWS. The price, in the company's own words, is carrying along all of the data center's problems and limitations.

AWS was chosen because it offered the largest scale and the broadest set of services and features.

Instead of the forklift, Netflix adopted the cloud-native approach: it virtually rebuilt all of its technology and fundamentally changed the way the company operates.

03 · ARCHITECTURE

The design after the migration

The cloud came to support business logic, distributed databases, big data processing and analytics, recommendations, transcoding and hundreds of other functions. Video, in turn, is delivered by Netflix Open Connect, the company's own globally distributed CDN. On the split into microservices, see the corresponding topic.

1. Device calls enter the cloud through an edge proxy, a role that a third-party survey attributes to Zuul.

2. The proxy forwards each call to the microservice that owns that function, such as recommendations or billing.

3. Services use distributed NoSQL databases, and the same survey cites EVCache and Dynomite for using Memcached and Redis at scale.

4. Video is delivered by Open Connect, Netflix's own CDN, distributed globally.

On failures, the company states that the cloud makes it possible to build highly reliable services from fundamentally unreliable but redundant components. The principles cited are redundancy and graceful degradation, plus regular exercises in production with the Simian Army, to survive failures in the cloud infrastructure and in its own systems without affecting the members' experience. See the topic on availability patterns.

The same third-party survey lists, among Netflix's open tools, Eureka for service discovery and Hystrix for isolating latency and fault tolerance at runtime.

04 · INCREMENTAL MIGRATION

Seven years, in parts

The migration advanced in parts. Most systems, including all customer-facing services, were already in the cloud before 2015. After that, the company spent time finding a safe and durable path for the billing infrastructure and for all aspects of customer and employee data management.

1. Before 2015, the customer-facing services were already in the cloud.

2. Billing and customer and employee data were left for later: the company needed a safe and durable path for them.

3. In early January 2016, the company declared the migration complete and shut down the last pieces of data center used by streaming.

The conclusion came in early January 2016, after seven years, with the shutdown of the last pieces of data center used by the streaming service.

05 · RESULTS AND COST

What the cloud delivered

The company reports having eight times more streaming members than in 2008, with total consumption growing by three orders of magnitude in eight years. The cloud's elasticity makes it possible to add thousands of virtual servers and petabytes of storage in minutes.

On January 6, 2016, the service was expanded to more than 130 new countries. Several AWS regions around the world make it possible to dynamically shift and expand infrastructure capacity.

As for availability, the company reports a steady increase, coming close to the four-nines uptime goal, despite difficult periods early in the migration.

Reducing cost was not the main reason for the change; even so, the cloud cost per streaming start ended up being a fraction of the data center cost. The company ties this to elasticity, which makes it possible to continuously optimize the instance mix and grow or shrink the infrastructure almost instantly, without large capacity buffers. See cost optimization at scale.