Real architecture · Mid
Netflix Metaflow
Real production architecture. Key points: ML workflow orchestration and versioning.
Short answer
Real production architecture. Key points: ML workflow orchestration and versioning. Netflix applies data science to hundreds of use cases. When the infrastructure team at machine learning asked the scientists what the hardest part of the job was, it expected to hear about data and models at scale.
01 · CONTEXT
Why a data-scientist-centered framework
Netflix applies data science to hundreds of use cases. When the infrastructure team at machine learning asked the scientists what the hardest part of the job was, it expected to hear about data and models at scale.
What it heard instead were stories of projects where getting the first version into production took surprisingly long, mostly for mundane software engineering reasons. It also heard about difficulties with data access and basic processing, and sat in meetings where scientists argued with stakeholders about how to version models without impacting production.
The team's conclusion was that almost everything the scientists wanted to do was already technically possible, but nothing was easy enough. The job became making common operations so easy that the scientist does not even notice they used to be hard.
Scientists want to control the model inputs and the feature engineering logic, and few have strong opinions about the data warehouse, the compute platform, or the workflow scheduler. They also want to write idiomatic Python.
02 · PROGRAMMING MODEL
A DAG of steps in plain Python
In Metaflow the scientist structures the data workflow as a directed acyclic graph of steps, and each step can be arbitrary Python code. In the post's hypothetical example, the flow trains two versions of a model in parallel and picks the one with the higher score.
The post acknowledges that Airflow and Luigi also run DAGs of arbitrary Python code, and says the difference lies in many carefully designed details. One of them: data and models are plain Python instance variables, which work even on a distributed platform thanks to the built-in content-addressed artifact store. In many other frameworks, loading and storing artifacts is left to the user.
The scientist writes the flow as Python code, free to use idiomatic Python.
The flow is a directed acyclic graph of steps, and each step is arbitrary code.
Data and models appear as plain Python instance variables and keep working on a distributed platform thanks to the content-addressed artifact store.
To inspect results, the scientist uses a notebook that fetches Metaflow artifacts, instead of a single interface for everyone.
03 · VERSIONING AND REPRODUCIBILITY
What the platform solves on its own
Metaflow comes with the ability to automatically snapshot all code and data to Amazon S3, which provides versioning and experiment tracking without user intervention. For the execution environment, the @conda decorator freezes it, with good reproducibility guarantees both locally and in the cloud.
| Concern | How the post describes the handling |
|---|---|
| State between steps | Plain Python instance variables, stored in the content-addressed artifact store. |
| Versioning and experiments | Automatic snapshot of all code and data to Amazon S3, without user intervention. |
| Dependencies | The @conda decorator freezes the execution environment. |
| Compute | Integration with AWS Batch, enabled with a single line: @batch. |
| Data loading | Built-in S3 client that loads data at up to 10Gbps. |
04 · FROM PROTOTYPE TO PRODUCTION
From laptop to scheduler
Metaflow lets you develop and test code quickly on a laptop, with a first-class local development experience, and Netflix encourages getting workflows to production as early as possible. Running early with fresh data is described as a very effective way to expose bugs and model problems.
The scientist develops and tests the code locally, on their own computer, with Metaflow's first-class local development experience.
For heavy processing, the flow uses AWS managed container compute, enabled with a single line of code.
Metaflow snapshots code and data to AWS object storage, which provides versioning without user intervention.
At Netflix, production is a centralized, highly available DAG scheduler, Meson, where the user exports the run with a single command.
Results are usually written to a table consumed by a dashboard, and sometimes the model becomes a microservice for real-time inference.
It is also common to chain workflows, so that the result of one is consumed by another. Since Meson is not open source, the team was working on an integration similar to AWS Step Functions, and some of these consumption modes were not yet available in the open-source release.
05 · SCALE AND LESSONS
Data scale and questions to practice
Netflix's data warehouse is hundreds of petabytes. A typical Metaflow workflow touches only a small fraction of it, but can still process terabytes.
Frequently asked questions
What is Netflix Metaflow?
Real production architecture. Key points: ML workflow orchestration and versioning.
What does the Netflix Metaflow design cover?
It is organized in these sections: Why a data-scientist-centered framework, A DAG of steps in plain Python, What the platform solves on its own, From laptop to scheduler, and Data scale and questions to practice.
How long is Netflix Metaflow and what level is it?
It takes about 15 minutes to read and is rated Mid level.
Now design it yourself
Run this case as an interview in the System Design school: you make the decisions, the tutor challenges them and you see how your design scores.