Uber Michelangelo (ML platform → GenAI)

01 · CONTEXT

Before the platform

This study combines two posts from Uber's engineering blog: one published on September 5, 2017, and another that traces eight years of Michelangelo's trajectory.

Michelangelo is, in the authors' words, an internal machine learning as a service platform, designed to cover the end-to-end workflow: managing data, training, evaluating, and deploying models, making predictions, and monitoring them.

Before it, data scientists used a wide variety of tools to create models, while separate engineering teams built bespoke systems to use them in production. There was no established path to deploy a model: in most cases, the engineering team had to create a serving container specific to the project. It was also not possible to train models larger than what fit on data scientists' desktop machines, nor was there a standard place to store experiment results or compare them.

According to the first post, construction began in mid-2015, focused on scalable training and on getting models into serving containers; then came the systems to manage and share feature pipelines and, more recently, developer productivity. The second post says Michelangelo launched in early 2016.

02 · THE LIFECYCLE

Six stages, one platform

The authors organize the work into six stages: manage data, train models, evaluate, deploy, make predictions, and monitor them. According to the post, the same workflow exists in almost all of Uber's ML use cases, from classification and regression to time series forecasting.

The platform is built on the company's data and compute infrastructure: a data lake with transactional and log data, Kafka brokers, a Samza streaming engine, and managed Cassandra clusters. The main open-source components cited are HDFS, Spark, Samza, Cassandra, MLLib, XGBoost, and TensorFlow.

The example that guides the post is UberEATS estimated delivery time: data scientists use gradient-boosted decision tree regression models, deployed in Michelangelo serving containers and called by UberEATS microservices through network requests.

1. Transactional and log data arrive in an HDFS data lake, accessible to Spark and Hive jobs. Regular jobs compute features, which can be private to a project or published to the Feature Store.

2. In the batch precompute path, historical features are loaded from HDFS into Cassandra, from where they can be read with low latency at prediction time.

3. In the near-real-time path, metrics go to Kafka and Samza jobs generate aggregated features with low latency; they are written straight to Cassandra and logged back to HDFS for future training.

4. For training, the system joins the right datasets from HDFS; jobs use datasets from the Feature Store and the training data repository and send the model to the model repository. At the end of training, the original configuration, the learned parameters, and the evaluation report are saved.

5. The model artifacts are packaged in a ZIP file and copied to the hosts; the serving containers load the new model from disk and start serving predictions.

6. In online prediction, the container passes the raw features through the DSL expressions, fetches additional features from the Feature Store, with values read from Cassandra, and returns the prediction to the client service over the network.

03 · FEATURES

Feature store and training-serving consistency

Models deployed online do not access data stored in HDFS. That is why the features they need are precomputed and stored in Cassandra, from where they can be read with low latency at prediction time. There are two ways to compute them:

The second piece of consistency is the feature selection and transformation DSL, a subset of Scala. The expressions are part of the model configuration and are applied at training and at prediction, which helps ensure that the same final set of features is generated and sent to the model in both cases. The second post adds that packaging the transformation with the model as a Spark PipelineModel eliminates one source of training-serving skew.

On top of these pipelines sits the feature store: to enter it, a feature needs a few extra metadata fields (owner, description, SLA, and others) and is then consumed online and offline by its canonical name in the model configuration. The system joins the right datasets from HDFS at training and fetches the right value from Cassandra at online prediction.

The first post mentions about 10 thousand features in the Feature Store. The second names Palette the feature store built to manage and share feature pipelines across teams, with support for batch and near-real-time computation, and says it hosts more than 20 thousand features.

04 · REGISTRY AND DEPLOY

Model registry and deployment

Each trained model becomes a versioned object in the model repository, stored in Cassandra. The record includes who trained the model, the full configuration (features used, hyperparameter values, and others), the reference to the training and test sets, and the accuracy metrics; the web interface and the API serve both to inspect a model and to compare it with others.

In the second post, Gallery appears as the registry of models and ML metadata, with a search API for all types of ML entities.

Michelangelo offers three deployment modes:

In all cases, the model artifacts (metadata files, parameter files, and compiled DSL expressions) are packaged in a ZIP file and copied to the relevant hosts in the data centers; the prediction containers load the new models from disk automatically.

More than one model can be deployed at the same time in the same container, which allows safe transitions from the old model to the new one and side-by-side A/B tests. At serving time, the model is identified by its UUID and an optional tag set at deploy. If the two models have the same signature (they expect the same set of features), the new one can be deployed under the same tag as the old one: the container starts using it immediately, without requiring any change to the client code.

05 · PREDICTION AND MONITORING

Serve, scale, and watch

At prediction time, the raw features pass through the compiled DSL expressions, which can modify them and fetch others from the Feature Store. Online models return the prediction to the client service over the network; offline models write the predictions to Hive, where batch jobs or users with SQL tools can consume them.

Because the models are stateless and share no state, scaling is simple: in online mode, just add hosts to the cluster and let the load balancer distribute the load; in offline mode, add Spark executors. The 2017 post reports typical P95 latency under 5 ms for models that do not need Cassandra and under 10 ms for those that do, and higher-traffic models serving more than 250 thousand predictions per second.

Training and evaluation always use historical data. That is why monitoring predictions is critical: to ensure the pipelines keep sending correct data and that the production environment has not changed to the point where the model is no longer accurate. Michelangelo can log and, optionally, retain a percentage of predictions and later join them with observed outcomes; for regression, it publishes R-squared, RMSLE, RMSE, and mean absolute error to Uber's time series systems, where users analyze charts and set alerts.

On top sits the API layer: a management application that serves the web interface and the network API and hosts the workflow system, which orchestrates batch data pipelines, training jobs, batch prediction jobs, and the deployment of models to batch and online containers.

For comparison with the 2017 post: the second post reports about 400 active ML projects, more than 20 thousand training jobs per month, more than 5 thousand models in production, and 10 million real-time predictions per second at peak.

06 · MICHELANGELO 2.0

Planes, quality, and developer experience

According to the second post, by the end of 2019 most of Uber's lines of business had already integrated ML, and Michelangelo's focus began shifting from enabling ML everywhere to doubling down on high-impact ML projects.

The post lists limitations of Michelangelo 1.0 for those projects: lack of a comprehensive definition of ML quality and of project tiering; insufficient support for deep learning models; inadequate support for collaborative model development; and fragmented tooling and developer experience.

Michelangelo 2.0 organizes the system into three planes. The control plane defines the user-facing APIs and manages the lifecycle of entities; the offline data plane does the heavy processing, such as feature computation, model training and evaluation, and batch inference; the online data plane handles real-time inference and feature serving, used by other microservices.

1. The control plane defines the user-facing APIs and manages the lifecycle of all entities; the APIs follow the conventions of the Kubernetes API and standardize operations on Project, Pipeline, Model, Deployment, and other entities.

2. The offline data plane does the heavy lifting, such as feature computation, model training and evaluation, and batch inference; its pipelines are defined as DAGs of steps.

3. Gallery is the registry of models and ML metadata, with a search API for all types of ML entities.

4. The online data plane handles real-time inference and feature serving, used by other microservices; it manages RPC services and streaming jobs for online prediction, feature access, and near-real-time computation.

5. The Model Excellence Score (MES) measures and monitors quality dimensions and metrics at each stage of the model lifecycle, using the SLA concept common among SREs.

To measure quality, the post describes the Model Excellence Score (MES), a framework for measuring and monitoring dimensions and metrics at each stage of the model lifecycle, based on the SLA concept used by SREs. To prioritize, there is a tiering scheme with four levels, with tier 1 the highest; tier-1 projects bring together models that serve critical functions of the core ride and order flows, such as ETA calculation, safety, and fraud detection.

In 2020, Project Canvas was launched, which applies software engineering best practices to ML development; one of its components is the ML Monorepo, a central repository that stores as code the sources of truth of ML development, with version control. MA Studio unifies existing and new offerings into a single workflow and includes a modern deployment stack, with safe, incremental zonal rollout, automatic rollback triggers, and production runtime validation.

For deep learning, feature transformation, which in 1.0 was packaged with the model and eliminated one source of training-serving skew, gained a native solution in 2.0 with Keras or PyTorch operators. The Spark-based XGBoost and DL trainers were replaced by Ray-based trainers, serving came to integrate Triton into the Online Prediction Service, and DL adoption in tier-1 projects went from nearly zero to more than 60%.

07 · FROM PREDICTIVE TO GENAI

Gateway, LLMOps, and lessons

The second post divides the trajectory into three phases: the first, from 2016 to 2019, was predictive ML for tabular data; the second, from 2019 to 2023, pushed DL adoption and collaborative model development in high-impact projects; the third begins in 2023, with the new wave of generative AI.

For generative AI, Uber created the Gen AI Gateway, a unified interface for accessing external and internally hosted LLMs. Among its capabilities are logging and auditing, cost guardrails and attribution, safety and policy guardrails, and PII redaction before sending input to external LLMs. The two types of model coexist because, according to the post, external ones have superior performance in general knowledge and complex reasoning, while open models fine-tuned with proprietary data serve specific Uber tasks at a fraction of the cost and with lower latency.

Michelangelo was extended to cover LLMOps capabilities: fine-tuning data preparation, prompt engineering, fine-tuning and evaluation, deployment and serving, and performance monitoring in production. The Model Catalog brings together ready-to-use LLMs, and a Ray-based trainer for LLMs uses open models from the Hugging Face Hub; the fine-tuned LLMs and their metadata live in the model repository, accessible to the inference infrastructure.

1. Applications call the Gen AI Gateway, which offers a unified interface for accessing external LLMs and internally hosted LLMs, respecting security standards and protecting privacy.

2. The gateway brings together logging and auditing, cost guardrails and attribution, safety and policy guardrails, and PII redaction before sending input to external LLMs.

3. Both destinations coexist: according to the post, external models have superior performance in general knowledge and complex reasoning, while open models fine-tuned with proprietary data serve specific Uber tasks at a fraction of the cost and with lower latency.

4. Fine-tuning runs on a Ray-based trainer over open LLMs from Hugging Face; the fine-tuned LLMs and their metadata live in the model repository, accessible to the inference infrastructure.