Hidden Technical Debt in ML Systems (Google)

01 · CONTEXT

Cheap to build, expensive to maintain

The paper starts from an observation: developing and putting ML systems into production is relatively fast and cheap, but maintaining them over time is difficult and expensive. Through the lens of technical debt, not all debt is bad, but all debt must be paid, and hidden debt is dangerous because it accumulates silently.

The central thesis is that ML systems have a special capacity to accumulate debt, because they add the maintenance problems of traditional code to a set of problems specific to ML. This debt can be hard to detect because it exists at the system level, not the code level, and the typical methods for paying down code debt are not enough for it.

A short version of the paper was presented at the SE4ML workshop in Montreal in 2014. The summary on the Google research page, in turn, says it is remarkably easy to accumulate massive, ongoing maintenance costs at the system level when applying machine learning.

02 · ERODED BOUNDARIES

Why models erode the separation between components

It is hard to enforce strict abstraction boundaries in ML systems, because ML is needed precisely when the desired behavior cannot be expressed in software logic without depending on external data. For this reason, traditional abstractions and boundaries can be subtly corrupted by the fact that data influences the system's behavior.

First comes entanglement. ML systems mix signals, and no input is truly independent; the paper calls this the CACE principle, for Changing Anything Changes Everything. Changing the input distribution of one feature can alter the importance, weights, or use of the others.

Isolating models and serving ensembles is one possible mitigation. But ensembles often work because the errors of the components are uncorrelated, and depending on the combination creates strong entanglement: improving one component can worsen the accuracy of the system.

Next come correction cascades. It can be tempting to train a model that learns a small correction on top of another, as a quick way to solve a slightly different problem. In a cascade, this can create an improvement deadlock, in which improving the accuracy of one component leads to losses at the system level. The mitigations cited are to learn the corrections inside the same model, with features that distinguish the cases, or to accept the cost of a separate model.

Finally, undeclared consumers. Without access control, some consumers of the prediction can become undeclared and silently use the model's output as input to another system. In classical software engineering, these problems are called visibility debt.

1. A system produces a signal that the model uses as input.

2. The model delivers predictions to a consumer that the model team knows about.

3. The prediction is also saved in files or logs, and another system starts reading that output as input without the model team knowing.

4. When the model changes, the other parts of the stack are probably affected, possibly in unintended and harmful ways.

5. The paper cites access restrictions and strict service-level agreements as a way to design the system to avoid undeclared consumers.

03 · DATA AND ANTI-PATTERNS

Data dependencies and system anti-patterns

Code dependencies can be identified by static analysis from compilers and linkers. Without equivalent tools for data, it is inappropriately easy to create large chains of data dependencies that are hard to undo. Static analysis tools for data dependencies are less common, but essential for checking errors, locating consumers, and enforcing migrations and updates.

Among the risks are unstable signals, which change behavior over time. This can happen implicitly, when the signal comes from another model that updates itself, or explicitly, when engineering ownership of the signal is separate from that of the model that consumes it. A common mitigation is a versioned copy of the signal, which has its own cost: possible staleness and maintaining multiple versions of the same signal.

There are also underutilized dependencies, signals that bring little incremental modeling benefit. They leave the system unnecessarily vulnerable to changes and could be removed without loss. One case is a feature included early in development and made redundant by new features without this being detected. The paper proposes exhaustive leave-one-feature-out evaluations, run regularly.

At the system level, the paper shows that only a small fraction of a real ML system is ML code, and that the infrastructure needed around it is vast and complex.

1. The learning code is a small block of the diagram; around it sits a vast and complex infrastructure.

2. In data preparation, new signals and new sources come in gradually; without care, the result becomes a jungle of scrapes, joins, and sampling steps, often with intermediate files.

3. Generic learning packages call for a lot of supporting code to get data in and out of them, and this code tends to lock the system into the quirks of a specific package.

4. Configuration also accumulates debt: in a mature, actively developed system, the configuration lines can far outnumber the lines of traditional code, and each line has the potential for errors.

5. Tests are not enough in the face of a changing world; the paper advocates live monitoring combined with automated response.

Of the anti-patterns, the first is glue code. Generic packages often require an enormous amount of supporting code. A mature system may have at most 5% ML code and at least 95% glue code, and it may be cheaper to create a clean native solution than to reuse a generic package. An important strategy is to wrap black-box packages in common APIs.

The second is the pipeline jungle. Testing it usually requires expensive end-to-end integration tests, and it can only be avoided by thinking holistically about data collection and feature extraction.

The third is dead experimental codepaths. In the short term, it becomes attractive to try alternative methods as conditional branches inside production code. Over time, this creates growing debt through the difficulty of maintaining backward compatibility and the exponential increase in cyclomatic complexity. A famous example cited: Knight Capital's system lost 465 million dollars in 45 minutes, apparently due to unexpected behavior of obsolete experimental codepaths.

04 · FEEDBACK AND THE EXTERNAL WORLD

Feedback loops and a changing world

Live ML systems often end up influencing their own behavior if they update over time. This leads to a form of analysis debt: it is hard to predict the behavior of a model before it is launched. Loops are harder to detect and handle if they occur gradually, as when models are updated infrequently.

1. Users receive the current model's predictions.

2. User actions come back as training data, and the retrained model returns to service: the loop closes in the diagram itself.

3. It is a direct feedback loop: the model directly influences the selection of its own future training data.

4. The paper cites as a mitigation using some randomization or isolating parts of the data from the model's influence; bandit algorithms do not necessarily scale to the action spaces of real problems.

5. The hidden loop, in contrast, does not appear in the diagram: two systems influence each other indirectly through the world, such as one that chooses products and another that chooses related reviews on the same page.

The external world also changes. Experience shows it is rarely stable, and this creates ongoing maintenance cost. Manually set decision thresholds can become invalid when the model updates with new data, and manually updating many thresholds across many models is slow and brittle. One mitigation is to learn thresholds by simple evaluation on separate validation data.

Unit and end-to-end tests are valuable, but in the face of a changing world they are not enough as evidence that the system works as intended. Relying on human intervention in response to alertas is one strategy, but it can be brittle for time-sensitive problems.

05 · ML RULES

The Rules of ML guide: train-serving skew

The Rules of Machine Learning guide presents Google's ML practices in a style similar to programming style guides. Two of its rules deal with infrastructure: the first model does not need to be sophisticated, but far more infrastructure problems will arise than expected; and the infrastructure must be testable, with the learning parts encapsulated so that everything around them can be tested.

A concrete risk is training-serving skew, a difference between performance during training and during serving. It can come from different handling of the data in the training and serving pipelines, from a change in the data between training and serving, or from a feedback loop between model and algorithm.

1. The guide recommends saving the set of features used in serving and sending it to a log for use in training.

2. The training pipeline consumes the features logged by the service, to train the way it serves.

3. The trained model is published to the repository and loaded by the service, closing the loop between training and serving.

4. The best solution, according to the guide, is to monitor the skew explicitly, so that changes in the system and the data do not introduce it unnoticed.

Two other rules reinforce the care: reuse code between the training pipeline and the serving pipeline whenever possible, and remember that, if you join data from a table at training and at serving, the table data may change.

The guide also asks that you know the system's freshness requirements: knowing how much performance degrades with a one-day, one-week, or one-quarter-old model helps prioritize monitoring. And it recommends sanity checks right before exporting the model.

06 · LESSONS

How to measure and pay down the debt

The paper acknowledges that the technical debt metaphor does not provide a strict metric that can be tracked over time. Noticing that the team still moves fast is not, by itself, evidence of low debt, since the total cost only shows up over time.

Instead of a metric, the paper proposes questions. How easily can an entirely new algorithmic approach be tested at full scale? What is the transitive closure of all data dependencies? Does improving one model or signal degrade others?

Paying down the debt requires changing the culture: it is important to create team cultures that reward feature removal, complexity reduction, and improvements in reproducibility, stability, and monitoring to the same degree as accuracy improvements. Research solutions that give a tiny accuracy benefit at the cost of huge increases in system complexity are rarely wise practice. Configurations, in turn, should go through full code review and be versioned in a repository.