Papers
Hidden Technical Debt in Machine Learning Systems
D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-François Crespo, Dan Dennison · NeurIPS 2015, 2015
The problem: building fast is cheap, maintaining is expensive
The authors, all at Google, start from an uncomfortable observation: developing and deploying an ML system is relatively fast and cheap, but maintaining it over time is difficult and expensive. To explain this dichotomy they use the technical debt metaphor, introduced by Ward Cunningham in 1992. As with financial debt, there are good strategic reasons to take it on, not all debt is bad, but all debt has to be paid. Paying it means refactoring code, improving unit tests, deleting dead code, reducing dependencies, tightening APIs and improving documentation. The goal of this payment is not to add functionality, but to enable future improvements and reduce errors. Postponing payment makes costs pile up, and hidden debt is dangerous precisely because it grows silently. The paper's central argument is that ML systems have a special capacity for generating debt: they inherit all the maintenance problems of traditional software and add problems of their own. This debt is hard to see because it lives at the system level, not the code level, since data influences behavior and corrupts the usual abstractions and boundaries. The paper does not propose new algorithms; it aims to raise awareness of long-term trade-offs. The abstract lists the risk factors: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world and system-level anti-patterns.
Complex models erode boundaries: entanglement and CACE
Classical software engineering relies on strong abstractions, with encapsulation and modularity, so that isolated changes are easy. The paper argues that this is hard to enforce in ML, because you use ML precisely when the desired behavior cannot be expressed in software logic without depending on external data. The real world does not fit into tidy encapsulations. The first effect is entanglement: signals are mixed inside the model, which prevents isolating improvements. If the model uses features x1 through xn and the distribution of x1 changes, the importance and use of the other n-1 features can change, whether the model is trained in batch or adapting online. Adding or removing a feature has a similar effect. No input is truly independent. The authors call this the CACE principle, 'Changing Anything Changes Everything', which also holds for hyperparameters, learning settings, sampling methods, convergence thresholds, and data selection. Isolating models and serving ensembles helps in some cases, but ensembles work well because the components' errors are weakly correlated, which creates strong entanglement: improving one component can make the whole worse. Another strategy is to detect changes in prediction behavior, for example with high-dimensional visualization or per-slice metrics.
Correction cascades and undeclared consumers
Two forms of boundary erosion get names of their own. The first is the correction cascade. There is a model m_a for problem A, and a need arises to solve a slightly different problem A'. The temptation is to train a model m'_a that takes m_a as input and learns a small correction, which is quick. The cost is a new dependency on m_a, which makes it more expensive to analyze future improvements. If corrections are stacked, with a model for A'' on top of m'_a and so on, the result can be an improvement deadlock: improving the accuracy of an isolated component causes harm to the system. The suggested mitigations are to augment m_a to learn the corrections within the same model, adding features that distinguish the cases, or to accept the cost of a separate model for A'. The second form is undeclared consumers. When the prediction of m_a becomes widely accessible, at runtime or through files and logs, other systems may start using it without that being recorded. In classic software this is called visibility debt. It creates tight, hidden coupling, so changing m_a, even to improve it, tends to cause impacts that are not understood. Undeclared consumers can also create hidden feedback loops. To avoid them, the system must be designed with access restrictions or strict SLAs, because under deadline pressure engineers use whichever signal is most convenient.
Data dependencies cost more than code dependencies
Code dependencies can be found by static analysis in compilers and linkers, but the authors argue that data dependencies have a similar capacity to generate debt and are harder to detect. Without equivalent tooling, it is easy to build long chains of data dependencies that are hard to untangle. There are two central kinds. Unstable dependencies: consuming signals produced by other systems is convenient, but some signals change behavior over time, either because they come from another model that updates itself, or because the team that owns the signal is a different one. A silent update that fixes a previously miscalibrated signal can hurt the consuming model, which adapted to the wrong calibration. The mitigation is to create a versioned copy of the signal, at the cost of staleness and of maintaining several versions. Underutilized dependencies: signals that give little incremental gain but make the system vulnerable to changes. The text lists how they creep in: legacy features, bundled features (all added together because of deadline pressure), epsilon-features (tiny accuracy gain with large complexity), and correlated features, where the model may credit the non-causal one and become fragile when the correlations change. They can be detected by regular leave-one-feature-out evaluations. For the whole set, the authors propose static analysis of data dependencies, such as a feature management system with annotations and automated checks.
Feedback loops and system anti-patterns
Production ML systems often influence their own behavior when they are updated, which creates analysis debt: it is hard to predict a model's behavior before launch. In direct loops, the model influences the selection of its own future training data. Bandits would be the theoretical solution, but they scale poorly to the action spaces of real problems; this is mitigated with some randomization or by isolating parts of the data from the model's influence. Hidden loops are harder: two systems influence each other indirectly through the world, such as one that chooses products and another that chooses the reviews shown on the same page, where changing one changes the clicks on the other. This can happen even between completely disjoint systems, such as the stock-prediction models of two investment firms. Next, the paper observes that only a tiny fraction of the code in many ML systems does learning or prediction (Figure 1 shows the ML code as a small box surrounded by vast infrastructure). The anti-patterns include glue code (a lot of code to fit generic packages, in a mature system perhaps at most 5% ML and at least 95% glue), pipeline jungles (jungles of scrapes, joins, and samplings in data preparation), and dead experimental codepaths, with the example of Knight Capital, which lost 465 million dollars in 45 minutes, apparently because of unexpected behavior from obsolete experimental code. The text also cites abstraction debt and smells such as plain-old-data types, multiple languages, and prototypes.
Configuration, the outside world, and how to measure debt
The authors point to configuration as another surprising source of debt. In mature, active systems, lines of configuration can outnumber lines of code, and each line can contain errors. They give examples such as a feature logged incorrectly during a certain period or code that changes depending on the log format. The principles of a good configuration system include: expressing it as a small change from the previous one, making manual errors hard, making it possible to see the difference between two models, asserting and verifying facts automatically, detecting redundant settings, and subjecting it to code review and a repository. As for the external world, it is rarely stable. Manually fixed decision thresholds can become invalid when the model is updated; one alternative is to learn thresholds from validation data. Unit and end-to-end tests are not enough, so real-time monitoring and automated response are needed. The paper suggests monitoring prediction bias (the distribution of predicted labels should look like that of observed ones), action limits, and upstream producers. Other areas include data testing debt, reproducibility debt, process management debt, and cultural debt. In the conclusion, the authors say that debt has no strict metric and that staying fast does not prove good practices. They propose questions, such as how easy it is to test a new approach at scale, the transitive closure of data dependencies, and whether improving one model degrades others, and they advocate a culture that rewards removing features and reducing complexity.
See the ideas at work
Practice the designs that use this paper in the System Design school.