Concepts
Technical debt in ML systems: entanglement and feedback loops
Intuition
Imagine a recipe in which all the ingredients have already been mixed in the same pot: if you change only the salt, the taste of everything changes, and you cannot remove one ingredient without altering the rest. Now imagine that the dish you serve changes what customers order tomorrow, and that other cooks, without telling you, have started copying your dish. Building this dish was quick and cheap, but changing it safely later gets more and more expensive, like a debt that grows silently while nobody is looking.
How it works
Sculley et al. use the technical debt metaphor: putting an ML system into production is fast and cheap, but maintaining it is hard and expensive, because it inherits the problems of traditional software and adds its own, which live at the system level rather than the code level. The first is entanglement: signals are mixed together inside the model, which prevents isolating improvements. If the model uses features x1 through xn and the distribution of x1 changes, the importance and usage of the other n-1 may change, whether the model is trained in batch or adapting online. The authors call this the CACE principle (Changing Anything Changes Everything), which also holds for hyperparameters, learning settings, sampling methods, convergence thresholds and data selection. Boundary erosion gets worse with undeclared consumers, which start using a model's prediction (at runtime, in files or in logs) without being registered, creating tight, hidden coupling: changing the model, even to improve it, causes impacts nobody understands. There are also feedback loops: in a direct loop the model influences the selection of its own future training data; in a hidden loop, two systems influence each other indirectly through the world (one picks products and another picks reviews displayed on the same page, and changing one alters the clicks on the other). This creates analysis debt, since it is hard to predict the model's behavior before launch. Finally, Figure 1 of the paper shows ML code as a small box surrounded by vast infrastructure, and the anti-patterns that fill it include glue code (in a mature system, perhaps at most 5% ML and at least 95% glue), pipeline jungles in data preparation and dead experimental code paths, as in the case of Knight Capital, which lost 465 million dollars in 45 minutes, apparently because of unexpected behavior from obsolete experimental code.
In practice
The paper does not propose a new algorithm, but rather mitigations and diagnostic questions: isolating models or serving ensembles helps in part, but since ensembles work because the errors of the components are weakly correlated, they themselves create strong entanglement. For undeclared consumers, the system should be designed with access restrictions or strict SLAs; for direct loops, some randomization is used or parts of the data are isolated from the model's influence. The authors also suggest detecting changes in prediction behavior (for example, with per-slice metrics), deleting dead code, and rewarding a culture of removing features and reducing complexity. Because debt has no strict metric and continuing to move fast does not prove good practices, they offer questions, such as how easy it is to test a new approach at scale and whether improving one model degrades others, which are worth asking when deciding when to stop piling on features and pay down the debt.
Use it in a design
Practice the cases that rely on this concept in the System Design school.