Designing Machine Learning Systems learning path: everything to study, topic by topic

Evaluating whether a problem is a good fit for ML: the need for patterns, data availability, scale, repetition, and tolerance for being wrong.

Translating business objectives into ML objectives: choosing classification vs regression, decoupling multiple objectives, and defining the right metrics.

Data sources, formats (row vs column, text vs binary), data models, OLTP vs OLAP, and the ETL/ELT processes that feed ML systems.

Why training data quality dominates model quality, data-centric thinking, and common pitfalls in assembling a first dataset.

Nonprobability vs random sampling, stratified, weighted, reservoir, and importance sampling, and the sampling biases each introduces.

Hand labels, natural labels, label multiplicity, feedback loop length, and dealing with the lack of and disagreement among labels.

Programmatic labeling with labeling functions, semi-supervised, transfer, and active learning as alternatives to expensive hand labels.

Resampling, two-phase learning, dynamic sampling, cost-sensitive learning, and choosing metrics that survive heavy class imbalance.

Common operations: handling missing values, scaling, discretization, encoding categorical features (including the hashing trick), and feature crossing.

Common causes of leakage: scaling before split, filling with statistics from the test set, group leakage, and leakage from data generation, plus detection methods.

Choosing among ML algorithms by tradeoffs, avoiding state-of-the-art bias, and balancing model complexity against data and latency budgets.

Establishing meaningful baselines (random, simple heuristic, human, existing solution) and ensembling via bagging, boosting, and stacking.

Evaluating models before deployment with perturbation tests, invariance tests, directional expectation tests, and model calibration against a baseline.

Model calibration for trustworthy probabilities and slice-based evaluation to catch hidden underperformance on critical subgroups (Simpson's paradox).

Batch (asynchronous) vs online (synchronous) prediction, the role of streaming features, and unifying batch and stream pipelines.

Tradeoffs between deploying on the cloud and on the edge: latency, connectivity, cost, privacy, and the hardware needed to run models on devices.

Reducing model size and latency through low-rank factorization, knowledge distillation, pruning, and quantization for resource-constrained inference.

Covariate, label, and concept drift, why models degrade in production, and how to detect and address each kind of shift.

Monitoring operational metrics, ML-specific metrics (accuracy, predictions, features, raw inputs), and building observability for ML systems in production.

Stateless vs stateful (incremental) retraining, retraining frequency, the four stages toward continual learning, and the challenges of keeping models fresh.

Safely evaluating models on live traffic with shadow deployment, A/B testing, canary release, interleaving experiments, and bandit-based selection.

Feature management, reuse, and consistency between training and serving, and how feature stores reduce train-serve skew across teams.

Components of an ML platform: model store and deployment, model serving, ML metadata, and the build-vs-buy decisions for infrastructure.

Sources of harm such as biased data and design, the framework for responsible AI, model transparency, and tradeoffs like privacy vs accuracy.