Designing Machine Learning Systems
About this book
Huyen treats machine learning as a production systems problem, covering everything around the model rather than the modelling itself. The book moves from framing a business problem as a machine learning problem and deciding whether the approach is appropriate at all, through data engineering, feature engineering and training data sampling, to model development, offline and online evaluation, and deployment. Later chapters cover data distribution shift, monitoring in production, continual learning, and the infrastructure choices sitting behind all of it. A closing chapter deals with the human side, including team structure and responsible development. Practical and specific about the failure modes that only appear once real traffic arrives.
Description via Product Digest.
Topics
Related concepts
- Supervised, unsupervised and reinforcement learningThe three ways a model learns, what each one needs from the data, and how to tell which problem is actually in hand.
- LLM Evaluation Datasets: Sampling, Labels and Test-Set LeakageBuild representative LLM evaluation datasets from real and designed cases, prevent leakage and interpret small sample results with appropriate uncertainty.
- Customising a foundation modelThe five ways to close the gap between a general model and a specific problem, ordered by cost, and the question that decides which one a problem needs.
- Robustness and distribution shiftWhy every accuracy figure belongs to the data it was measured on, and what happens to a deployed model as the world drifts away from that data.
- The machine learning lifecycleThe stages between a business question and a model in production, why the work does not stop at deployment, and where each cloud vendor's tooling sits along the line.
- Goal misgeneralisationThe failure where a system learns a goal that fits all of its training data and turns out to be the wrong goal once the distribution shifts.
- How to Build LLM Evaluations: Cases, Graders and Release CriteriaBuild an LLM evaluation around real task requirements, with test cases, deterministic checks, human rubrics and explicit release criteria.
- The foundation model lifecycleThe seven stages that carry a foundation model from raw data to a system in use, and which five of them a team building on somebody else's model actually owns.
- LLM Model Migration: Evaluation, Compatibility and RolloutPlan an LLM model migration with API compatibility checks, controlled evaluations, prompt adaptation, canary rollout and a viable fallback.