Concept 2 of 4

The machine learning lifecycle

2 questions test this

A model is a small part of a machine learning system. The lifecycle describes everything around it, and most of the effort and most of the failures live in the stages before and after training.

Before the model

Data collection gathers what you will learn from, and decides the ceiling on everything that follows. A model cannot know something absent from its training data.

Exploratory data analysis is the look before the work. Distributions, missing values, obvious errors, and whether the data actually contains the signal you assumed. This is where most projects should stop and many do not.

Preprocessing cleans and shapes. Handling missing values, removing duplicates, normalising scales, encoding categories as numbers, and splitting into training, validation and test sets.

Feature engineering turns raw fields into things a model can use. A timestamp becomes day of week and hour. Two columns become a ratio. Domain knowledge enters the system here, and on classical machine learning problems it usually beats a cleverer algorithm.

Building the model

Training runs the algorithm over the training set to produce a model.

Hyperparameter tuning adjusts the settings you chose rather than the weights the model learned. Learning rate, network depth, number of trees. Tuning means training several times and comparing, which is why it costs what it does.

Evaluation measures performance on data the model has not seen. Judging a model on its training data tells you only how well it memorised.

Getting it to users

Deployment puts the model where requests can reach it. The syllabus names two shapes. A managed API service hosts the model for you and charges per call or per hour. Self hosting puts it on infrastructure you run, which costs more attention and gives more control over latency, data residency and cost at scale.

The other choice is when inference happens. Real time inference answers a request while somebody waits, and its constraint is latency. Batch inference scores a large set on a schedule, and its constraint is throughput. A fraud check at checkout is real time. Overnight churn scores for a mailing are batch, and paying for a low latency endpoint to produce them is waste.

After deployment

Monitoring watches the deployed model, and it watches two different things. Operational health covers latency, errors and cost. Model health covers accuracy against outcomes as they arrive, and drift, which is the world moving away from the data the model learned on. Drift is silent. Nothing breaks, the predictions simply get worse.

Practise this

You need one feature of your own that already runs on a model, and the discipline to write a person's name rather than a team's in every row.

Fill in the grid for that feature. An owner is one named person who would be called if the row went wrong, so a squad, a rota or "the platform team" counts as blank. Add the date the row was last done and the date it is next due, and where no date exists, leave it empty rather than inventing one.

Feature ..........................................

Stage                      Named owner   Last done   Next due
Data collection            ___________   _________   ________
Exploratory analysis       ___________   _________   ________
Preprocessing              ___________   _________   ________
Feature engineering        ___________   _________   ________
Training                   ___________   _________   ________
Hyperparameter tuning      ___________   _________   ________
Evaluation                 ___________   _________   ________
Deployment                 ___________   _________   ________
Monitoring, operational    ___________   _________   ________
Monitoring, model health   ___________   _________   ________
Retraining trigger         ___________   _________   ________
Retirement                 ___________   _________   ________

Read the blanks from the bottom up. Nearly every team fills the rows down to deployment and thins out below it, and the ones reliably empty are model health monitoring, the retraining trigger and retirement, which is to say nobody is watching the model get worse, nobody has agreed what triggers a retrain, and nobody has thought about switching it off. A missing next due date is worth as much as a missing name, because a row owned in principle and never scheduled behaves exactly like an unowned one.

The grid turns a diagram everybody nods at into a short list of specific gaps with your own names against them.

Where AWS puts the stages

Amazon SageMaker covers the pipeline end to end, and the AWS exam expects a few components by name. SageMaker Data Wrangler for preparation, SageMaker Feature Store for reusable features, SageMaker Model Monitor for drift and quality after deployment, SageMaker Clarify for bias and explanations, and SageMaker JumpStart for pre trained models you can start from.

The instinct worth having is to look for an existing model first. Sources of models run from a managed service that needs no training, through an open source pre trained model you adapt, to a custom model you train from scratch, and the cost rises steeply along that line.

Where Google Cloud puts the stages

The Google exam uses a shorter list of five stages and asks for the tool at each. Data ingestion into Cloud Storage or BigQuery. Data preparation with BigQuery and Dataflow. Model training and tuning on Gemini Enterprise Agent Platform, with Agent Platform AutoML where a custom predictive model is wanted without writing training code. Deployment to an Agent Platform endpoint. And model management, covering versioning, monitoring for drift and the Agent Platform Feature Store for reusable features.

The same instinct applies. Model Garden holds Google's models, open models and third party ones, and a prebuilt API frequently answers the problem before any of this is needed.

Common misconceptions

Deployment is the end of the project.

It is the point at which the model starts to decay. The world it learned from keeps moving, so monitoring for drift and retraining are part of the lifecycle rather than maintenance bolted on afterwards.

Hyperparameter tuning is part of training.

Parameters are learned during training. Hyperparameters are the settings you choose before it, such as learning rate or tree depth, and tuning means training repeatedly under different settings to compare them.

Building your own model is the normal path.

The opposite instinct is the right one. A managed service or an open source pre trained model answers most problems, and custom training is what you reach for when neither fits.

2 questions test this concept

A model that classified messages accurately at launch has degraded over two months. No code has changed, no errors are logged, and the business has launched two new product lines. What is happening?

  • ADrift, because the world the model learned from has moved away from the one it now works in.
  • BOverfitting, which should be addressed by simplifying the model.
  • CUnderfitting, which should be addressed by adding capacity.
  • DA data leak between the training and test sets.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
Designing Machine Learning Systems, On the lifecycle as an operational system rather than a project.
Book
Fundamentals of Data Engineering, On the data work that sits underneath every stage.
Template
Experiment brief, The belief being tested, a hypothesis with a threshold and a date, how long the test runs and on how much traffic, and a table saying in advance what you will do with each possible result.