Evaluation answers two questions that are easy to run together. Is the model right often enough, and is being right this often worth what it cost.
The confusion matrix underneath everything
Every classification metric comes from four counts. True positives and true negatives are the cases the model got right. False positives are things it flagged that were fine. False negatives are things it missed.
Which of those two errors hurts more is a product question, and it decides every metric choice that follows.
The metrics the exam names
Accuracy is the share of predictions that were correct. It is intuitive and it misleads badly on imbalanced data. If fraud is one transaction in a thousand, a model answering no every time is 99.9 per cent accurate and worthless.
Precision asks, of the cases the model flagged, how many were real. Low precision means people wasting time on false alarms.
Recall asks, of the real cases, how many the model found. Low recall means the thing you were trying to catch getting through.
You cannot maximise both. Loosening the threshold to catch more cases pulls in more false alarms. The F1 score combines them into one number, the harmonic mean, which stays low unless both are decent. It is the metric to reach for on imbalanced problems where you care about the rare class.
AUC, the area under the ROC curve, measures how well the model separates the two classes across every possible threshold. Its value is that it is threshold independent, so it tells you whether the model has signal before anybody decides how aggressively to act on it. Half is a coin toss and one is perfect separation.
For regression the errors are distances rather than counts. Mean absolute error averages how far off you were. Root mean squared error does the same while punishing large misses harder, which matters when one big error is worse than several small ones.
Fit
Overfitting is a model that learned the noise in its training data as though it were signal. It performs beautifully on data it has seen and badly on anything new. Underfitting is a model too simple to capture the pattern, performing poorly everywhere.
The way you see either is by keeping data back. A validation set guides choices during development and a test set is touched once, at the end. Reusing the test set to tune anything quietly turns it into training data.
The business metrics
The syllabus is explicit that model metrics are not the whole evaluation, and this is the part product people are best placed to answer.
Cost per user or per inference decides whether the thing survives contact with scale. A model that costs more per call than the decision is worth cannot be fixed by improving its F1 score.
Development cost includes the labelling, the compute and the months, and it is what the alternative gets compared against.
Customer feedback catches what offline metrics cannot, which is whether people trust the output enough to act on it.
Return on investment is the honest question. A model two points more accurate than a simple rule, costing a hundred times more to run, is a worse answer.
On AWS, SageMaker Model Monitor watches these numbers after deployment and SageMaker Clarify breaks performance down by subgroup, which is where an aggregate score hiding a bad result for one group becomes visible.