Evaluating a foundation model is harder than evaluating a classifier, because a generative task usually has no single right answer. Several outputs can be correct and differently worded, so the metrics available are approximations of a judgement only a person can make. The automated scores come from machine translation and summarisation research of the early 2000s, the benchmarks come from the academic race to compare models, and human evaluation comes from the plain fact that nothing else sees what matters most. The AWS syllabus names ROUGE, BLEU and BERTScore, and the Practitioner course cares because a model shipped on a leaderboard score alone is a model nobody has tested on the work it will do.
The sections below take the automated metrics and what each one sees, the published benchmarks and their contamination problem, human evaluation, the newer practice of using a model as the judge, the business measures that come last and where the tooling sits, and end with an hour long blind comparison.
The automated metrics and what each one sees
Four automated metrics cover what the syllabus asks, and each one sees a different thing.
ROUGE, Recall Oriented Understudy for Gisting Evaluation, measures how much of a reference text appears in the generated one. It is recall leaning and it is the standard for summarisation, where the question is whether the important material survived. It was published in 2004 for exactly that purpose.
BLEU, Bilingual Evaluation Understudy, measures how much of the generated text appears in the reference. It is precision leaning and it comes from machine translation, where the question is whether what was produced was warranted. It dates from a 2002 paper at IBM and was the first automated score shown to track human judgement of translations closely enough to be useful.
Both count overlapping word sequences, which is their shared weakness. Saying the same thing in different words scores badly. Copying the reference wording while being wrong scores well. They are useful for detecting that a change made things worse, and useless for deciding whether output is any good.
BERTScore compares meaning, by embedding both texts and measuring similarity token by token. A correct paraphrase scores well, which is the thing the other two get wrong. It was published in 2019, it costs more to compute and it inherits whatever the embedding model misunderstands.
Perplexity measures how surprised a model is by a piece of text. It is a property of the model and of no task, useful when comparing base models and largely irrelevant to whether an application works.
Published benchmarks and the contamination problem
Where the metrics need a reference answer, benchmarks supply thousands of them. They are published datasets with published scores, covering general knowledge, reasoning, coding and safety. MMLU, from 2020, asks fifty seven subjects' worth of multiple choice questions. The HELM effort at Stanford in 2022 assembled dozens of scenarios and scored models on all of them at once. Leaderboards built on such sets make models comparable in a single table.
They are also contaminated by scale. A benchmark on the public internet is plausibly in the training data of everything that came after it, so a model may have seen the questions, and several studies have found scores dropping sharply when a benchmark is rephrased or its answers reordered. The other limit is that they measure general capability, which correlates weakly with narrow applied work.
Both limits point the same way. A benchmark is a filter, and the decision is made elsewhere. It says which models are worth testing.
Human evaluation as the reference
The testing that decides is human. People rate output, or choose between two, against criteria that match the case, such as accuracy, tone, completeness or safety. Pairwise choice is the more reliable form, because people agree far more about which of two answers is better than about what score a single answer deserves, and it is the form the public comparison platforms adopted in 2023, ranking models from millions of anonymous head to head votes.
It is slow and expensive, and it is the only method that sees the things that matter most. Whether an answer would embarrass the company. Whether it is subtly wrong in a way a customer would notice. Whether it is useful.
The practical arrangement is a small evaluation set of the team's own cases, thirty to a hundred, with agreed answers or agreed criteria, scored by people once to establish a baseline. After that automated metrics can watch for regression between versions, with periodic human review to confirm the proxy still tracks.
Using a model as the judge
Between the cheap metrics and the expensive people sits a third option that has become standard. A strong model is given the rubric and asked to score or compare outputs, which is called using a model as a judge. A 2023 study that introduced the practice found that a frontier model agreed with human raters about as often as human raters agreed with each other, at a fraction of the cost.
The same study found the biases, and they are the reason the arrangement needs supervision. A judging model prefers the first of two answers it is shown, which is fixed by showing both orders. It prefers longer answers, which is fixed by a rubric that penalises padding. It prefers answers in its own style, which is fixed by using a different model as judge from the one being judged. A team that uses a model judge without checking a sample against people has automated its blind spots along with its scoring.
The business measures that come last
None of the scores above is the end of it, and the syllabus is explicit that the final question is whether the model meets business objectives. Productivity, user engagement and task completion.
That is the right final question. A model scoring well on ROUGE and producing summaries nobody reads has failed, and no amount of metric explains why. The useful comparison is always against what people were doing before.
Where the tooling sits
Every platform now offers model evaluation with both automatic metrics and human review. Amazon Bedrock has a model evaluation feature, Azure AI Foundry has evaluations and Google Cloud offers an evaluation service inside Vertex AI, alongside open source options such as Ragas and DeepEval that run anywhere. Routing low confidence output to a person is the mechanism for keeping somebody in the path once the system is live, and it is usually a queue and a rule, with no product involved.
Practise this
The exercise needs one task the team actually ships, ten real inputs for it and two models that can be opened in a chat window.
Write the rubric before seeing any output, because a rubric written afterwards quietly ratifies whatever the model already preferred produced. Then run the same ten inputs through both models with the prompt held identical, and paste the outputs into one document with the model names stripped, so that the scoring is blind.
Rubric, written first.
3 Correct and complete. I would send it as it stands.
2 Correct but needs an edit before I would send it.
1 Plausible and wrong, or missing something a customer would notice.
0 Refused, off topic, or the wrong format.
Prompt, identical for both models:
Summarise this support ticket as a handover note.
Return three bullets: what the customer wants, what has
been tried, and what is blocked. No preamble.
Ticket:
<one of the ten real cases>
Score sheet: input id, model A, model B, one line of why.
Look at the rows where the two models disagree most, because ten cases are too few for a two point gap in a total to mean anything. Watch also for a row where the score given and the scorer's instinct part company, which usually means the rubric is missing a criterion the team actually cares about.
An hour of this answers the question a public leaderboard was never asked, which is which model is better at the team's work.