Evaluating a generative model is harder than evaluating a classifier, because there is usually no single right answer. Several outputs can be correct and differently worded, and the metrics available are approximations of a judgement only a person can make.
The automated metrics
ROUGE, Recall Oriented Understudy for Gisting Evaluation, measures how much of a reference text appears in the generated one. It is recall leaning and it is the standard for summarisation, where the question is whether the important material survived.
BLEU, Bilingual Evaluation Understudy, measures how much of the generated text appears in the reference. It is precision leaning and it comes from machine translation, where the question is whether what you produced was warranted.
Both count overlapping word sequences, which is their shared weakness. Say the same thing in different words and you score badly. Copy the reference wording while being wrong and you score well. They are useful for detecting that a change made things worse, not for deciding whether output is any good.
BERTScore compares meaning rather than words, by embedding both texts and measuring similarity. A correct paraphrase scores well, which is the thing the other two get wrong. It costs more to compute and it inherits whatever the embedding model misunderstands.
Perplexity measures how surprised a model is by a piece of text. It is a property of the model rather than of a task, useful when comparing base models and largely irrelevant to whether your application works.
Benchmarks
Published datasets with published scores, covering general knowledge, reasoning, coding and safety. They make models comparable and they are contaminated by scale, since a benchmark on the public internet is plausibly in the training data of everything that came after it.
Treat them as a filter rather than a decision. They tell you which models are worth testing.
Human evaluation
The reference the rest is checked against. People rate output, or choose between two, against criteria that match your case, such as accuracy, tone, completeness or safety.
It is slow and expensive, and it is the only method that sees the things that matter most. Whether an answer would embarrass you. Whether it is subtly wrong in a way a customer would notice. Whether it is useful.
The practical arrangement is a small evaluation set of your own cases, thirty to a hundred, with agreed answers or agreed criteria, scored by people once to establish a baseline. After that automated metrics can watch for regression between versions, with periodic human review to confirm the proxy still tracks.
Business measures
The syllabus is explicit that model scores are not the end of it, and asks whether the model meets business objectives. Productivity, user engagement and task completion.
That is the right final question. A model scoring well on ROUGE and producing summaries nobody reads has failed, and no amount of metric explains why. The useful comparison is always against what people were doing before.
Every platform now offers model evaluation with both automatic metrics and human review, Amazon Bedrock evaluations, Azure AI Foundry evaluations and the evaluation tooling in Google's Agent Platform, alongside open source options such as Ragas and DeepEval that run anywhere. Routing low confidence output to a person is the mechanism for keeping somebody in the path once the system is live, and it is usually a queue and a rule rather than a product.
Practise this
You need one task you actually ship, ten real inputs for it, and two models you can open in a chat window.
Write the rubric before you see any output, because a rubric written afterwards quietly ratifies whatever the model you already preferred produced. Then run the same ten inputs through both models with the prompt held identical, and paste the outputs into one document with the model names stripped, so that you are scoring blind.
Rubric, written first.
3 Correct and complete. I would send it as it stands.
2 Correct but needs an edit before I would send it.
1 Plausible and wrong, or missing something a customer would notice.
0 Refused, off topic, or the wrong format.
Prompt, identical for both models:
Summarise this support ticket as a handover note.
Return three bullets: what the customer wants, what has
been tried, and what is blocked. No preamble.
Ticket:
<one of your ten real cases>
Score sheet: input id, model A, model B, one line of why.
Look at the rows where the two models disagree most, rather than at the totals, because ten cases are too few for a two point gap in a total to mean anything. Watch also for a row where the score you gave and your instinct part company, which usually means the rubric is missing a criterion you actually care about.
An hour of this answers the question a public leaderboard was never asked, which is which model is better at your work.