Evaluating a generative model is harder than evaluating a classifier, because there is usually no single right answer. Several outputs can be correct and differently worded, and the metrics available are approximations of a judgement only a person can make.
The automated metrics
ROUGE, Recall Oriented Understudy for Gisting Evaluation, measures how much of a reference text appears in the generated one. It is recall leaning and it is the standard for summarisation, where the question is whether the important material survived.
BLEU, Bilingual Evaluation Understudy, measures how much of the generated text appears in the reference. It is precision leaning and it comes from machine translation, where the question is whether what you produced was warranted.
Both count overlapping word sequences, which is their shared weakness. Say the same thing in different words and you score badly. Copy the reference wording while being wrong and you score well. They are useful for detecting that a change made things worse, not for deciding whether output is any good.
BERTScore compares meaning rather than words, by embedding both texts and measuring similarity. A correct paraphrase scores well, which is the thing the other two get wrong. It costs more to compute and it inherits whatever the embedding model misunderstands.
Perplexity measures how surprised a model is by a piece of text. It is a property of the model rather than of a task, useful when comparing base models and largely irrelevant to whether your application works.
Benchmarks
Published datasets with published scores, covering general knowledge, reasoning, coding and safety. They make models comparable and they are contaminated by scale, since a benchmark on the public internet is plausibly in the training data of everything that came after it.
Treat them as a filter rather than a decision. They tell you which models are worth testing.
Human evaluation
The reference the rest is checked against. People rate output, or choose between two, against criteria that match your case, such as accuracy, tone, completeness or safety.
It is slow and expensive, and it is the only method that sees the things that matter most. Whether an answer would embarrass you. Whether it is subtly wrong in a way a customer would notice. Whether it is useful.
The practical arrangement is a small evaluation set of your own cases, thirty to a hundred, with agreed answers or agreed criteria, scored by people once to establish a baseline. After that automated metrics can watch for regression between versions, with periodic human review to confirm the proxy still tracks.
Business measures
The syllabus is explicit that model scores are not the end of it, and asks whether the model meets business objectives. Productivity, user engagement and task completion.
That is the right final question. A model scoring well on ROUGE and producing summaries nobody reads has failed, and no amount of metric explains why. The useful comparison is always against what people were doing before.
On AWS, Amazon Bedrock provides model evaluation with both automatic metrics and human review workflows, and Amazon Augmented AI routes low confidence outputs to people, which is the mechanism for keeping a person in the path once the system is live.