Two observations shape almost everything about how these systems are built and why they are hard to test.
Scaling laws
Model performance improves predictably as parameters, data and compute increase, and the relationship is smooth enough to extrapolate. A lab can choose a budget and forecast the loss it will reach before spending it.
That predictability is why the last several years looked the way they did. It turned capability into a planning exercise, and it justified spending at a scale that would otherwise be reckless.
The important qualification is what is being predicted. Scaling laws predict loss, the model's error at predicting the next token. They do not tell you which abilities that loss corresponds to.
Emergence
Individual abilities do not improve smoothly along with loss. Many are absent, absent, absent, and then present, across a relatively narrow band of scale. Multi step arithmetic and following an instruction format behave this way.
So a lab can forecast the training curve accurately and still be unable to say what the finished model will do. That gap between a predictable aggregate and unpredictable specifics is the safety problem in this area.
There is a genuine argument that some of this is measurement. A task scored strictly right or wrong shows a sudden jump where a partial credit measure over the same outputs shows a steady climb. The model was improving all along and the metric only noticed at the end.
That does not remove the practical difficulty. Whether the underlying ability was gradual or not, the point at which it becomes usable, and therefore dangerous, still arrives suddenly from the outside.
What follows for evaluation
You cannot enumerate what to test for. A model may have an ability nobody anticipated, so an evaluation covering the abilities you thought of is incomplete in a way you cannot bound.
Testing has to be repeated at each scale. A capability absent in the smaller model says nothing about the larger one, so results do not carry forward across a version.
Elicitation matters more than the model. The same weights can appear incapable and capable depending on prompting, tools and attempts, which is why a capability finding is really a finding about a model and a method together.
What follows for a product
Two practical things.
A model upgrade is a new system. Capabilities appear between versions, so the evaluation set has to be rerun rather than assumed to carry over, and that includes the adversarial cases.
Bigger is not automatically better for you. Scale buys breadth. On a narrow task with clear inputs, a small model often matches a large one at a fraction of the cost and latency, and the burden of proof belongs with the expensive option.