Concept 1 of 2

Scaling and emergence

4 questions test this

Two observations shape almost everything about how these systems are built and why they are hard to test.

Scaling laws

Model performance improves predictably as parameters, data and compute increase, and the relationship is smooth enough to extrapolate. A lab can choose a budget and forecast the loss it will reach before spending it.

That predictability is why the last several years looked the way they did. It turned capability into a planning exercise, and it justified spending at a scale that would otherwise be reckless.

The important qualification is what is being predicted. Scaling laws predict loss, the model's error at predicting the next token. They do not tell you which abilities that loss corresponds to.

Emergence

Individual abilities do not improve smoothly along with loss. Many are absent, absent, absent, and then present, across a relatively narrow band of scale. Multi step arithmetic and following an instruction format behave this way.

So a lab can forecast the training curve accurately and still be unable to say what the finished model will do. That gap between a predictable aggregate and unpredictable specifics is the safety problem in this area.

There is a genuine argument that some of this is measurement. A task scored strictly right or wrong shows a sudden jump where a partial credit measure over the same outputs shows a steady climb. The model was improving all along and the metric only noticed at the end.

That does not remove the practical difficulty. Whether the underlying ability was gradual or not, the point at which it becomes usable, and therefore dangerous, still arrives suddenly from the outside.

What follows for evaluation

You cannot enumerate what to test for. A model may have an ability nobody anticipated, so an evaluation covering the abilities you thought of is incomplete in a way you cannot bound.

Testing has to be repeated at each scale. A capability absent in the smaller model says nothing about the larger one, so results do not carry forward across a version.

Elicitation matters more than the model. The same weights can appear incapable and capable depending on prompting, tools and attempts, which is why a capability finding is really a finding about a model and a method together.

What follows for a product

Two practical things.

A model upgrade is a new system. Capabilities appear between versions, so the evaluation set has to be rerun rather than assumed to carry over, and that includes the adversarial cases.

Bigger is not automatically better for you. Scale buys breadth. On a narrow task with clear inputs, a small model often matches a large one at a fraction of the cost and latency, and the burden of proof belongs with the expensive option.

Common misconceptions

Scaling laws predict what a model will be able to do.

They predict loss, which is how well it predicts the next token. The relationship between loss and any particular ability is not something the laws give you, which is why a lab can forecast the training curve and not the capabilities.

Emergent abilities appear from nowhere.

Several apparent jumps come from the measure rather than the model. A task scored right or wrong shows a sudden leap where a partial credit measure shows a smooth climb, so some emergence is an artefact of how it was tested.

A bigger model is a better model.

Bigger models are better on average and worse on cost, latency and energy, and on a narrow task a small one frequently matches them. Scale buys breadth rather than quality at any particular thing.

4 questions test this concept

What do scaling laws actually predict?

  • ALoss, meaning how well the model predicts the next token, rather than which abilities it will have.
  • BThe specific capabilities a model will gain at a given size.
  • CThe point at which a model becomes safe to deploy.
  • DHow much training data is required for a given accuracy on a named task.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On what scale does and does not buy.
Book
The Hundred-Page Machine Learning Book, On the training mechanics underneath.