Domain 4 of 7
Capability, Scaling and Evaluation Scaling laws and why capability arrives unevenly, what an evaluation can and cannot establish, elicitation and the dangerous capability evaluations frontier labs now run.
Concepts in this domain
01 Scaling and emergence Why capability improves predictably with size while individual abilities appear abruptly, and what that does to the job of testing a model. 02 Capability evaluations What an evaluation can establish about a model, why the method used to test a model decides the answer as much as the model does, and what a dangerous capability evaluation is for. 03 Agents and autonomy risk What changes when a model is given tools, memory and several steps to itself, and why the danger is in the permissions, whatever the intelligence. 04 Deceptive alignment and situational awareness What it would mean for a system to behave well only while it believes it is being watched, what the evidence for that actually shows, and why the possibility changes how evaluations are run. Try a question from this domain
What do scaling laws actually predict?
A The specific capabilities a model will gain at a given size. B Loss, meaning how well the model predicts the next token, rather than which abilities it will have. C The point at which a model becomes safe to deploy. D How much training data is required for a given accuracy on a named task. 12 questions on this domain.
One per page, with a worked explanation.
Start the set