Domain 2 of 7
Training and Alignment of Models Pre training, supervised fine tuning, reinforcement learning from human feedback and constitutional methods. What each stage adds, and why the helpfulness of a model is a product of that training.
Concepts in this domain
01 How models are trained to behave Pre training, supervised fine tuning, reinforcement learning from human feedback and constitutional methods, and why a helpful model is a manufactured artefact that can be unmade the same way. 02 Foundation models and how they work Tokens, embeddings and transformers explained for somebody who has to make decisions about a model, with nothing in it about building one. 03 Supervised, unsupervised and reinforcement learning The three ways a model learns, what each one needs from the data, and how to tell which problem is actually in hand. 04 The alignment problem Two questions hiding inside one word, whether the objective specified was the right one and whether the trained system actually acquired it. 05 Scalable oversight The problem of supervising a system on work a person cannot check, the research directions aimed at it, and the control available in the meantime. Try a question from this domain
What does a base model do before any further training is applied?
A It answers questions, but refuses nothing and holds no conversation. B It behaves as a helpful assistant, and less reliably than a tuned model. C It continues text, and does not answer questions, refuse anything or hold a conversation. D It produces random output until it is given examples to imitate. 15 questions on this domain.
One per page, with a worked explanation.
Start the set