A foundation model is a large model trained on a very broad dataset and built to be adapted, with no single task in mind. The term was coined at Stanford in 2021 to name a shift that had already happened. Previously a model was trained for a task. A foundation model is trained once, at enormous cost, and then pointed at many tasks by whoever is using it, and that reuse is the whole idea. This page explains the mechanics for somebody who has to make decisions about a model, with nothing in it about building one.
The sections below take the mechanics in the order they meet the input. Text is broken into tokens, tokens are turned into embeddings, a transformer does the work, other families do related work with images, and the last section says what all of it means when choosing.
Tokens
Models do not read characters or words. Text is broken into tokens, which are fragments. Common words tend to be a single token, unusual ones split into several, and punctuation and spaces count.
This is far from trivia. Context limits, throughput limits and the bill are all counted in tokens. A rough English figure is four characters to a token, so a thousand words is around thirteen hundred tokens. Other languages tokenise less efficiently, which means the same sentence in one language can cost noticeably more than in another.
Chunking is the related operation on the way in. A long document is cut into passages small enough to embed and retrieve usefully. Where the cut falls matters, because a chunk that splits a table or an argument in half retrieves badly.
Embeddings and vectors
Once text is tokens, each token is turned into an embedding, which is a list of numbers representing meaning. Text, an image or audio goes in and a vector of a few hundred to a few thousand dimensions comes out, positioned so that things meaning similar things sit near each other.
That property is what makes semantic search possible. "How do I cancel" and "terminate my subscription" share no words and land close together, so a search over embeddings finds the second when the first was asked. Vector databases exist to store these and find nearest neighbours quickly. Every major provider offers somewhere to put them, and several ordinary databases have gained the ability, so the choice is usually about what the organisation already runs. AWS names OpenSearch Service, Aurora, Neptune, DocumentDB and RDS for PostgreSQL, Google Cloud offers vector search in Vertex AI and AlloyDB, and Azure has vector support in AI Search and Cosmos DB.
Transformers
The embeddings are then processed by a transformer, which is the architecture behind almost every current large language model. It was introduced in a 2017 paper from Google called Attention Is All You Need, and its contribution is attention, a mechanism that lets the model weigh how much each token matters to each other token, whatever the distance between them. Earlier architectures read strictly in sequence and lost the thread over long passages.
Attention also processes a sequence in parallel, with no need to move one step at a time, which is what made training on internet scale data practical. Almost everything that followed rests on that.
A large language model is a transformer trained on text at scale. Its training objective is ordinary to the point of being surprising, which is predicting what comes next. Everything else is a consequence of doing that extremely well over an enormous amount of writing.
The context window is the number of tokens the model can attend to at once, and it is a hard boundary. It has grown from a few thousand tokens to a million or more in current models, and research published in 2023 found that models attend less well to material in the middle of a long context than to material at either end, so more input is not reliably more accuracy.
Other model families
Multimodal models take or produce more than one kind of data. A model that reads an image and answers questions about it is multimodal, and so is one that takes text and returns audio.
Diffusion models generate images by starting from noise and removing it step by step towards something matching the prompt. They are the family behind most image generation, and they work quite differently from a transformer predicting the next token.
What this means when choosing a model
Understanding the mechanics changes the questions asked of a vendor. Token pricing means cost scales with conversation length, and a small number of users with long conversations can cost more than a large number with short ones. A context window is a hard boundary, and retrieval augmented generation exists largely to work within it. Embeddings from different models are not interchangeable, so changing the embedding model means reindexing everything that had been embedded.