A foundation model is a large model trained on a very broad dataset and built to be adapted rather than used as it stands. That last part is the whole idea. Previously a model was trained for a task. A foundation model is trained once, at enormous cost, and then pointed at many tasks by whoever is using it.
Tokens
Models do not read characters or words. Text is broken into tokens, which are fragments. Common words tend to be a single token, unusual ones split into several, and punctuation and spaces count.
This is not trivia. Context limits, throughput limits and the bill are all counted in tokens. A rough English figure is four characters to a token, so a thousand words is around thirteen hundred tokens. Other languages tokenise less efficiently, which means the same sentence in one language can cost noticeably more than in another.
Chunking is the related operation on the way in. A long document is cut into passages small enough to embed and retrieve usefully. Where you cut matters, because a chunk that splits a table or an argument in half retrieves badly.
Embeddings and vectors
An embedding is a list of numbers representing meaning. Text, an image or audio goes in and a vector of a few hundred to a few thousand dimensions comes out, positioned so that things meaning similar things sit near each other.
That property is what makes semantic search possible. "How do I cancel" and "terminate my subscription" share no words and land close together, so a search over embeddings finds the second when you asked the first. Vector databases exist to store these and find nearest neighbours quickly, and on AWS the exam names Amazon OpenSearch Service, Amazon Aurora, Amazon Neptune, Amazon DocumentDB and Amazon RDS for PostgreSQL as places embeddings can live.
Transformers
The transformer is the architecture behind almost every current large language model. Its contribution is attention, a mechanism that lets the model weigh how much each token matters to each other token, whatever the distance between them. Earlier architectures read strictly in sequence and lost the thread over long passages.
Attention also processes a sequence in parallel rather than one step at a time, which is what made training on internet scale data practical. Almost everything that followed rests on that.
A large language model is a transformer trained on text at scale. Its training objective is ordinary to the point of being surprising, which is predicting what comes next. Everything else is a consequence of doing that extremely well over an enormous amount of writing.
Other model families
Multimodal models take or produce more than one kind of data. A model that reads an image and answers questions about it is multimodal, and so is one that takes text and returns audio.
Diffusion models generate images by starting from noise and removing it step by step towards something matching the prompt. They are the family behind most image generation, and they work quite differently from a transformer predicting the next token.
What this means when you are choosing
Understanding the mechanics changes the questions you ask a vendor. Token pricing means cost scales with conversation length rather than with users. A context window is a hard boundary, and Retrieval Augmented Generation exists largely to work within it. Embeddings from different models are not interchangeable, so changing your embedding model means reindexing everything you had embedded.