A large language model receives a prompt and builds a response a piece at a time. Those pieces are called tokens. A token can be a whole word, part of a word, a punctuation mark or a piece containing whitespace.
Tokens matter when building an application because models limit how many they can process in a request. To understand that limit, start with how ordinary text becomes model input.
How a tokenizer splits text into tokens
Before the model processes a prompt, software called a tokenizer divides the text into tokens. For example, it might keep a familiar word as one token and split a longer word into several pieces. The exact split depends on the tokenizer, so a word does not always equal one token.
Each token has a number assigned to it, called a token ID. The tokenizer sends those numbers to the model as its input. The numbers identify the text pieces; they are not scores or measures of importance.
The practical consequence is that two prompts with the same word count can have different token counts. Code, punctuation and text in different languages can all change the relationship between words and tokens. Measure the text with the tokenizer used by the intended model when the count matters.
The text changes representation before the model processes it. Follow the steps below from an ordinary request to the generated answer. The diagram shows the stages without assuming a particular tokenizer or token split.
How token counts affect the context window
A model's context window limits the number of tokens it can handle in a request. The instructions, documents and conversation history all use part of that capacity. The application also needs to allow for the response, following the selected model's input and output limits.
For rough planning, 1,500 words of English prose might use about 2,000 tokens. This is an estimate, not a conversion rule. A document with code or unusual formatting may use a different number.
Suppose a support assistant needs access to 200 policies of that size. At the estimated count, the policies alone would occupy about 400,000 tokens. They would not fit in a context window of 200,000 tokens, even before the customer's question was added.
That is a reason to select the relevant policies for each question. The lesson on context windows explains how to budget the available space, while retrieval explains how to find the documents a question needs.
Why a token is different from a letter
Because a token can contain several letters, the model does not necessarily receive each letter as a separate input. However, this does not make the spelling of a word unknowable. A model can learn which letters occur in a word and may answer a letter counting question correctly.
The important distinction for an application is between a generated answer and a count computed directly from the text. If an exact count is required, a short program can calculate it without asking the model to generate an answer.
word = "strawberry"
print(word.count("r")) # 3
How next token prediction produces a response
To generate a response, the model uses the prompt to assign scores to possible next tokens. The generation software selects a token, adds it to the response and continues. Each new choice depends on the prompt and the response produced so far.
This dependency helps explain why an early mistake can affect the rest of an answer. If the model begins by calling a delivery complaint a billing problem, the explanation that follows may support that mistaken classification. It can also correct itself, so the application should check the completed answer.
Sampling settings control how tokens are selected from the possibilities. Changing those settings can change the answer, but a more predictable answer is not necessarily a more accurate one.
What attention does with the prompt
The model needs a way to use information from different parts of the prompt when generating a response. In a transformer model, a mechanism called attention helps combine that information. The original transformer paper, Attention Is All You Need, describes how the mechanism works.
For a support assistant, the relevant information might include the customer's question and a cancellation rule supplied several paragraphs earlier. Attention helps the model use information across that text, but it does not guarantee that the model will apply the right rule.
Adding more documents can therefore make an answer worse if they contain outdated rules or unrelated material. Supply the evidence the task needs, identify which rules are current and test whether the model uses them correctly.
A classification prompt with an explicit decision rule
Consider a ticket that mentions both a duplicate charge and a delivery problem. Asking for a single category leaves the priority decision unresolved. State which issue should take precedence.
Classify the ticket as BILLING or DELIVERY.
If both apply, choose BILLING because duplicate charges need review.
Return only the category.
Ticket: My parcel says delivered, but my card was charged twice.
The prompt settles a decision the model would otherwise have to make for itself. When a ticket contains both problems, the application wants billing to take priority. The expected answer for this example is therefore BILLING.
If software consumes the label, validate it against the allowed values or use structured output. Clear prompting reduces ambiguity; validation decides whether the response can enter the rest of the application.
Practise measuring tokens and testing prompt instructions
Take one short request, one long document and one code sample. Count their tokens with the intended model's tokenizer, then compare those counts with word based estimates. Record where the estimate fails.
Next, write six support tickets, including two that mention both billing and delivery. Label the expected answers before running the prompt above. Compare the model's answers with those labels, then remove the priority rule and repeat the test.
Keep a table containing each ticket, its expected label and both model answers. Inspect any differences to see whether the priority rule helped. Six cases are enough to inspect this behaviour, but they are not enough to establish how reliably the prompt will work on real traffic. The Python evaluation lab develops this comparison further.