Choosing a foundation model and configuring it are two decisions that sit together. The first is which model, decided against a small set of criteria that actually separate one from another. The second is how the model is configured at the moment of the call, through a handful of inference parameters that change the answer without changing the model. Both vendor examinations test the criteria and the parameters, and the Practitioner course cares about them because the wrong choice on either is a safety problem before it is a cost one.
The sections below take the selection criteria in the order they eliminate candidates, then the inference parameters and what each lever does, then where the vendor platforms place the material, and end with a fifteen minute experiment.
Choosing the model
Modality comes first and eliminates most candidates. Text in and text out, images in, images out, audio or some combination. A model that cannot take the input is no candidate whatever else it does well.
Model size and complexity trade quality against cost and latency. A large model reasons better over hard, open ended work. A small one is faster and dramatically cheaper, and on a narrow task such as classification or extraction it is frequently just as good. The common error is paying frontier prices for work a small model finishes.
Latency is a product constraint. Something a user waits for has a budget of a second or two. A batch job overnight has no such limit, and the right model differs between the two.
Cost is per token, counted separately for input and output, with output usually the dearer. That means cost scales with conversation length and document size, and a team used to per seat pricing is surprised to find that the number of users is a weak predictor of the bill.
Context length is the ceiling on input plus output. It decides how much can be supplied in one call, and it is the constraint that makes retrieval necessary on large corpora.
Multilingual capability varies far more between models than the marketing suggests, and tokenisation efficiency varies with it, so the same task can cost more in one language than another.
Customisation is whether the model can be fine tuned at all, and under what licence. Some hosted models cannot be, which forecloses an option a team may want later.
Inference parameters
Once the model is chosen, four parameters change the answer without changing the model, and knowing which lever does what is most of the skill.
Temperature controls randomness. Near zero, the model repeatedly picks the likeliest next token, giving consistent and conservative output. Higher values flatten the probabilities and admit less likely choices, giving variety and more mistakes. Extraction, classification and anything feeding a downstream system want low. Brainstorming and copy variants want high.
Top p, also called nucleus sampling, restricts the choice to the smallest set of tokens whose probabilities add up to p. It is a different route to the same tradeoff, and tuning both at once makes the effect of either hard to reason about.
Maximum output length caps generation. It protects against a runaway bill and it truncates, so an answer stopping mid sentence usually means the cap was too low, and the model had more to say.
Stop sequences end generation when a particular string appears, which is how a model is kept from carrying on past the part that was wanted.
Where AWS puts this
Amazon Bedrock provides managed access to foundation models from several providers behind one API, with these parameters exposed per call. SageMaker JumpStart deploys pre trained models into the customer's own account where control over hosting is needed. PartyRock is a playground for trying prompts without writing code. Google Cloud offers the equivalent catalogue through its model garden, with the same parameters exposed.
The cost points for Bedrock are worth knowing, because they are the ones that change an architecture. On demand token pricing charges per call, which suits variable load, while provisioned throughput reserves capacity and suits predictable high volume. Custom models carry the cost of the customisation on top of either. Regional coverage matters wherever data residency is a constraint, since not every model is offered in every region.
Practise this
The exercise needs one real task from the product, five genuine examples of it and any client that lets temperature be set, whether a hosted API, a local model or a playground.
Hold the prompt identical and change one thing at a time. Run the five examples at three temperatures, then run them again at the lowest temperature with a short context and a long one. Score every output before looking at which setting produced it, because knowing the setting is how a person talks themselves into a difference that is not there.
Prompt, pasted unchanged at every setting
Extract the customer name, the product mentioned, and the action
requested from the message below. Return three lines, one field
per line. Where a field is absent, write UNKNOWN.
<one real message>
Grid 1, temperature Grid 2, context at low temperature
0 0.7 1.0 short long
ex 1 ___ ___ ___ ex 1 ___ ___
ex 2 ___ ___ ___ ex 2 ___ ___
... ...
total ___ ___ ___ total ___ ___
Score 3 = correct and in format
2 = correct, format wrong
1 = wrong
The useful result is usually where the score does not move. On extraction and classification the three temperatures often total the same, which says that the knob everyone reaches for is not what holds the quality back. Where they separate, note whether correctness or format was lost, since a format failure is a prompt problem and sampling had nothing to do with it. Context often shifts the totals more than temperature does, and a long context that scores worse is the more interesting finding.
The team leaves with the two or three settings that matter for its task, which is a shorter list than the one that matters in general.