Concept 4 of 4

Choosing and configuring a foundation model

2 questions test this

Two decisions sit here. Which model, and how it is configured at the moment of the call.

Choosing the model

Modality comes first and eliminates most candidates. Text in and text out, images in, images out, audio, or some combination. A model that cannot take your input is not a candidate whatever else it does well.

Model size and complexity trade quality against cost and latency. A large model reasons better over hard, open ended work. A small one is faster and dramatically cheaper, and on a narrow task such as classification or extraction it is frequently just as good. The common error is paying frontier prices for work a small model finishes.

Latency is a product constraint. Something a user waits for has a budget of a second or two. A batch job overnight has no such limit, and the right model differs between the two.

Cost is per token, counted separately for input and output, with output usually the dearer. That means cost scales with conversation length and document size rather than with users, which surprises teams used to per seat pricing.

Context length is the ceiling on input plus output. It decides how much you can supply in one call, and it is the constraint that makes retrieval necessary rather than optional on large corpora.

Multilingual capability varies far more between models than the marketing suggests, and tokenisation efficiency varies with it, so the same task can cost more in one language than another.

Customisation is whether the model can be fine tuned at all, and under what licence. Some hosted models cannot be, which forecloses an option you may want later.

Inference parameters

These change the answer without changing the model, and knowing which lever does what is most of the skill.

Temperature controls randomness. Near zero, the model repeatedly picks the likeliest next token, giving consistent and conservative output. Higher values flatten the probabilities and admit less likely choices, giving variety and more mistakes. Extraction, classification and anything feeding a downstream system want low. Brainstorming and copy variants want high.

Top p, also called nucleus sampling, restricts the choice to the smallest set of tokens whose probabilities add up to p. It is a different route to the same tradeoff, and tuning both at once makes the effect of either hard to reason about.

Maximum output length caps generation. It protects you from a runaway bill and it truncates, so an answer stopping mid sentence usually means the cap was too low rather than that the model had finished.

Stop sequences end generation when a particular string appears, which is how you keep a model from carrying on past the part you wanted.

Practise this

You need one real task from your product, five genuine examples of it, and any client that lets you set temperature, whether that is a hosted API, a local model or a playground.

Hold the prompt identical and change one thing at a time. Run the five examples at three temperatures, then run them again at your lowest temperature with a short context and a long one. Score every output before you look at which setting produced it, because knowing the setting is how you talk yourself into a difference that is not there.

Prompt, pasted unchanged at every setting

  Extract the customer name, the product mentioned, and the action
  requested from the message below. Return three lines, one field
  per line. Where a field is absent, write UNKNOWN.

  <one real message>

Grid 1, temperature      Grid 2, context at low temperature
         0    0.7   1.0            short   long
  ex 1  ___   ___   ___     ex 1    ___    ___
  ex 2  ___   ___   ___     ex 2    ___    ___
   ...                       ...
  total ___   ___   ___     total   ___    ___

Score  3 = correct and in format
       2 = correct, format wrong
       1 = wrong

The useful result is usually where the score does not move. On extraction and classification the three temperatures often total the same, which tells you the knob everyone reaches for is not what holds your quality back. Where they separate, note whether you lost correctness or format, since a format failure is a prompt problem rather than a sampling one. Context often shifts the totals more than temperature does, and a long context that scores worse is the more interesting finding.

You leave with the two or three settings that matter for your task, which is a shorter list than the one that matters in general.

Where AWS puts this

Amazon Bedrock provides managed access to foundation models from several providers behind one API, with these parameters exposed per call. SageMaker JumpStart deploys pre trained models into your own account when you need control over hosting. PartyRock is a playground for trying prompts without writing code.

The cost points for Bedrock are worth knowing, because they are the ones that change an architecture. On demand token pricing charges per call, which suits variable load, while provisioned throughput reserves capacity and suits predictable high volume. Custom models carry the cost of the customisation on top of either. Regional coverage matters wherever data residency is a constraint, since not every model is offered in every region.

Common misconceptions

The largest model is the safest default.

It is the most expensive and the slowest, and on narrow tasks a smaller model matches it. Model size is a cost and latency decision as much as a quality one, and the usual mistake is paying frontier prices for classification.

Temperature controls how accurate the model is.

It controls randomness in sampling. A low temperature makes the model pick the likeliest continuation more consistently, which reads as more reliable, and it cannot make a model know something it does not.

Maximum output length makes the model write longer answers.

It sets where generation is cut off. Raising it does not produce more content, it only stops truncation, and a response that ends mid sentence usually means the cap was hit rather than that the model finished.

2 questions test this concept

A team is selecting a hosted model for a specialist domain. The plan is to begin with prompting and retrieval, then fine tune on their own annotated cases once about six months of labelled examples have built up. The shortlist is being scored on answer quality, latency and cost per call. What is missing from the scoring?

  • AWhether each candidate can be fine tuned at all, and under what licence, since some hosted models cannot be adapted.
  • BThe price of the fine tuning run itself, which is the figure that decides whether the second phase of the plan is affordable.
  • CNothing important, since prompting and retrieval reach the same quality on a specialist domain and the tuning phase can be dropped.
  • DNothing important, since any hosted model can be fine tuned through its provider once there is enough labelled data.
Check whether it stuck.

One per page, with a worked explanation.

Start the set
Related material
Book
AI Engineering, On model selection as an engineering tradeoff.
Template
RICE scoring sheet, Reach, Impact, Confidence and Effort columns with the score formula built in, defined confidence bands, and a column recording the assumption behind each input.