Two decisions sit here. Which model, and how it is configured at the moment of the call.
Choosing the model
Modality comes first and eliminates most candidates. Text in and text out, images in, images out, audio, or some combination. A model that cannot take your input is not a candidate whatever else it does well.
Model size and complexity trade quality against cost and latency. A large model reasons better over hard, open ended work. A small one is faster and dramatically cheaper, and on a narrow task such as classification or extraction it is frequently just as good. The common error is paying frontier prices for work a small model finishes.
Latency is a product constraint. Something a user waits for has a budget of a second or two. A batch job overnight has no such limit, and the right model differs between the two.
Cost is per token, counted separately for input and output, with output usually the dearer. That means cost scales with conversation length and document size rather than with users, which surprises teams used to per seat pricing.
Context length is the ceiling on input plus output. It decides how much you can supply in one call, and it is the constraint that makes retrieval necessary rather than optional on large corpora.
Multilingual capability varies far more between models than the marketing suggests, and tokenisation efficiency varies with it, so the same task can cost more in one language than another.
Customisation is whether the model can be fine tuned at all, and under what licence. Some hosted models cannot be, which forecloses an option you may want later.
Inference parameters
These change the answer without changing the model, and knowing which lever does what is most of the skill.
Temperature controls randomness. Near zero, the model repeatedly picks the likeliest next token, giving consistent and conservative output. Higher values flatten the probabilities and admit less likely choices, giving variety and more mistakes. Extraction, classification and anything feeding a downstream system want low. Brainstorming and copy variants want high.
Top p, also called nucleus sampling, restricts the choice to the smallest set of tokens whose probabilities add up to p. It is a different route to the same tradeoff, and tuning both at once makes the effect of either hard to reason about.
Maximum output length caps generation. It protects you from a runaway bill and it truncates, so an answer stopping mid sentence usually means the cap was too low rather than that the model had finished.
Stop sequences end generation when a particular string appears, which is how you keep a model from carrying on past the part you wanted.
Practise this
You need one real task from your product, five genuine examples of it, and any client that lets you set temperature, whether that is a hosted API, a local model or a playground.
Hold the prompt identical and change one thing at a time. Run the five examples at three temperatures, then run them again at your lowest temperature with a short context and a long one. Score every output before you look at which setting produced it, because knowing the setting is how you talk yourself into a difference that is not there.
Prompt, pasted unchanged at every setting
Extract the customer name, the product mentioned, and the action
requested from the message below. Return three lines, one field
per line. Where a field is absent, write UNKNOWN.
<one real message>
Grid 1, temperature Grid 2, context at low temperature
0 0.7 1.0 short long
ex 1 ___ ___ ___ ex 1 ___ ___
ex 2 ___ ___ ___ ex 2 ___ ___
... ...
total ___ ___ ___ total ___ ___
Score 3 = correct and in format
2 = correct, format wrong
1 = wrong
The useful result is usually where the score does not move. On extraction and classification the three temperatures often total the same, which tells you the knob everyone reaches for is not what holds your quality back. Where they separate, note whether you lost correctness or format, since a format failure is a prompt problem rather than a sampling one. Context often shifts the totals more than temperature does, and a long context that scores worse is the more interesting finding.
You leave with the two or three settings that matter for your task, which is a shorter list than the one that matters in general.
Where AWS puts this
Amazon Bedrock provides managed access to foundation models from several providers behind one API, with these parameters exposed per call. SageMaker JumpStart deploys pre trained models into your own account when you need control over hosting. PartyRock is a playground for trying prompts without writing code.
The cost points for Bedrock are worth knowing, because they are the ones that change an architecture. On demand token pricing charges per call, which suits variable load, while provisioned throughput reserves capacity and suits predictable high volume. Custom models carry the cost of the customisation on top of either. Regional coverage matters wherever data residency is a constraint, since not every model is offered in every region.