Retrieval decides what material reaches the window, and the model then answers in prose. That is enough until a feature has to store a decision, route a ticket or update a record, because a program needs named fields with declared types and a paragraph gives it neither. Structured output covers every arrangement that closes that gap, and three of them are in common use.
The three arrived at different moments and offer very different guarantees. Asking for JSON inside the prompt is as old as instruction following itself. Function calling then reached the main provider interfaces during 2023 and hands the model a schema it is asked to follow. Constrained decoding goes further again, compiling that same schema into a grammar that stops the sampler from choosing any token which would break the document, and OpenAI shipped it as Structured Outputs on 6 August 2024. The difference between the three is the difference between a request, a description and a mechanism.
The sections below take the routes in order of how much each one promises, and show what a grammar does to the sampler at the moment one token is chosen. A table then sets the three against guarantee, cost and failure mode. The last two sections give the published measurements, including the finding that a constraint on the output changes what the model does with the rest of the task.
Asking for JSON inside the prompt
Asking inside the prompt costs nothing and adds no dependency. The prompt names the shape and the model usually produces it.
Extract the order details from the email below and return JSON.
<email>
{{email_body}}
</email>
A parser meets that result before any person does. The common failure there is a perfectly well formed object with conversation wrapped around it.
raw = response.output_text
data = json.loads(raw)
# json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
#
# The model answered:
#
# Sure! Here's the extracted data:
# (a markdown code fence opens here)
# {"order_id": "A-4471", "items": 3, "total": "GBP 82.40"}
# (and closes here)
# Let me know if you'd like anything else.
Nothing in the prompt forbade the greeting, the fence or the closing offer, so the model supplied all three. A stronger prompt closes each opening explicitly and shows the shape by example.
Extract the order details from the email below.
Output a single JSON object and nothing else. Do not write any text before or
after it. Do not wrap it in a code fence.
Fields:
order_id string, the reference beginning with a letter and a hyphen
items integer, the count of distinct line items
total string, the amount with its ISO 4217 code, e.g. "GBP 82.40"
currency string, the three letter code on its own
Example output for a different email:
{"order_id":"B-1120","items":2,"total":"EUR 45.00","currency":"EUR"}
<email>
{{email_body}}
</email>
The second prompt names the container, forbids the two decorations the first one invited, and gives one filled example so the model has a shape to copy. That lifts the success rate a long way and guarantees nothing, because the instruction competes with every other token in the window and the sampler remains free to ignore it.
Function calling with the schema as a description
Function calling moves the shape out of the prose and into a field the provider understands. The application declares a function with a name, a description and a JSON Schema for its parameters, and the model replies with a call to that function.
{
"name": "record_order",
"description": "Store one order extracted from a customer email. Call this once per email, after reading the whole message.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"pattern": "^[A-Z]-[0-9]{4}$",
"description": "The reference printed near the top of the email."
},
"items": {
"type": "integer",
"minimum": 1,
"description": "Count of distinct line items, not total quantity."
},
"amount_minor": {
"type": "integer",
"description": "The total in minor units. 82.40 pounds is 8240."
},
"currency": { "type": "string", "enum": ["GBP", "EUR", "USD"] }
},
"required": ["order_id", "items", "amount_minor", "currency"],
"additionalProperties": false
}
}
The schema is serialised into the window along with everything else, so the model reads it the way it reads the rest of the prompt. In its original form the provider never checks the emitted arguments against the schema, which means a missing field, an extra field or the string "3" where an integer was declared all reach the application looking like a successful call. Several providers later added a strict mode that applies the third route to the arguments, and that mode is the one worth turning on.
Constrained decoding and the token mask
Constrained decoding changes the sampler itself. Every step of generation produces a score for each token in the vocabulary, and the sampler picks from that distribution. A grammar sits between the two. Compiled from the schema, it tracks the state of the partial document, works out which tokens could continue a valid one, and sets the score of every other token to negative infinity before the sampler runs.
Partial output so far: {"order_id": "A-4471", "items":
Grammar state: value position, declared type integer, minimum 1
Continuations allowed: a space, then one of 1 2 3 4 5 6 7 8 9
Continuations masked: every other token in the vocabulary
token raw score after the mask
" 8.9 -inf would open a string
3 7.4 7.4 a digit is legal here
null 6.1 -inf null was not declared
{ 5.2 -inf an object was not declared
} 4.7 -inf a required field is unfilled
A masked token carries probability zero, so no temperature can bring it back. The document is valid by construction and needs no repair loop for its shape. Two costs come with that. Compiling and stepping the grammar takes work on every token, and a grammar enforces only the parts of JSON Schema the engine implements, so a schema using an unsupported feature is either refused outright or quietly under enforced.
The three routes on guarantee, cost and failure mode
| Route | What it guarantees | What it costs | How it fails |
|---|---|---|---|
| Prompting for JSON | Nothing. The instruction is one claim on attention among many | Only the tokens the instruction occupies | Silently. A greeting, a code fence or a trailing offer arrives and the parser raises |
| Function calling | A named call carrying an arguments object, checked against the schema only in strict mode | The schema occupies window space on every request | A plausible arguments object with a field missing, a type wrong or a value the enum never listed |
| Constrained decoding | A document that parses and matches every schema feature the engine implements | Grammar compilation, then a mask computed at every token | Loudly at setup, since an unsupported feature is refused, or quietly by leaving that feature unenforced |
The ordering is no ranking of how often a team should reach for each one. Prompting is the right answer where the output goes to a person. The moment a program consumes the output, the question becomes which of the two enforced routes the provider offers and what the schema actually needs.
What the published measurements found
OpenAI published the first widely quoted figure alongside Structured Outputs on 6 August 2024. On its own evaluation of complex JSON schema following, gpt-4o-2024-08-06 with the feature enabled scored 100 per cent, and gpt-4-0613 scored under 40 per cent. A vendor measured its own model on its own set there, and the shape of the result is still the point, because a mechanism that masks illegal tokens has no path to an invalid document.
A more independent picture came from JSONSchemaBench, published by Saibo Geng and colleagues in January 2025, which ran 10,000 real world JSON schemas through six constrained decoding frameworks. The benchmark separates three measures worth carrying into any procurement conversation. Declared coverage is the share of schemas a framework accepts at all. Empirical coverage is the share whose generated output validates. Compliance rate is the second figure divided by the first, meaning how often a framework that agreed to a schema then honoured it.
The spread was wide. Guidance held the highest compliance rate, above 0.94 on most collections, and on the hardest collection of schemas taken from GitHub its empirical coverage fell to 0.41, with Outlines reaching 0.03. The closed providers behaved differently again. OpenAI and Gemini recorded a compliance rate of 1.00 on the schemas they accepted, because each implements a subset of JSON Schema and refuses anything outside it. A refusal at setup is the failure a team can actually see.
What a constraint does to the rest of the task
Forcing the shape is not free for the content. Zhi Rui Tam and colleagues published a study in August 2024, presented at the EMNLP industry track that year, setting the same models on the same tasks in free text and under format restriction. Reasoning performance fell under restriction, and stricter formats cost more than looser ones. The detail that matters for an engineer is the control they ran alongside it. On one task under a JSON constraint the parsing error rate came to 0.148 per cent while the accuracy gap against free text was 38 percentage points, so the loss lives inside the generation and never in the parsing.
JSONSchemaBench measured downstream quality too and reported the opposite sign, with Guidance about 3 per cent ahead of unconstrained generation on its tasks. Both results hold, because the tasks differ. Extraction and classification gain from a constraint that removes the model's freedom to waffle, while multi step reasoning loses, since the tokens the model would have spent working towards an answer are exactly the tokens the grammar forbids.
That points at a shape worth knowing. The model reasons in free text in one call, and a second short constrained call does nothing but format the conclusion the first one reached.
Call 1, unconstrained, temperature 0.2
system: Work through the customer's email step by step. State the order
reference you found, how you counted line items and how you read the
total. Finish with a one line conclusion.
Call 2, constrained to the record_order schema, temperature 0
system: Convert the analysis below into the required object. Copy the values
the analysis states. Do not recompute anything.
user: {{analysis_from_call_1}}
The second call has no reasoning left to do, so the grammar takes nothing away from it. The cost is a second request, which the page on cost and latency prices in full.
A grammar decides the shape of a document and holds no opinion on its content. Somebody still has to design the schema the grammar compiles from, and decide what the application does with a document that parses cleanly and says something untrue.