Schemas and validation

A grammar guarantees the shape of a document and says nothing about the values inside it. That leaves two jobs for a person. Somebody designs the schema the grammar compiles from, and somebody decides what the application does with an object that parses cleanly and carries a value nobody should act on.

Both jobs are older than language models. JSON Schema has been in draft since 2010, its current revision was published in December 2020, and every web service that accepts a request already validates against one at the boundary. What changes with a model is the source of the document. A misbehaving client sends garbage that fails loudly, whereas a model sends something fluent, well typed and wrong, which passes every check a team thought to write.

The sections below set a weak schema against a strong one and name the moves that separate them. They then give the three layers of checking that belong at the boundary, price a repair loop in tokens, and close on a measured finding about what an output constraint does to a model that also holds tools.

Designing a schema a model can fill

A schema written for a database is a description of storage. A schema written for a model is also a prompt, because every name, every enumerated value and every description reaches the window and is read there. The weak version below is the one most teams write first.

{
  "type": "object",
  "properties": {
    "summary":       { "type": "string" },
    "priority":      { "type": "string" },
    "category":      { "type": "string" },
    "refund_amount": { "type": "number" },
    "confidence":    { "type": "number" },
    "customer": {
      "type": "object",
      "properties": {
        "profile": {
          "type": "object",
          "properties": {
            "tier":    { "type": "string" },
            "history": { "type": "array", "items": { "type": "object" } }
          }
        }
      }
    }
  }
}

Six faults sit in twenty lines. Priority and category are open strings, so one run returns "Urgent", the next returns "urgent" and the one after that returns "P1". The refund is a floating point number, so 82.40 comes back as 82.39999999999999 and the rounding argument moves into the billing code. The customer profile is a nested tree the model cannot know, because none of it was in the message it read, so the model invents a tier and an empty history. The confidence field asks for a calibrated probability from something that has no calibrated probabilities to report. Nothing is required, so an empty object validates. And additionalProperties defaults to true, so a field nobody declared arrives and is stored.

{
  "type": "object",
  "additionalProperties": false,
  "required": ["summary", "priority", "category", "refund_minor", "evidence"],
  "properties": {
    "summary": {
      "type": "string",
      "maxLength": 240,
      "description": "One sentence an agent can read in the queue. No greeting, no recommendation."
    },
    "priority": {
      "type": "string",
      "enum": ["p1_outage", "p2_blocked", "p3_degraded", "p4_question"],
      "description": "Use p1_outage only when the customer states the service is unavailable to all of their users."
    },
    "category": {
      "type": "string",
      "enum": ["billing", "access", "data_import", "reporting", "other"],
      "description": "Use other when no category above fits. Do not stretch a category to avoid it."
    },
    "refund_minor": {
      "type": ["integer", "null"],
      "minimum": 0,
      "description": "The refund the customer explicitly asked for, in pence. Null when the message states no amount. Never estimate one."
    },
    "evidence": {
      "type": "array",
      "minItems": 1,
      "maxItems": 3,
      "items": { "type": "string", "maxLength": 300 },
      "description": "Sentences copied word for word from the customer message that support the priority and category above."
    }
  }
}

Six moves separate the two.

Every closed set became an enumeration, so the model picks from a list and the grammar refuses anything outside it.
Money became an integer in minor units, which removes floating point from a number that will end up on an invoice.
The field the model could not know was deleted. A customer tier comes from the customer record, and the application joins it after extraction.
A null branch was added to the refund, so silence in the source has a legal representation and the model is never cornered into inventing a figure.
Descriptions carry the decision rules, because a description is prompt text sitting immediately beside the field it governs.
required and additionalProperties were both set, so an empty object and a surprise field each fail at the boundary.

The evidence array earns its place last and does the most work. A category supported by a quotation the reader can find in the original message is a category somebody can check in two seconds, and a fabricated one becomes visible the moment the quotation is searched for.

Three layers of checking at the boundary

Validation at the boundary is three separate jobs, and teams routinely build the first two and skip the third.

def ingest(raw: str, message: str) -> Ticket:
    # 1. Parse. Constrained decoding makes this branch unreachable. Keep it,
    #    because the constraint is a runtime flag somebody can switch off.
    doc = json.loads(raw)

    # 2. Schema. Types, enums, required fields, lengths, bounds.
    jsonschema.validate(doc, TICKET_SCHEMA)

    # 3. Semantics. Everything JSON Schema has no way to say.
    for quote in doc["evidence"]:
        if quote.strip() not in message:
            raise Unsupported(f"evidence not found in source: {quote[:60]}")

    if doc["priority"] == "p1_outage" and doc["category"] == "billing":
        raise Implausible("a billing question is never a p1 outage")

    if doc["refund_minor"] is not None and doc["refund_minor"] > 50_000:
        raise NeedsApproval("a refund above 500.00 goes to a person")

    return Ticket(**doc)

The third layer is the only one that knows anything about the business. A schema can say that a refund is an integer of at least zero, and it has no way to say that five hundred pounds is the limit of what an automated path may grant. It can require an evidence array of one to three strings, and requiring those strings to appear in the message the model actually read is beyond it. Each of those checks costs a few lines and catches a class of failure that the first two layers pass without comment.

The evidence check is worth singling out. It is a grounding test that runs in microseconds, needs no second model call, and turns an invented quotation into an exception instead of a ticket.

The repair loop and the cost of a retry

An object that fails layer one or layer two can often be fixed by showing the model what broke. That is the repair loop, and its two design decisions are the cap and the choice of which failures are eligible.

MAX_REPAIRS = 2

def extract(message: str) -> Ticket:
    history = []
    for attempt in range(MAX_REPAIRS + 1):
        raw = model(build_prompt(message), history)
        try:
            return ingest(raw, message)

        except (json.JSONDecodeError, jsonschema.ValidationError) as err:
            metrics.increment("extract.repair", attempt=attempt)
            history += [
                assistant(raw),
                user(
                    f"That response failed validation. The error was: {err}\n"
                    "Return the corrected object only, with no other text."
                ),
            ]

        except (Unsupported, Implausible, NeedsApproval):
            # A business rule, not a formatting mistake. Retrying here teaches
            # the model to change its answer until the rule stops firing.
            raise

    metrics.increment("extract.exhausted")
    raise ExtractionFailed(message_id=message_id_of(message))

Only a shape failure is eligible. A semantic failure sent back for repair asks the model to produce whatever gets past the check, and the model obliges, which converts a caught error into an uncaught one. The cap matters for the same reason a loop cap matters anywhere. Without it a malformed schema and a stubborn model can spend an afternoon in conversation with a validator.

Each repair resends the whole prompt. That makes the cost of validation failures easy to work out, and larger than teams expect.

Attempt	Input tokens	Output tokens	Share of requests reaching it
First call	4,000	300	100 per cent
First repair	4,420	300	7 per cent
Second repair	4,840	300	1 per cent

Each repair carries the original prompt, the failed object and the validator message, which is where the extra 420 tokens come from. Averaged over a thousand requests the input cost is 4,000 plus 0.07 of 4,420 plus 0.01 of 4,840, or 4,357.8 tokens. A validation failure rate of 7 per cent therefore costs about 9 per cent more tokens than a clean run, and the figure is worth computing before anybody argues about whether constrained decoding is worth the grammar overhead.

Tool calling suppressed by an output constraint

The last finding is the one that surprises teams, and it only appears once a feature does two things at once. Fangzheng Li, Aimin Zhang and Chen Lv published a study on 24 June 2026 of open weight models asked to satisfy a JSON Schema constraint while also holding tools. Tool invocation stopped. The models kept returning schema compliant answers, and the calls simply never happened.

Their account of the mechanism is mechanical and easy to check. The grammar compiled from the schema produces a token mask, and the tokens that open a tool call are not legal continuations of the constrained document, so the mask makes them unreachable at decode time. The model has not decided against the tool, because the sampler can no longer reach it. They name the pattern Constraint Priority Inversion, since schema satisfaction wins over the task the tool was there to serve.

The tell is visible in the output. Models produced objects carrying a field that announced a lookup was needed, with no lookup in the trace beside it. Any team that ships a constrained agent can watch for exactly that, which is a field saying work is required and a tool log with nothing in it.

The mitigation they propose is the two pass shape already familiar from the page on structured output. One call holds the tools and no output constraint, and a second call, constrained, formats what the first one gathered.

Pass 1   tools on, no schema constraint
         the model searches, reads and decides, in free text

Pass 2   tools off, schema constraint on
         the model fills the object from the transcript of pass 1

Nothing about that result argues against constrained decoding. It argues against turning it on at the same moment and in the same call as the tools, which is the configuration a framework will happily assemble by default. The same caution covers the reasoning loss Tam and colleagues measured in 2024. A constraint belongs on the call that formats, and never on the call that thinks.

Between them, a schema a model can fill and three layers of checking settle what a good answer looks like. Neither says anything about what should happen when the honest answer is that there is no answer, which arrives more often than most designs allow for.