A team running the underwriting agent reads that their model provider now offers schema-constrained output. They send recommendation.v1.json with each request, the share of outputs that fail the parser drops to nothing in their logs, and someone proposes to delete the validation code: "the provider guarantees the schema now". That proposal is the mistake this page is about. It confuses a guarantee about the shape of the text with a guarantee about the record, and it reads a guarantee that has documented exceptions as one that has none.
Here is what the validation code would have caught in the following week, using this site's own UW-S cases (scripted outputs built on the facts of the series' shared cases). An output for SUB-1004 stops in the middle of its second reason because it reached the output limit (UW-S03). An output says request_info and lists nothing as missing (UW-S08). An output cites policy_version "2025.4", copied from a past decision, in a run that read 2026.2 (UW-S12). The first is a cut-off the providers' own documents warn about. The other two have exactly the agreed shape. No decoding setting can know that a rule does not apply to this roof.
The model recommends; a person decides. Between the model and that person sit two kinds of help: what the provider does while the text is written, and what your workflow checks after. This page separates the two.
Four ways to ask a model for structure#
Plain English. There are four common ways to get structured output from a model, and they differ in who enforces the shape and when.
Prompt only. The instructions ask for JSON in a given shape, perhaps with an example. Nothing enforces it. The model usually follows; when it does not, you find out in the parser. Case UW-S01 (a fluent paragraph with no JSON) and UW-S02 (correct JSON inside a markdown code fence) are what this mode produces on a bad day.
JSON mode. The provider promises that the output is syntactically valid JSON, and nothing about its keys or values. An output in JSON mode can be {"decision": "Escalate"} and keep the promise.
Schema-constrained output. You send a JSON Schema, and the provider restricts what the model can write so that the output follows it. The technique behind it is called constrained decoding, explained in the next section. Providers name this feature differently (structured outputs, JSON outputs, response schema); this site calls the general idea a structured output mode.
Tool arguments. You describe a tool with an input schema, and the model "calls" it by writing arguments for that schema. The underwriting agent writes {"submission_id": "SUB-1004"} for get_roof_assessment. Whether those arguments are constrained or only requested depends on the provider and on a setting, often called strict.
Diagram: Four ways to ask for structure · each removes one more kind of failure, none removes the last three
Read left to right. Each mode moves one class of failure from possible to reduced. The workflow's validator on the right is the only step that sees the whole record and the facts of the run.
- Prompt only
- Ask in the instructions: The model usually follows the requested shape; nothing makes it.
- JSON mode
- Syntax promised: The output parses as JSON; its keys and values are not checked.
- Schema-constrained
- Shape constrained: Decoding follows the schema the provider supports; cut-offs and refusals remain.
- Tool arguments: The model fills a tool's input schema; constrained only with a strict setting the provider supports.
- Your validator
- Four stages: Parse, schema, cross-field and domain checks see the record and the world of the run.
Read the diagram as a ladder. Each step up removes one kind of failure from the producer's side: prose, then the wrong shape. The box on the right never goes away, because it is the only one that sees the whole record and the facts of the run.
Before AI. The closest older idea is a client stub generated from an interface definition, such as a gRPC client built from a protobuf file. The stub cannot send a field of the wrong type, and still every server validates the request, because a correctly typed request can carry a customer id that does not exist. Schema-constrained output is a generated stub for a producer that writes text: it shapes the request, and the business checks stay on the receiving side.
How constrained decoding works#
A language model writes text one token at a time. A token is a piece of text: a word, part of a word, a bracket, a quote. At each step the model gives a score to every token in its vocabulary, and the sampler picks one. Nothing in that loop knows about your schema.
Plain English. Constrained decoding adds one check to every step: before the sampler picks, the tokens that would break the schema are removed from the choice. If the output so far is {"recommendation": ", the next characters must begin one of the four allowed words, so a token that starts approved" is not available, while approve is. The schema is first translated into a grammar, a set of rules saying which text may come next at every point, and the grammar is what the decoder consults. That is the concept; how a provider implements it is product-specific, and the documents read for this page describe it only in general terms. One of them, the strict tool use page of the Claude documentation, calls it "constraining the model's token sampling to schema-valid outputs (a technique called grammar-constrained sampling)".
The example above is illustrative. The tokens of a real model do not line up with the words of an enum, and a grammar has to handle that. What follows from the mechanism is solid, though, and it explains every documented gap below.
Diagram: Constrained decoding, one request · the grammar removes tokens, the workflow still checks
Time runs down the page. The provider compiles the schema into a grammar, then at every step removes the tokens the grammar forbids before one is picked. Generation can still end early (a cut-off) or with a refusal. The text that comes back goes through all four validation stages, because the grammar knows the shape and nothing about the world.
- The workflow (your code) to Provider (the service): Prompt + schema, output limit. The request carries the instructions, the case and the JSON Schema of the recommendation.
- Provider (the service) to Grammar (compiled schema): Compile the schema (first use is slower). Providers document extra latency the first time a schema is used, and limits on schema complexity.
- Model to Grammar (compiled schema): Scores for every next token. The model proposes; it would happily write any token.
- Grammar (compiled schema) to Model: Only allowed tokens remain. Tokens that would break the grammar are removed before sampling.
- Model: Repeat until the object closes, or the limit, or a refusal. The constraint works token by token. A limit or a refusal can end the text before the object is complete.
- Provider (the service) to The workflow (your code): Text + stop reason. The stop reason says whether the object completed, was cut off or was refused.
- The workflow (your code): Parse, schema, cross_field, domain. All four stages run anyway. The grammar cannot know that a rule does not apply to this roof.
Three consequences follow from the loop in the diagram.
The constraint applies to the next token, not to the finished object. If generation stops early, because the output limit is reached, the text is a valid beginning of an object and not an object. This is called truncation, and it breaks the parser exactly like UW-S03.
The constraint applies only while the model writes the structured part. A model that declines to answer for safety reasons may produce a refusal (a message saying it will not do the task) instead of the object, and providers document that this refusal takes precedence over the schema.
The constraint knows the grammar, not the world. It can forbid "approved". It cannot forbid "decline" for a roof of 8 years, a citation of a rule that does not apply, or "2025.4" where the run read "2026.2". Those are values the schema allows.
And one cost follows: the schema must be compiled into a grammar before it can be used. Two of the providers read for this page document extra latency on the first request with a new schema, and both document limits on how large or complex a schema may be.
What each mode leaves possible#
The lab takes the four modes and seven classes of failure from the earlier pages. Pick a mode to see which failures it is designed to prevent, which it reduces, and which stay exactly as possible as before. It is a concept table, not a claim about any one provider: "reduced" means the mode is designed to prevent the failure when the provider supports the mode and your schema.
The same grid as a table, from the lab's data (scripts/build_data.py, MODES):
| Failure class (layer) | Prompt only | JSON mode | Schema-constrained | Tool arguments |
|---|---|---|---|---|
| Not JSON at all: prose, a code fence (syntax) | possible | reduced | reduced | reduced |
| Wrong shape: missing field, wrong type, word outside the enum (schema) | possible | possible | reduced | depends |
| Cut off by the output limit (syntax) | possible | possible | possible | possible |
| A refusal instead of the object (syntax or schema) | possible | possible | possible | possible |
| Contradicts itself: request_info with nothing missing (semantics) | possible | possible | possible | possible |
| Well shaped and false: a rule that does not apply, an old policy version (business validity) | possible | possible | possible | possible |
| A string that carries an instruction or markup (security) | possible | possible | possible | possible |
Read it by rows, not by columns. The best mode changes two rows from "possible" to "reduced" and leaves five untouched. The bottom three rows are the layers of four layers that need the record as a whole, the facts of the run or the place a string lands. No setting of the decoder reaches them.
The example's outputs, sorted by what a mode could have prevented#
The demonstration runs the first attempt of seven cases through the example's four validation stages. It shows where each one fails first. A successful run proves the ordering of the cases by stage; it does not prove that any provider would or would not produce these outputs. The model here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.
from model_sim import load_case
from validate import recommendation_ctx, run_stages
# Outputs that a schema-constrained mode is designed to prevent, and outputs it is not.
for cid in ["UW-S01", "UW-S03", "UW-S04", "UW-S08", "UW-S10", "UW-S12", "UW-S15"]:
case = load_case(cid)
ctx = recommendation_ctx(case["facts"], case["evidence"])
res = run_stages(case["attempts"][0], ctx)
first = res["first_failure"] or "-"
errs = [e for s in res["stages"] for e in s["errors"]]
e = errs[0] if errs else None
detail = ("passes all four stages" if e is None
else e["message"] if e["stage"] == "parse" else f'{e["path"]} {e["code"]}')
print(f"{cid} first failure {first:<12} {detail}")
Output of a real run in a scratch copy of examples/uw-contracts:
UW-S01 first failure parse unexpected character 'I' at position 0
UW-S03 first failure parse unexpected end of input at position 190
UW-S04 first failure schema /recommendation enum
UW-S08 first failure cross_field /missing X-MISSING-EMPTY
UW-S10 first failure domain /recommendation D-RULE-RESULT
UW-S12 first failure domain /policy_version D-POLICY-VERSION
UW-S15 first failure - passes all four stages
Sort the seven by the grid. UW-S01 and UW-S04 are what a supported structured output mode is designed to prevent. UW-S03 is a cut-off: "unexpected end of input" is the parser's word for truncation, and the documents below say it can still happen. UW-S08, UW-S10 and UW-S12 are schema-valid by construction; a constrained decoder would let each of them through unchanged. UW-S15 passes all four stages and still carries markup and an injected instruction in a reason, which is a security problem, not a shape problem.
What the providers' documents say today#
Each statement below is a product fact, read on 2026-11-17 from the provider's own documentation and logged in this site's fact file. Product documentation changes often; read the linked page again before you depend on a detail. Parameter names are quoted as the pages show them. No provider is ranked, and none is described beyond what its page says.
OpenAI#
The structured outputs guide says: "Structured Outputs constrains completed, non-refusal model output to your supplied JSON Schema." The two qualifiers in that sentence, completed and non-refusal, are the two documented exceptions. On JSON mode the guide is direct: "While both ensure valid JSON is produced, only Structured Outputs ensure schema adherence", and JSON mode requires an instruction to produce JSON: the API "will throw an error if the string "JSON" does not appear somewhere in the context".
The schema subset has rules of its own. The guide says "all fields or function parameters must be specified as required", that an optional value can be emulated "by using a union type with null", that additionalProperties: false is required, that the root must be an object and not use anyOf, and that allOf, not, if, then and else are among the keywords not supported. It lists pattern, format, minimum, maximum, minItems and maxItems among supported properties, and does not list minLength or maxLength for strings (it names them only as unsupported for fine-tuned models). It also states limits: "up to 5000 object properties total, with up to 10 levels of nesting". And it ends where this site would: "Your application still needs to handle refusals, incomplete responses, and API errors", and validate business rules the schema does not express.
For tools, the function calling guide says that setting strict to true "will ensure function calls reliably adhere to the function schema, instead of being best effort", recommends always enabling it, and notes that "Chat Completions requests remain non-strict by default".
Anthropic (Claude)#
The structured outputs page describes two features: JSON outputs, requested with output_config.format and type: "json_schema", and strict tool use, strict: true on a tool definition. It states the guarantee in strong words, "Structured outputs guarantee schema-compliant responses through constrained decoding", and then gives a section on invalid outputs. A refusal comes back with stop_reason: "refusal", and "The output might not match your schema because the refusal message takes precedence over schema constraints." A cut-off comes back with stop_reason: "max_tokens", and "The output might be incomplete and not match your schema."
The same section documents a third exception that matters directly for enums: "Structured outputs don't guarantee the capitalization of string enum and const values", and the response "completes normally, with no error and no special stop_reason". For this contract that means an output could say "Escalate" where the enum says "escalate", and only your validator would notice. The same page advises: "Compare enum values case-insensitively, and avoid enum values that differ only in capitalization." This site makes a different choice and says why: the schema stage compares exactly and a re-ask fixes the case, because a validator that silently lowercases is code changing what the model wrote (see repair), and the re-ask rate tells you how often it happens. Both designs work if every consumer does the same; the dangerous one is a mix, where one reader accepts "Escalate" and another rejects it.
The supported subset excludes, among others, "Numerical constraints (such as minimum, maximum, multipleOf)", "String constraints (minLength, maxLength)" and "Array constraints beyond minItems of 0 or 1", and additionalProperties must be false; an unsupported feature returns a 400 error, and the page says SDK helpers can transform schemas with unsupported constraints. It documents first-request latency "while the grammar compiles", a cache of compiled grammars, and explicit limits per request, for example 20 strict tools and 24 optional parameters across all strict schemas. The page also records a migration: the earlier output_format parameter "has moved to output_config.format" and is deprecated.
The strict tool use page says strict mode "guarantees Claude's tool inputs match your JSON Schema" and that the tool name "is always valid".
Google (Gemini)#
The structured output page read today says that Gemini models can be configured "to generate responses that adhere to a provided JSON Schema", that the mode "supports a subset of the" JSON Schema specification, and that "Not all JSON Schema features are supported." It is specific about which keywords the decoder does not enforce: "minLength and maxLength are not enforced during constrained decoding and are ignored by the server", while "minItems and maxItems are enforced". Its advice is the same as this site's: "While output is syntactically correct JSON, always validate values in your application", and "Implement robust error handling for schema-compliant but semantically incorrect outputs." The page read today did not mention truncation. Its address has also changed: the former ai.google.dev URL now redirects to a page under aistudio.google.com, and the request shape shown there differs from the one older code uses, so this page names no Gemini parameter.
vLLM (an open-source inference engine)#
Constrained decoding is not only a hosted product feature. vLLM, an open-source engine for serving models yourself, documents structured outputs with "xgrammar or guidance as backends", and request options for a JSON schema, a regular expression, a fixed list of choices or a context-free grammar: "the output will follow the JSON schema", "the output will be exactly one of the choices". It also warns that "Changing backends can change which tokens are allowed and therefore the generated output", and that older request fields such as guided_json "were removed in v0.12.0". Running your own engine moves the grammar compiler into your system; it does not move the business rules into the grammar.
What the four documents agree on#
| Question | OpenAI | Anthropic | vLLM | |
|---|---|---|---|---|
| Schema-constrained output documented | yes | yes | yes | yes |
| A subset of JSON Schema, not all of it | yes | yes | yes | depends on backend |
minLength and maxLength enforced in the read docs | not listed as supported | not supported | ignored by the server | not stated |
| Refusal can replace the object | yes | yes | not stated on the page read | not stated |
| Cut-off output can break the shape | yes | yes | not stated on the page read | not stated |
| Advises validating the output anyway | yes | not stated in those words | yes | not stated |
"Not stated" means the page read today does not say; it does not mean the behaviour is absent. The table is a reading record for 2026-11-17, not a comparison of quality.
Which parts of our contract survive the trip#
The recommendation contract uses more keywords than some providers enforce. The demonstration lists every keyword in recommendation.v1.json with the place it applies. It proves only what the schema contains; whether a provider enforces each keyword comes from the documents above, not from this run.
from validate import load_schema
def keywords(schema, path=""):
"""Every (JSON Pointer of the subschema, keyword) pair that constrains a value."""
out = []
for k, v in schema.items():
if k in ("description", "title", "$schema", "$id"):
continue
if k == "properties":
for name, sub in v.items():
out += keywords(sub, f"{path}/{name}")
elif k == "items":
out += keywords(v, f"{path}/items")
else:
out.append((path or "(root)", k))
return out
for where, kw in keywords(load_schema("recommendation.v1")):
print(f"{where:<26} {kw}")
(root) type
(root) additionalProperties
(root) required
/recommendation type
/recommendation enum
/reasons type
/reasons minItems
/reasons maxItems
/reasons/items type
/reasons/items minLength
/reasons/items maxLength
/citations type
/citations maxItems
/citations/items type
/citations/items pattern
/missing type
/missing maxItems
/missing/items type
/missing/items pattern
/stated_confidence type
/stated_confidence minimum
/stated_confidence maximum
/requires_human type
/policy_version type
/policy_version pattern
Hold that list against the documents. maxLength on each reason (400 characters) and minLength (1) are ignored by one provider and not supported by another as read today. minimum and maximum on stated_confidence are on Anthropic's unsupported list. maxItems on reasons, citations and missing falls under its "Array constraints beyond minItems of 0 or 1". So the same contract is enforced in full by your validator, and in part by any decoder, and the part differs by provider. A stated_confidence of 1.2 or a reason of 900 characters is a value the decoder may allow and the schema stage will reject.
In our insurance agent. The workflow keeps one full schema as the contract and validates every output with all four stages, whatever mode produced it. When it uses a schema-constrained mode, it treats the schema sent to the provider as a derived artifact, tested against the full one, and it expects the parse and schema stages to fail rarely. A rise in those failures is then a signal worth an alert: a cut-off, a refusal, or a provider change.
Truncation and refusal: the two exits every mode keeps#
Both exits come from the same place: the model stops writing the object before it is finished, for a reason the grammar does not control.
Truncation means the output ended because a limit was reached, not because the object was complete. The provider reports it in the response (a stop reason or a status), and the text is a prefix of an object. The repair page handles it: re-ask with a larger limit, and never let code "close the brackets", because a closed prefix of UW-S03 would be a schema-valid recommendation with a reason cut in half and no policy version.
A refusal means the model declined the task. The two providers that document it return it marked in the response, and one says the refusal "takes precedence over schema constraints". For the workflow a refusal is not a recommendation and not an error of the model's format; it is a reason to hand the case to a person with the refusal attached, through the bounded fallback the example already has.
Production pitfall. Checking only the text and never the stop reason misses both exits. A truncated output in a lax parser, or a refusal message in a field, can look like data. Read the provider's stop reason or status first, log it with the failure record, and routemax_tokensandrefusalto their own handling before any parse.
Tool arguments are a contract too#
When the underwriting agent calls get_roof_assessment, it writes the arguments, and those arguments are model output like any other. The tool runner in the example checks them against schemas/tool_inputs.json before the call; run_contracts.py --path shows one bad call, {"submission_id": "1004"}, rejected because it does not match ^SUB-[0-9]{4}$. A strict tool setting would have constrained the shape of that argument, and its format where the provider enforces pattern. No setting checks that SUB-1004 is the submission this run is about. Four contracts is the home of the tool input boundary.
What this is not. This page is about what the model's output is allowed to look like. Where tool schemas are published and how a tool call is executed belong to tools.sammuthu.com. How the instruction that asks for structure is written belongs to prompt.sammuthu.com. The runtime that sends the request and reads the stop reason is the harness, at harness.sammuthu.com.
For a coding agent. A coding agent that callsrun_tests(path)writes{"path": "tests/test_roof_rules.py"}. A strict tool setting makes surepathis a string. It does not make sure the path exists, stays inside the repository, or is not../../etc. The runner checks those before it runs anything; the CI side of that story is on diff-to-deploy.sammuthu.com.
Constraining the output also changes itA grammar removes choices, and the model then writes with what is left
A grammar removes options at each step; it does not make the model want the remaining ones. If the model's best continuation is forbidden, it writes the best allowed one, which can be a value that is well formed and less accurate than what it would have written in prose. Field order matters for the same reason: a model that must write recommendation before reasons commits to a decision before it writes its argument. One provider documents that required properties are output before optional ones. Whether to put reasons first is a design question for schema design, and the answer should be measured on your own cases, which belongs to eval.sammuthu.com.
Two schemas: the contract and the one you sendDerive the provider schema from the contract, and test the difference
Each provider accepts its own subset, so a full contract often cannot be sent as is. The usual design keeps the full JSON Schema as the contract the workflow validates against, and generates a provider schema from it by dropping or rewriting what that provider does not support. The keywords dropped are exactly the checks that move from the decoder to your validator, so list them, per provider, in code. A test then sends the golden examples of testing through both schemas and asserts that every example the provider schema accepts but the contract rejects is one you expected. Without it, a provider schema that quietly lost additionalProperties: false would let UW-S07's premium_usd through the decoder, and you would learn it from the cockpit.
Portability across providersThe concept is shared; the request shape, subset and stop signals are not
All four documents read here describe the same concept. They differ in where the schema goes in the request, which keywords are enforced, how refusal and truncation are reported, and how a changed schema affects caching and latency. A workflow that may switch providers keeps a thin adapter per provider that sends the derived schema and maps the provider's stop signals to three outcomes of its own: complete, truncated, refused. Everything after that adapter (the four stages, repair, the record) stays provider-independent. A library that hides the provider behind one call can do the mapping for you; check which outcomes it exposes, because one that turns a refusal into an exception and a truncation into a parse error has merged two different repairs.
Constrained decoding on your own engineMore control, and the grammar compiler becomes your code to run
Serving a model yourself with an engine that supports grammars gives options a hosted feature may not: a regular expression for one field, a fixed list of choices, a custom grammar. It also makes you the owner of the backend choice, its version and its behaviour; the vLLM documentation warns that changing backends can change which tokens are allowed. Treat a backend change as a contract-relevant release: rerun the golden examples and watch the parse and schema failure rates of the observability page.
Checklist#
- Use a schema-constrained mode where the provider supports your schema, and keep all four validation stages on every output.
- Read the stop reason or status before parsing; handle truncation and refusal as their own outcomes.
- List, per provider, the keywords of your contract it does not enforce; those checks live in your validator.
- Decide once how enum values are compared (this site: exactly, with a re-ask on a case mismatch) and make every consumer compare the same way, even when a provider constrains them.
- Keep the full schema as the contract and test the provider schema derived from it.
- Set strict mode on tools whose arguments reach real systems, and still validate the arguments before the call.
- Re-read the provider's documentation before an upgrade; record the date you read it.
In one breath#
There are four ways to ask a model for structure: ask in the prompt, ask for JSON, constrain the output to a schema, or have it fill a tool's arguments. Constrained decoding removes tokens that would break the grammar, so it can stop prose and words outside an enum, but the providers themselves document that a cut-off, a refusal, or an unsupported keyword can still break the shape, and no decoder knows whether a well-shaped value is true. Use the strongest mode you have, read the stop reason, and validate every output as if no mode existed.
The workflow now knows what each provider promises and what it leaves to the workflow. The final response is only one of the places where a model writes data, though. The same agent writes tool arguments, reads tool results it did not write, and hands work to another agent. Four contracts walks SUB-1004 across all four boundaries and asks, at each one, who produces the payload and who must check it.