Case UW-S10 is one of this site's own cases (the UW-S cases are scripted outputs, built on the facts of the series' shared cases). The facts come from UW-002: a masonry warehouse, a roof of 8 years that the roof-data provider confirms, wind zone A, sprinklers. The underwriting agent writes this:
{"recommendation": "decline", "reasons": ["The roof is too old for wind zone rules under R-ROOF-01."], "citations": ["R-ROOF-01"], "missing": [], "stated_confidence": 0.66, "requires_human": true, "policy_version": "2026.2"}
It parses. Every field has the right type. decline is one of the four allowed words. R-ROOF-01 matches the pattern of a rule id. A decline cites a rule, as a decline must. Each check that looks at the text alone says yes. And the recommendation is false: R-ROOF-01 is about roofs older than 25 years in wind zone C, and this roof is 8 years old in zone A. If the workflow stops checking after the schema, this output reaches the underwriter cockpit as a well-formed reason to turn away a clean risk.
The model recommends; a person decides. The underwriter would probably catch it. But the point of a contract is that the underwriter should not have to re-check every field by hand. This page is about the checks between "the model returned something" and "the underwriter sees it": what each one asks, in which order they run, and what an error looks like when one says no.
Four questions about one output#
Plain English. Validation means checking a value and getting back a list of what is wrong with it. An empty list means "no problem found by this check". It never means "correct". For example: checking that stated_confidence is a number between 0 and 1 returns an empty list for 0.66, and that says nothing about whether 0.66 means anything.
The four layers of four kinds of right become four pieces of code here. Each piece is called a stage: one step of validation, with its own input, its own question and its own errors. The code names are parse, schema, cross_field and domain.
| Stage | Question | Reads | Needs to know | Fails on, in our cases |
|---|---|---|---|---|
parse | Is this text JSON at all? | raw text | the JSON grammar | UW-S01 prose, UW-S03 cut off |
schema | Does the value have the agreed shape? | parsed value | the JSON Schema of the contract | UW-S04 "approved", UW-S06 "high" |
cross_field | Does the record agree with itself? | a schema-valid value | rules that tie fields together | UW-S08 request_info with nothing missing |
domain | Is it true for this run? | a self-consistent value | the policy fixture, the facts, the evidence of this run | UW-S10 a rule that does not apply |
Read the "Needs to know" column from top to bottom. Each stage needs more knowledge than the one before. The parser needs only the grammar of JSON. The schema stage needs the contract. The cross-field stage needs rules that the contract's authors wrote about combinations of fields. The domain stage needs the world: which version of the policy fixture this run read, what the roof-data provider returned, which evidence was gathered. That is why the domain stage is the only one that can catch UW-S10. The text is fine; the world disagrees with it.
Common confusion. Layer 3, semantics, is wider than the cross-field stage. Semantics also covers units (square metres in a square-foot field) and the difference betweennulland0. Some of that code can check from the record alone. Some of it needs the source the value came from, and in this example that check lives in the domain stage of the intake contract (UW-S17,UW-S18). The stage names say where code runs; the layer names say what kind of right is being checked.
Diagram: Four stages, one output · each needs more knowledge than the one before
Read left to right. The raw text of the model enters on the left. Each stage either hands the value on or stops with errors. Only an output that passes all four is wrapped in a record by code; the underwriter still decides.
- In
- Raw output: Whatever the model returned: JSON, prose, a fence, half an object.
- 1 · parse
- Parse: Turns text into a value or stops with the position of the first bad character.
- 2 · schema
- Schema: Checks the shape: required fields, types, enums, patterns, a closed object.
- 3 · cross_field
- Cross-field: Checks that fields agree with each other.
- 4 · domain
- Domain: Checks the output against the policy fixture, the facts and the evidence of this run.
- Out
- Recommendation record: Code wraps the accepted output with stamped identifiers. A person decides.
- Error records: Any stage that says no stops the line and hands its errors to the repair step.
The diagram shows the normal path. One raw output enters on the left. Each stage either passes the value on or stops with errors. Only an output that passes all four becomes a recommendation record, and even then the underwriter decides.
The smallest version: four functions in a row#
The smallest primitive. A stage is a function from a value to a list of errors. Four of them in a row, stopping at the first that returns errors, is the whole mechanism. In examples/uw-contracts/validate.py the function is run_stages(text, ctx). The ctx (the context) carries what the later stages need to know: the schema, the policy version, the rules, the facts and the evidence ids of this run.
The demonstration below runs UW-S10 through the four stages. What a successful run proves: the first three stages pass, and the domain stage alone rejects the output, with two errors.
import json
from model_sim import load_case
from validate import recommendation_ctx, run_stages
case = load_case("UW-S10") # decline citing R-ROOF-01 for a roof of 8 years
ctx = recommendation_ctx(case["facts"], case["evidence"])
res = run_stages(case["attempts"][0], ctx) # stop at the first stage that fails
print("first failure:", res["first_failure"])
for s in res["stages"]:
print(f" {s['stage']:<12} {s['status']}")
for e in res["stages"][3]["errors"]:
print(json.dumps(e))
Output of a real run, in a scratch copy of the example (the model here is a scripted stand-in, so these results show how the mechanism works, not how good any model is):
first failure: domain
parse pass
schema pass
cross_field pass
domain fail
{"stage": "domain", "path": "/recommendation", "code": "D-RULE-RESULT", "message": "the rule check gives approve (no rule triggered); the output says decline"}
{"stage": "domain", "path": "/citations/0", "code": "D-CITATION-UNSUPPORTED", "message": "R-ROOF-01 does not apply to these facts"}
The domain stage does not ask the model whether it is right. It runs the same deterministic rule check the workflow runs anyway, eligibility() in validate.py, and compares. The rule check says approve, because no rule fires for these facts. The output says decline. One more design choice is visible in the code: a mismatch is allowed when the model recommends escalate, because sending a case to a senior underwriter is always a safe thing to recommend. That is a decision of this example, written down where the check is. Your domain may decide differently.
In our insurance agent. The rule check exists before the model does: it is ordinary code over the policy fixture. The model adds reasons a person can read and a judgement where the rules leave room. The domain stage keeps the two from disagreeing silently.
The error record#
Plain English. A validation error is a small record that says which stage failed, where in the value, which check, and why in words. For example: {"stage": "schema", "path": "/recommendation", "code": "enum", "message": "\"approved\" is not one of ..."}.
The path is a JSON Pointer. A JSON Pointer is a short string that names one value inside a JSON document: /citations/0 is the first item of the citations array, and the empty string "" is the whole document. It is defined by RFC 6901, which says a pointer "defines a string syntax for identifying a specific value within a JavaScript Object Notation (JSON) document". Two characters need escaping inside a key: ~ is written ~0 and / is written ~1.
Each part of the record has one reader in mind:
| Field | For whom | Why it is there |
|---|---|---|
stage | the repair loop, the dashboards | Decides the next action: a parse error and a domain error call for different repairs (next page) |
path | code and people | Points at the value without quoting it; a dashboard can count failures per field |
code | code | A stable id (enum, X-MISSING-EMPTY, D-RULE-RESULT, R-HUMAN-01) to count, alert on and test against; messages may be reworded, codes may not |
message | the model on a re-ask, the underwriter on a fallback | Says what is wrong in words that can be acted on |
Before AI. A Spring service that validates a request body with Bean Validation returns a 400 with a list of field errors: the property path, the constraint, the message. The idea is the same. What changes with a model as the producer is the reader of the message: it is fed back to the model as the next instruction, so it must be precise, short and free of anything you would not want repeated back.
Look at the codes once more. Schema errors use the JSON Schema keyword (enum, required, type), because the validator produces them. Cross-field errors start with X-, domain errors with D-, and a check that enforces a rule of the policy fixture uses the rule id itself, R-HUMAN-01. A team that reads a failure dashboard in production can tell at a glance whether the producer broke the shape or the truth.
Why the order is not a matter of taste#
The obvious design is to run every check and report everything at once. It sounds more helpful: one round trip, all the problems. It is how a form on a web page often behaves.
Here is what happens when the checks run on a value the schema has rejected. The output says request_info, and missing is the string "roof_age_years" instead of a list. The demonstration runs the same text twice, once stopping at the first failing stage and once running every stage:
import json
from validate import recommendation_ctx, run_stages
from model_sim import load_case
case = load_case("UW-S01")
ctx = recommendation_ctx(case["facts"], case["evidence"])
good = json.loads(case["attempts"][-1])
bad = json.dumps(dict(good, missing="roof_age_years", recommendation="request_info"))
for stop in (True, False):
res = run_stages(bad, ctx, stop_at_first=stop)
print("stop_at_first =", stop, [(s["stage"], s["status"]) for s in res["stages"]])
for s in res["stages"]:
for e in s["errors"]:
print(" ", e["stage"], e["path"], e["code"], "-", e["message"])
stop_at_first = True [('parse', 'pass'), ('schema', 'fail'), ('cross_field', 'not_run'), ('domain', 'not_run')]
schema /missing type - expected array, got string
stop_at_first = False [('parse', 'pass'), ('schema', 'fail'), ('cross_field', 'pass'), ('domain', 'fail')]
schema /missing type - expected array, got string
domain /recommendation D-RULE-RESULT - the rule check gives escalate (R-ROOF-03); the output says request_info
Look at the second run. The cross-field stage says pass. Its check is "request_info must name at least one missing field", written as "the list missing is not empty". A non-empty string is not empty either, so the check passes on a value it was never written for. Nothing crashed. The check lied.
That is the reason for the order. Each stage is written assuming the stage before it passed. The cross-field check assumes missing is a list because the schema said so. The domain check assumes citations holds strings that look like rule ids. Run a later stage on a value the earlier stage rejected, and it either crashes (the explorer lab below marks that as crashed) or, worse, returns an answer about a value it does not understand.
Inside one stage the rule is the opposite: collect every error. The schema stage reports a wrong enum and a missing field together, because both are true at once and one re-ask can fix both.
Diagram: Why stop at the first failing stage · a later check on a wrong type does not crash, it lies
Top lane: the schema rejects missing as a string and the line stops; one true error reaches the repair step. Bottom lane: the same value runs through every stage; the cross-field check passes a value it was never written for, and the domain stage reports a misleading error.
- Stop at first
- missing is a string: The output says request_info and missing is "roof_age_years", not a list.
- schema: fail: expected array, got string.
- Stop: The re-ask carries only the type error, which is what must be fixed.
- Run everything
- Same output: The same value is checked by every stage regardless.
- cross_field: pass: "missing is not empty" is true of a non-empty string, so the check passes on a value it was never written for.
- domain: fail: The rule check says escalate; the error points at the recommendation, not at the type.
Production pitfall. A validator that "helpfully" reports domain errors on a schema-invalid value produces messages like "the rule check gives escalate; the output says request_info" when the real problem is a type. A re-ask built from that message sends the model after the wrong fix. Report the earliest failing stage only.
Try it: switch stages off#
The lab below takes the outputs of UW-S04 to UW-S15. All of them parse; that is why they were chosen. Switch a stage off and see which stage catches the output next, or whether it reaches the underwriter. Two cases are worth trying first. UW-S10 with the domain stage off reaches the cockpit as a confident decline. UW-S15 passes every stage with everything on, and still carries markup and an injected instruction in a reason: a schema checks shape, not truth, and not what a string does where it lands.
| Case | The output, in a few words | Caught at | Error code |
|---|---|---|---|
UW-S04 | "recommendation": "approved" | schema | enum |
UW-S05 | policy_version absent | schema | required |
UW-S06 | "stated_confidence": "high" | schema | type |
UW-S07 | extra field premium_usd | schema | additionalProperties |
UW-S08 | request_info, nothing missing | cross_field | X-MISSING-EMPTY |
UW-S09 | decline, no citation | cross_field | X-CITATIONS-EMPTY |
UW-S10 | decline under R-ROOF-01, roof 8 years | domain | D-RULE-RESULT, D-CITATION-UNSUPPORTED |
UW-S11 | cites R-ROOF-09 | domain | D-RULE-UNKNOWN |
UW-S12 | policy_version 2025.4 | domain | D-POLICY-VERSION |
UW-S13 | requires_human false | domain | R-HUMAN-01 |
UW-S14 | evidence id never gathered | domain | D-EVIDENCE-UNKNOWN |
UW-S15 | markup and an instruction in a reason | none | (flagged untrusted by code) |
Where each stage runs, and who owns it#
A stage is code, and code has an owner. In this example all four stages run in the workflow, the code around the agents, right after the model returns and before anything is stored. Each stage's knowledge has a different owner, though:
| Stage | Owner of the check | Changes when |
|---|---|---|
parse | nobody: the JSON grammar | never |
schema | the owners of the contract (producer and consumer agree) | the contract gets a new version |
cross_field | the owners of the contract | someone finds a contradiction the schema cannot express |
domain | the owners of the rules: here, whoever maintains the policy fixture | the policy fixture changes (2025.4 to 2026.2), or new evidence types appear |
This split is useful when something fails in production. A rise in schema failures is a conversation with whoever changed the producer (the prompt, the model setting, the provider). A rise in domain failures after a policy release is a conversation with the rule owners: the model may be right about the old rules.
For a coding agent. A coding agent that reports its work as achange_report(files_changed,tests_run,tests_passed,risk,needs_review) can be checked the same way. Schema:riskis one oflow,medium,high. Cross-field:tests_passedis not greater thantests_run. Domain: every path infiles_changedappears in the actual diff of the branch, which only CI can see. The review and CI side of that story lives on diff-to-deploy.sammuthu.com.
What this is not. These stages check one output against its contract. Checks around the whole agent (what it may call, what it may see, what a person must approve) are guardrails, owned by guardrails.sammuthu.com. Measuring how often outputs are right over many runs is evaluation, owned by eval.sammuthu.com.
How libraries and standards do it#
The pattern is the same everywhere; the split between stages differs.
JSON Schema (a standard). The 2020-12 core specification defines output formats for validation results. Each result carries a keywordLocation and an instanceLocation, both JSON Pointers: "the location of the JSON value within the instance being validated". It names four formats, flag (a single boolean), basic (a flat list), detailed and verbose (hierarchies following the schema), and says an implementation "SHOULD provide at least one of the "flag", "basic", or "detailed" format". The error record on this page is close to the basic shape with the stage added. Read the details in the specification itself; this site's validator does not claim to implement those formats.
Pydantic (a library) and the Java pair, Jackson with Bean Validation (a library and a standard). The recorded run in examples/uw-contracts/twins/ (Pydantic 2.14.3; Jackson 2.23.0 and Hibernate Validator 9.1.6.Final, run on 2026-11-17) shows the two shapes of the same idea. Pydantic parses, converts and checks constraints in one call and returns one list of errors. The Java path has two steps: Jackson stops at the first problem it meets while reading, and Bean Validation runs only if reading succeeded. So a range error is never reported together with a type error. Neither library knows your policy fixture: the domain stage is always your code. The typed models page compares the two in detail.
Checklist#
- Name the stages in code and in every error record; never report a bare "invalid".
- Stop at the first failing stage; collect every error inside a stage.
- Give every check a stable code, separate from its message; test against the code.
- Use JSON Pointers for paths, so dashboards can count failures per field.
- Put rule knowledge in the domain stage, reading the same policy version the run read.
- Check evidence ids and policy versions against what this run really fetched, not against a pattern.
- Write messages that are safe to send back to the model: short, specific, no secrets, no quoted untrusted text.
Short-circuit by field, not only by stageRunning later checks on the fields that passed
Stopping at the first failing stage is the simple rule. A finer rule runs later stages only on the fields the schema accepted, so that one bad field does not hide a domain error on another. It is more work to get right: every cross-field and domain check must declare which fields it reads, and must be skipped when any of them failed. The gain is one fewer re-ask in cases with two independent problems. Measure how often that happens before you build it; in most contracts with seven to ten fields, the simple rule costs little.
What each stage costsMicroseconds, then lookups, then sometimes a network call
Parse and schema checks are local and cheap. Cross-field checks are local too. Domain checks may need data: the policy fixture of the right version, the facts of the submission, the list of evidence ids gathered in this run. Keep that data in the run's context instead of fetching it again inside the validator, both for speed and so that a replay sees exactly what the original validation saw. A domain check that calls a remote service at validation time makes validation itself a source of timeouts, which then need their own handling.
Validate on both sides of a boundary?Producer-side checks are a courtesy; consumer-side checks are the contract
When a model produces the output, the producer cannot validate itself in any binding way: the workflow does it. When code produces a payload for another service, it is common to validate before sending (to fail early with a clear message) and again on receipt (because the receiver cannot trust the sender's version of the schema). The receiving check is the one that protects the system. The boundaries page applies this to all four contracts on the path.
In one breath#
Validation is four questions in a fixed order: is it JSON, does it have the agreed shape, does it agree with itself, is it true for this run. Each stage needs more knowledge than the one before, and each is written assuming the one before passed, so you stop at the first failure and collect everything inside it. The error record says stage, path, code and message, so the repair loop, the dashboards and the person reading it all know what to do next. Only the domain stage knows the world, and only your code can write it.
You can now tell, for any failing output, which stage should catch it and what the error should say. What is still open is what to do with that error: fix it in code, send it back to the model, ask a person, or stop. That is the next page, repair.