A colleague tidies the recommendation schema. The enum on recommendation looks redundant next to the domain stage ("the rule check catches wrong words anyway"), additionalProperties: false "makes evolution harder", and the range on stated_confidence "is obvious". They delete the three lines. All unit tests of the workflow pass: they test the happy path, and the happy path still works. A week later the underwriter cockpit shows a recommendation with an extra field premium_usd, and a stated confidence of 62.
Nothing told anyone that the schema had become weaker, because nothing tested the schema. The workflow's tests used the schema; they never asked what it rejects. This page treats the contract as what it is: code that makes promises, and needs tests of those promises.
All runs on this page use the scripted stand-in model and the generator in examples/uw-contracts/; the counts show how the mechanism works, not how good any model is.
Golden examples: must pass, must fail#
Plain English. A golden example is a fixed input with a fixed expected result, kept with the code and checked on every change. For example: the valid recommendation of UW-004 must pass all four stages; the same recommendation with "Escalate" (capital E) must fail at the schema stage.
Plain English. A negative example is a golden example that must fail, and it states where. For example: "trailing comma: must fail at parse". Negative examples are the ones that catch a weakened schema. A schema that accepts everything passes every positive test.
The example keeps them in examples/uw-contracts/golden/: 3 that must pass and 14 that must fail, each with the stage it must fail at. A few of the negatives:
| Golden example | Must fail at | What it guards |
|---|---|---|
trailing-comma | parse | the strict parser |
nan-confidence | parse | NaN is not JSON, even if some parsers accept it |
duplicate-key | parse | a second recommendation key must not silently win |
enum-case ("Escalate") | schema | the enum, exactly |
confidence-percent (62) | schema | the range of stated_confidence |
citation-lowercase | schema | the pattern of rule ids |
escalate-no-citation | cross_field | a cross-field rule |
approve-against-rules | domain | the rule check |
human-false | domain | R-HUMAN-01 |
Each row names a line of the contract that someone could delete. If the colleague's change had run against these, confidence-percent would have been accepted, and the test would have said so before the change was merged. The real run in a scratch copy:
golden: 17 of 17 as expected
The "at a named stage" part matters. A negative example that only says "must fail" still passes when the failure moves from the schema stage to the domain stage, and that move is a real change: the error message, the repair and the cost are different. With the enum deleted, enum-case would still fail, but at the domain stage, with a misleading error about the rule check.
Before AI. A Spring team keeps request fixtures for its controller tests: a valid JSON body that must return 200 and invalid bodies that must return 400 with a given field error. Golden examples are the same idea, applied to output instead of input, and checked stage by stage instead of by status code.
Property-based tests: what is true of every valid output#
Golden examples are cases someone thought of. The next step is to state a rule that must hold for every case and let a program generate cases.
Plain English. A property-based test states a property that must hold for all inputs of some kind, generates many inputs, and checks the property on each. For example: "every recommendation built correctly from the rules of a case passes all four stages", checked on 200 generated recommendations.
The example's generator, generate(seed) in gen.py, picks one of five shared cases (UW-001, UW-002, UW-003, UW-004, UW-007), builds the right recommendation for it under policy 2026.2, and varies what may vary: how many reasons, the stated confidence. It is seeded: the same seed always gives the same payload, so a failure can be reproduced from one number.
property: generated payloads passing: 200 of 200
seed 7: UW-001 {"recommendation":"decline","reasons":["The provider confirms a roof of 31 years in wind zone C."],"citations" ...
That property protects against the opposite mistake from the colleague's: a schema made stricter by accident. If someone added minItems: 2 to reasons, generated outputs with one reason would start failing, and the property would name the seed.
A second useful property is the round trip. Plain English. A round trip writes a value out and reads it back, and checks that nothing changed. For example: serialize a recommendation record to JSON, parse it with the strict parser, and compare. The Pydantic and Java comparison of the typed models page is full of round trips that fail quietly: a null that comes back as 0.0, a number that comes back as a string.
Fuzzing: hostile inputs on purpose#
Plain English. Fuzzing feeds a program large numbers of deliberately broken inputs and checks that it handles each one correctly. For example: take generated valid recommendations and break each in one named way (drop a field, change a type, cut the text), then check that the stages reject every one of them.
The example has 14 named mutations. Each one is applied to 60 generated payloads, and the stages are run with the real schema and with a deliberately weakened one: the same schema with only types and required fields left (no enums, no patterns, no ranges, open objects). The colleague's tidy-up, made extreme. The real run:
mutation real schema: caught at weak schema: accepted
none accepted 60 60
drop_required schema 60 0
wrong_type schema 60 0
enum_near_miss schema 60 0
extra_field schema 60 60
null_field schema 60 0
out_of_range schema 60 60
empty_list schema 35, cross_field 22, accepted 3 38
unknown_rule domain 60 0
flip_human domain 60 0
old_policy domain 60 0
truncate parse 60 0
fence parse 60 0
duplicate_key parse 60 0
Three things stand out.
The weak schema lets two whole mutations through. Every payload with an extra field and every payload with an out-of-range confidence reaches the cockpit. That is exactly the week of the colleague's change, reproduced in a second.
Some failures just move. enum_near_miss shows 0 accepted under the weak schema, so it looks fine. It is not: the near misses now fail at the domain stage, because the rule check disagrees with a word it does not know. The error is less clear, and a word like "refer" that happened to match the rule outcome would get through. A table of "accepted" alone hides this; keep the stage counts for both schemas.
The real contract has a gap, and the fuzzer found it. Under empty_list, 3 payloads pass every stage with the real schema. All three are request_info with an empty citations list. The contract never said that request_info must cite a rule; the cross-field stage checks citations only for decline and escalate. Is that a bug? It is a decision nobody made. This site keeps the gap on purpose, with a test that documents it, so that the decision is visible: a team should either add the rule (R-ROOF-02 must be cited when the roof age is missing) or write down why not.
Diagram: What a weakened schema does to fuzzed outputs · some get through, some only move
Read left to right. The same mutated payloads go through the real schema and a schema with only types and required keys. Some mutations now reach the cockpit; others are still caught, but later and with a less clear error.
- Generated, then broken
- Valid payloads: Built from the rules of five shared cases; all 200 pass every stage.
- One mutation: Each payload is broken in one named way.
- Real schema
- Caught: Every mutation is rejected, each at its own stage, except one gap.
- Weakened schema
- Gets through: Extra fields and out-of-range numbers pass every stage.
- Caught later: Near-miss words now fail at the domain stage instead of the schema.
- A finding
- A gap in the contract: request_info with no citation passes every stage of the real contract.
Shrinking is the last piece of the method. When a fuzzer finds a failure, the smallest input that still fails is the most useful one to read. The lab below shows the smallest payload the weak schema lets through and the real one rejects. Property-based testing libraries do this automatically; the documentation of Hypothesis, a property-based testing library for Python read on 2026-11-17, says it lets the tool "randomly choose which of those inputs to check - including edge cases you might not have thought about". The example's generator is far simpler and does not shrink: it is standard-library code you can read in a few minutes.
Try it: fuzz the contract#
Pick a mutation, a seed and a count. The lab regenerates the same payloads as the Python code (the generator is the same in both, checked row by row) and runs them through the real and the weakened schema. Try extra_field, out_of_range, then empty_list.
| Mutation | Real schema catches it at | Weak schema lets through (of 60, seed 1) |
|---|---|---|
extra_field | schema | 60 |
out_of_range | schema | 60 |
enum_near_miss | schema | 0 (caught later, at domain) |
empty_list | schema or cross_field; 3 pass | 38 |
unknown_rule, flip_human, old_policy | domain | 0 |
truncate, fence, duplicate_key | parse | 0 |
Contract suites: both sides run the same tests#
The tests above live with the validator. But a contract has two sides, and each side can break it without touching the other's code.
Plain English. A contract test is a test that checks one side of a boundary against the shared contract, not against the other side's code. For example: the cockpit team checks that its reader accepts every golden example that must pass, and the agent team checks that its outputs pass the same validator.
The idea is old in service integration. The documentation of Pact, a contract testing tool read on 2026-11-17, calls itself "a code-first consumer-driven contract testing tool" and describes a contract as a collection of examples, each covering "a single concrete request/response pair", generated from the consumer's tests. For a model's output there is a twist: the producer is not code you can call in a test. So the contract suite has three parts:
| Part | Run by | Checks |
|---|---|---|
| Golden examples and the fuzz suite | the owners of the contract, on every schema change | the schema rejects what it must and accepts what it must |
| Producer suite: recorded outputs of the agent | the agent team, on every change of instructions or model setting | the producer still writes outputs that pass, at the same first-attempt rate |
| Consumer suite: the golden "must pass" examples and old stored records | each consumer team (the cockpit, the audit export) | the consumer reads every valid shape, including old versions |
The producer suite uses the replay artifacts of the observability page: real outputs of the current producer, replayed through the validator. It is the only way to test a producer whose output is not deterministic: compare rates over a set, not single outputs.
In our insurance agent. A change of the underwriting agent's instructions runs the producer suite on a fixed set of submissions before release, and the first-attempt failure rate by field is compared with the last release. The stated_confidence drift of day 18 in the observability page would have shown up there, before any underwriter saw a delay.
Diagram: One contract, three suites · each side of the boundary runs its own part
Read left to right. The contract owners keep golden examples and fuzzing. The agent team runs recorded outputs and compares rates. Each consumer reads every valid shape, including old records. All three use the same schemas.
- The contract
- Schemas and validator: recommendation.v1, v2 and the four stages, versioned together.
- Contract owners
- Golden and fuzz: Must pass, must fail at a stage, named mutations.
- Producer (the agent team)
- Recorded outputs: Replay a fixed set; compare first-attempt failure rates by field.
- Consumers
- Read every valid shape: Golden must-pass examples and old stored records, all versions.
What this is not. Whether the agent's recommendations are right (graders, golden datasets of labelled cases, judges, release gates) is evaluation, owned by eval.sammuthu.com. A contract test asks whether outputs have the agreed form and pass the agreed checks. The two share tools and must not share a name.
For a coding agent. Thechange_reportschema of a coding agent gets the same three suites: golden reports (one withrisk: "moderate"that must fail), a producer suite of recorded reports from the agent, and a consumer suite that CI runs. Running these in CI is described on diff-to-deploy.sammuthu.com.
Checklist#
- Keep golden examples with the schema: must-pass and must-fail, each must-fail with the stage it fails at.
- Write one negative example per constraint line of the schema, so deleting the line fails a test.
- Generate valid outputs from the rules with a seed, and check that all of them pass (catches a schema made stricter by accident).
- Fuzz with named mutations, against the real schema and a weakened copy, and keep the stage counts for both.
- Treat every output that passes the fuzzer unexpectedly as a decision to make and document.
- Run a producer suite (recorded outputs, rates by field) on every change of instructions or model setting.
- Run a consumer suite (golden examples plus old stored records) in every consumer's CI.
Checking schema changes in CIA diff of two schemas, classified by direction
A schema change can be classified mechanically before any test runs: added required properties break backward compatibility, added properties on a closed object break forward compatibility, removed enum values break backward compatibility. schema_changes() in evolve.py does this for the cases this site teaches. In CI, run it on every pull request that touches a schema and require a note for every breaking direction. The evolution page explains the directions.
Generating from the schema itselfUseful for shape, blind to meaning
Some tools generate test data directly from a JSON Schema. That covers every type and enum branch cheaply, and it is a good fuzz source for the schema stage. It cannot generate outputs that are valid in the domain (a decline that cites the rule that really applies), because the schema does not know the rules. That is why this site's generator works from the rules of a case, and the fuzzer breaks valid outputs, rather than the reverse.
Testing a producer that is not deterministicRates over a fixed set, with a margin
The same submission can produce different outputs from one run to the next. A producer test that asserts one exact output is flaky by construction. Assert rates over a fixed set instead (first-attempt failure rate by field at most the last release's plus a margin), and decide the margin from repeated runs. How many runs and what margin is a statistical question, covered on eval.sammuthu.com.
In one breath#
A schema is code that makes promises, so test the promises: golden examples that must pass and must fail at a named stage catch a schema made weaker, generated valid outputs catch a schema made stricter, and named mutations against the real and a weakened schema show what each line of the contract is worth and where the contract has gaps nobody decided on. Then make both sides run the suite: the producer with recorded outputs and rates, each consumer with golden examples and old records.
You can now build a contract, check it, repair it, evolve it, secure it, watch it and test it. The next page steps back and asks which standards and libraries do which of these jobs for you: frameworks and standards.