On day 18 of a month (in this site's synthetic log), nothing in the underwriting workflow was released. The contract did not change, the validator did not change, the policy fixture did not change. But the share of first attempts that failed validation went from about 4 in 100 to about 16 in 100, and stayed there. The repair loop of the repair page caught almost all of it: a re-ask, attempt 2 valid, a recommendation in the cockpit. The underwriters noticed nothing except that some recommendations took a little longer. The bill for model calls noticed.
What changed was the producer. Someone updated the instructions and a model setting of the underwriting agent, and the new producer, now and then, wrote "stated_confidence": "high" instead of a number. Each such output failed at the schema stage, was re-asked and fixed. The repair loop hid the problem. Only the failure records showed it.
This page is about those records: what a validation failure should leave behind, how to count failures so that a change shows up, and what to keep so that any failure can be examined again later. The numbers on this page come from a seeded, synthetic log; they show how the mechanism works, not how any real system behaves.
A failure is a record, not a log line#
Plain English. A failure record is one row of data per validation error, written at the moment the error happens, with the identifiers that tie it to everything else. For example: "run run-7f10, attempt 2, schema stage, path /recommendation, code enum, action re-ask".
The obvious alternative is a log line: WARN validation failed: "approve/escalate" is not one of .... It is readable, and it cannot be counted by field, by stage or by version without parsing the message, which someone will reword next quarter. The error record of the validation page already has the right parts: stage, path, code, message. A failure record adds the identifiers.
Here are the three failure records of case UW-S16 (this site's own scripted case: "approve or escalate", ambiguous three times), from a real run of failure_records() in examples/uw-contracts/observe.py:
{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "e92802de838ccaec", "attempt_number": 1, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "re_ask"}
{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "55eb02eb35f3188d", "attempt_number": 2, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "re_ask"}
{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "66c729ed08628ea3", "attempt_number": 3, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "fallback"}
Read the identifiers from left to right. They are different things, and the series keeps them apart:
| Identifier | Means | Same across the three records? |
|---|---|---|
submission_id (SUB-1004) | one quote request | yes (carried in the full record) |
run_id | one execution of the workflow for that submission | yes: one run |
trace_id | 32 hex characters; on this site one trace per run | yes |
span_id | 16 hex characters; one step inside the trace, here one attempt | no: three attempts, three spans |
attempt_number | which try inside the run | no: 1, 2, 3 |
The sizes are not invented here. The W3C Trace Context recommendation (Level 1, 2021) defines a trace id as "a 16-byte array, for example, 4bf92f3577b34da6a3ce929d0e0e4736" (32 hex characters) and a parent id as "an 8-byte array" (16 hex characters), and says an all-zero value is invalid. The example derives both from the run id with a hash so that the runs are reproducible; real systems take them from their tracing library. What traces and spans are, and how to collect them, is the subject of observe.sammuthu.com. This page only uses the identifiers to join failure records to the rest.
Diagram: One run, three attempts, three failure records · the identifiers that tie them together
Read left to right. One submission gets one run and, on this site, one trace. Each attempt is a span with its attempt number. Each validation error becomes a failure record carrying all of these, so it can be joined to everything else.
- The request
- Submission: One quote request from a broker.
- One run, one trace
- Run: One execution of the workflow for the submission.
- Trace: On this site one trace per run.
- Three attempts, three spans
- Attempt 1: schema /recommendation enum, re-ask.
- Attempt 2: schema /recommendation enum, re-ask.
- Attempt 3: schema /recommendation enum, fallback.
- Kept
- Failure records: Counted by stage, field, code, schema and producer version.
- Replay artifacts: Enough to validate the same output again and get the same errors.
In our insurance agent. When an underwriter reports "this recommendation looked odd", the support engineer starts from the submission id in the cockpit, finds the run, and reads its failure records in attempt order: what failed, what the re-ask said, what passed in the end. Withoutattempt_numberandspan_id, three identical-looking errors are indistinguishable from one error logged three times.
Rates that show a change#
Plain English. A validation failure rate is the share of outputs that fail validation, over a period. For example: 29 failing first attempts out of 680 runs is 4.3 percent.
A single rate for everything is almost useless. It moves when traffic mixes change, when a new case type arrives, when a policy version is released. The useful rates are cut by the parts of the failure record: by stage, by field (the path), by error code, by schema version, by producer version. Here is the synthetic log of the example, 1,200 runs over 30 days, cut three ways, before and after day 18 (failures, runs, percent):
synthetic log: 115 failure records over 30 days, 1200 runs
all fields days 1-17: (29, 680, 4.3) days 18-30: (86, 520, 16.5)
/stated_confidence days 1-17: (0, 680, 0.0) days 18-30: (54, 520, 10.4)
/recommendation days 1-17: (7, 680, 1.0) days 18-30: (5, 520, 1.0)
The total says "something got worse". The cut by field says what: /stated_confidence went from zero failures to about one run in ten, while /recommendation did not move at all. And the producer field of each record (producer 2026-10-01 before, producer 2026-11-03 after) says who. That is a diagnosis an engineer can act on in an hour: compare the two producer versions, find the instruction that now says "state your confidence", fix it.
Plain English. Schema drift is when the data a producer really sends moves away from the contract, or from what it used to send, while the contract itself stays the same. For example: the new producer writes confidence as a word, though the schema still says number.
Drift has three common sources in an agent system, and each points to a different owner:
| Source | Example | Shows up as | Owner |
|---|---|---|---|
| The model's producer changes | new instructions or model setting | a new failure code on one field, starting on a release day | whoever owns the agent's prompt and settings |
| An outside producer changes | the roof-data provider sends status: "partial" (UW-S20) | schema failures at the tool boundary, on a new enum value | the integration owner, with the provider |
| The world changes | a new policy version makes old reasoning wrong | domain failures rise; schema failures do not | the rule owners |
Diagram: Three sources of drift · each shows up in a different cut and has a different owner
Read each lane left to right: what changed, how it shows in the failure records, and who acts. None of them changes the contract; all of them change the data.
- The agent's producer
- New instructions: The underwriting agent starts writing a field differently.
- One field, one code: A new failure code on one path, starting on a release day.
- An outside producer
- Provider changes: The roof-data provider sends a value the enum does not know.
- Tool boundary: Failures at the tool result contract; the run degrades to request_info.
- The world
- New policy version: Reasoning that was right under the old rules is now wrong.
- Domain stage: Domain failures rise while schema failures stay flat.
Production pitfall. The repair loop is good at hiding drift. A producer that fails on attempt 1 and succeeds on attempt 2 costs a second model call and some seconds of waiting each time, and no output ever reaches the underwriter broken. Alert on first-attempt failure rates by field, not only on fallbacks.
Try it: find the drift#
The lab shows the 30 days of the synthetic log as daily failure rates. Filter by stage, by field and by schema version. The total jumps on day 18 (the producer changes); a smaller rise appears from day 22 for version 2 traffic, where the new evidence_ids field sometimes names evidence the run never gathered.
| Filter | Days 1 to 17 | Days 18 to 30 |
|---|---|---|
| All fields | 29 of 680 runs (4.3 percent) | 86 of 520 runs (16.5 percent) |
/stated_confidence | 0 of 680 | 54 of 520 (10.4 percent) |
/recommendation | 7 of 680 (1.0 percent) | 5 of 520 (1.0 percent) |
Replay: keeping what is needed to check again#
Plain English. A replay artifact is everything needed to run the same validation again later and get the same result: the raw output, the exact schema, the facts and evidence the domain stage used, and the version of the validator. For example: the text "approved" plus recommendation.v1 as it was that day plus the facts of SUB-1004 plus policy 2026.2.
Why keep all of that, and not only the error? Three questions arrive weeks later, and each needs the original inputs:
- Was the validator right? An underwriter disputes a fallback. Replaying the raw output against the schema of that day answers whether the output really failed, or whether the validator had a bug.
- Would the new schema accept it? Before releasing version 2 of a contract, replay last month's outputs through the new validator and count what changes (the evolution page uses this as a compatibility test).
- Is the rule check reproducible? Domain failures depend on facts and policy version. Without them, a domain error cannot be explained, only repeated.
The example builds one with replay_artifact() and checks it with replay(). The real run on attempt 1 of UW-S04:
replay artifact keys: ['attempt_number', 'errors', 'evidence', 'facts', 'first_failure', 'policy_version', 'raw_output', 'run_id', 'schema', 'schema_name', 'schema_sha256_16', 'trace_id', 'validator_version']
replay: True
replay: True means the stored raw output, validated again with the stored schema and facts, produced exactly the same first failing stage and the same error list. The artifact stores the schema itself, not only its name, plus a short hash of it: schemas get edited in place more often than teams admit.
Plain English. A dead-letter store is where outputs go that could not be processed, with the reason, so that a person can look at them and they can be replayed. For example: every fallback record of the repair loop, with its replay artifact. In this example the recommendation record with status: "no_valid_recommendation" plays that role; the underwriter sees it in the cockpit and an engineer can replay it.
Before AI. A Kafka consumer that cannot process a message sends it to a dead-letter topic with headers saying why. The difference here is what "the message" is. With a model as producer, the raw text is the only copy of what was produced: the same call tomorrow returns something else. So the replay artifact must keep the raw output, not a parsed or repaired version of it.
What this is not. Collecting traces, metrics and logs, sampling them, and building dashboards is observability in general, owned by observe.sammuthu.com. Measuring whether outputs are right across many runs, with graders and golden datasets, is evaluation, owned by eval.sammuthu.com. This page is about what a contract failure should record.
For a coding agent. A coding agent'schange_reportis validated in CI. Record each failure with the pipeline run, the attempt and the field. Whentests_passedstarts arriving as"all"on one model setting, the cut by field and by agent version shows it the same week. CI-side detail is on diff-to-deploy.sammuthu.com.
What not to record#
A failure record and a replay artifact contain model output, and model output can contain whatever the model read: the broker's email, an applicant's details, an injected instruction. Three rules keep the evidence from becoming a liability:
- Store raw output where access is controlled, separately from the metrics. The failure counts can go to every dashboard; the raw text cannot.
- Keep messages free of raw values. The error messages in this example quote enum words and rule ids, never free text (see security).
- Give raw artifacts a retention period, decided with whoever owns the data, and delete them when it ends. Counts can live longer than the text they came from.
Checklist#
- Write one failure record per validation error, with
run_id,trace_id,span_id,attempt_number, stage, path, code, schema version and producer version. - Alert on first-attempt failure rates by field and code, not only on fallbacks.
- Record the producer version (instructions, model setting) on every record, so drift has a date and an owner.
- Keep a replay artifact for every failure: raw output, schema body and hash, facts, evidence, policy version, validator version.
- Replay a month of outputs through any new schema version before releasing it.
- Keep raw text behind access control and a retention period; let only counts travel widely.
Keeping artifacts for successes tooA sample of accepted outputs is the baseline drift is measured against
Failures alone cannot show drift that passes validation: a producer that starts writing shorter reasons, or always 0.9 as stated confidence, never fails. Keep replay artifacts for a sample of accepted outputs too, and track simple distributions per field (length of reasons, share of each recommendation word, the spread of stated confidence). A change in a distribution on a release day is drift even when every output is schema-valid. Deciding whether the new distribution is worse is evaluation, not validation.
Which versions to stampFive versions explain almost every failure
A failure can come from five things changing, and each needs its own version on the record: the schema (schema_version, schema_sha256_16), the validator code, the policy fixture (policy_version), the producer (prompt and model setting, here producer), and for tool results the provider's API. Missing any one of them turns some future investigation into guesswork. They are cheap to stamp and impossible to reconstruct.
In one breath#
Every validation failure should leave a record with stage, path, code and the identifiers that tie it to its run, its attempt and its trace, plus the versions of the schema and the producer. Cut failure rates by field and code, because the repair loop hides a drifting producer from everyone except the dashboard, and keep a replay artifact with the raw output, the exact schema and the facts, because a model's output cannot be produced again. Store the text carefully; let the counts travel.
You can now see what fails, where and since when. The next page asks how to find failures before production does: golden examples, generated inputs, hostile inputs and contract suites, in testing contracts.