All chapters
  1. Start hereParseable is not valid. Valid is not correct
  2. Foundations · 1From free text to fields and records: why contracts matter
  3. Foundations · 2Syntax, schema, semantics, business validity: four different kinds of right
  4. Foundations · 3JSON and JSON Schema: types, enums, nullable fields, unions and nested objects
  5. Building · 4Typed models: Pydantic in Python, records and Bean Validation in Java
  6. Building · 5Provider output modes: prompting, JSON mode, constrained decoding and tool arguments
  7. Building · 6Tool input, tool result, agent handoff, final response: four contracts on one path
  8. Building · 7Schema design: names, required or optional, and where each value came from
  9. Operating · 8Validation stages: parse, schema, cross-field, domain rules
  10. Operating · 9Errors and repair: retry, clarify, correct within bounds, fall back
  11. Operating · 10Schema evolution: versions, compatibility and migrations
  12. Operating · 11Security: a valid string can still be an attack
  13. Operating · 12Observing contracts: validation failures, schema drift and replay artifacts
  14. Operating · 13Testing contracts: golden examples, property-based tests, fuzzing and contract suites
  15. Reference · 14Frameworks and standards: what each one checks for you
  16. Reference · 15Advanced: partial objects in a stream, polymorphism, multilingual data and typed events
  17. Reference · 16The insurance lab: one submission, four contracts, end to end
  18. ReferenceGlossary
  19. AboutAbout this site and its sources
Operating · 12

Observing contracts: validation failures, schema drift and replay artifacts

A validation failure is evidence. What to record for each one, how to see a schema drifting before it breaks, and what to keep so that a failure can be replayed.

What you will understand
Failures are evidence
A rejected output tells you about the producer, the contract or the rules, if you keep it.
The failure record
Which identifiers tie one error to its run, its attempt and its trace.
Drift
The producer changes while the contract stays the same; rates by field show it first.
Replay
What to keep so a failure can be validated again, byte for byte, months later.

On day 18 of a month (in this site's synthetic log), nothing in the underwriting workflow was released. The contract did not change, the validator did not change, the policy fixture did not change. But the share of first attempts that failed validation went from about 4 in 100 to about 16 in 100, and stayed there. The repair loop of the repair page caught almost all of it: a re-ask, attempt 2 valid, a recommendation in the cockpit. The underwriters noticed nothing except that some recommendations took a little longer. The bill for model calls noticed.

What changed was the producer. Someone updated the instructions and a model setting of the underwriting agent, and the new producer, now and then, wrote "stated_confidence": "high" instead of a number. Each such output failed at the schema stage, was re-asked and fixed. The repair loop hid the problem. Only the failure records showed it.

This page is about those records: what a validation failure should leave behind, how to count failures so that a change shows up, and what to keep so that any failure can be examined again later. The numbers on this page come from a seeded, synthetic log; they show how the mechanism works, not how any real system behaves.

A failure is a record, not a log line#

Plain English. A failure record is one row of data per validation error, written at the moment the error happens, with the identifiers that tie it to everything else. For example: "run run-7f10, attempt 2, schema stage, path /recommendation, code enum, action re-ask".

The obvious alternative is a log line: WARN validation failed: "approve/escalate" is not one of .... It is readable, and it cannot be counted by field, by stage or by version without parsing the message, which someone will reword next quarter. The error record of the validation page already has the right parts: stage, path, code, message. A failure record adds the identifiers.

Here are the three failure records of case UW-S16 (this site's own scripted case: "approve or escalate", ambiguous three times), from a real run of failure_records() in examples/uw-contracts/observe.py:

{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "e92802de838ccaec", "attempt_number": 1, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "re_ask"}
{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "55eb02eb35f3188d", "attempt_number": 2, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "re_ask"}
{"run_id": "run-7f10", "trace_id": "75b0918fb7ba74d9739c4b42989b647d", "span_id": "66c729ed08628ea3", "attempt_number": 3, "stage": "schema", "path": "/recommendation", "code": "enum", "action": "fallback"}

Read the identifiers from left to right. They are different things, and the series keeps them apart:

IdentifierMeansSame across the three records?
submission_id (SUB-1004)one quote requestyes (carried in the full record)
run_idone execution of the workflow for that submissionyes: one run
trace_id32 hex characters; on this site one trace per runyes
span_id16 hex characters; one step inside the trace, here one attemptno: three attempts, three spans
attempt_numberwhich try inside the runno: 1, 2, 3

The sizes are not invented here. The W3C Trace Context recommendation (Level 1, 2021) defines a trace id as "a 16-byte array, for example, 4bf92f3577b34da6a3ce929d0e0e4736" (32 hex characters) and a parent id as "an 8-byte array" (16 hex characters), and says an all-zero value is invalid. The example derives both from the run id with a hash so that the runs are reproducible; real systems take them from their tracing library. What traces and spans are, and how to collect them, is the subject of observe.sammuthu.com. This page only uses the identifiers to join failure records to the rest.

Diagram: One run, three attempts, three failure records · the identifiers that tie them together

Read left to right. One submission gets one run and, on this site, one trace. Each attempt is a span with its attempt number. Each validation error becomes a failure record carrying all of these, so it can be joined to everything else.

  1. The request
    1. Submission: One quote request from a broker.
  2. One run, one trace
    1. Run: One execution of the workflow for the submission.
    2. Trace: On this site one trace per run.
  3. Three attempts, three spans
    1. Attempt 1: schema /recommendation enum, re-ask.
    2. Attempt 2: schema /recommendation enum, re-ask.
    3. Attempt 3: schema /recommendation enum, fallback.
  4. Kept
    1. Failure records: Counted by stage, field, code, schema and producer version.
    2. Replay artifacts: Enough to validate the same output again and get the same errors.
In our insurance agent. When an underwriter reports "this recommendation looked odd", the support engineer starts from the submission id in the cockpit, finds the run, and reads its failure records in attempt order: what failed, what the re-ask said, what passed in the end. Without attempt_number and span_id, three identical-looking errors are indistinguishable from one error logged three times.

Rates that show a change#

Plain English. A validation failure rate is the share of outputs that fail validation, over a period. For example: 29 failing first attempts out of 680 runs is 4.3 percent.

A single rate for everything is almost useless. It moves when traffic mixes change, when a new case type arrives, when a policy version is released. The useful rates are cut by the parts of the failure record: by stage, by field (the path), by error code, by schema version, by producer version. Here is the synthetic log of the example, 1,200 runs over 30 days, cut three ways, before and after day 18 (failures, runs, percent):

synthetic log: 115 failure records over 30 days, 1200 runs
all fields           days 1-17: (29, 680, 4.3)   days 18-30: (86, 520, 16.5)
/stated_confidence   days 1-17: (0, 680, 0.0)   days 18-30: (54, 520, 10.4)
/recommendation      days 1-17: (7, 680, 1.0)   days 18-30: (5, 520, 1.0)

The total says "something got worse". The cut by field says what: /stated_confidence went from zero failures to about one run in ten, while /recommendation did not move at all. And the producer field of each record (producer 2026-10-01 before, producer 2026-11-03 after) says who. That is a diagnosis an engineer can act on in an hour: compare the two producer versions, find the instruction that now says "state your confidence", fix it.

Plain English. Schema drift is when the data a producer really sends moves away from the contract, or from what it used to send, while the contract itself stays the same. For example: the new producer writes confidence as a word, though the schema still says number.

Drift has three common sources in an agent system, and each points to a different owner:

SourceExampleShows up asOwner
The model's producer changesnew instructions or model settinga new failure code on one field, starting on a release daywhoever owns the agent's prompt and settings
An outside producer changesthe roof-data provider sends status: "partial" (UW-S20)schema failures at the tool boundary, on a new enum valuethe integration owner, with the provider
The world changesa new policy version makes old reasoning wrongdomain failures rise; schema failures do notthe rule owners

Diagram: Three sources of drift · each shows up in a different cut and has a different owner

Read each lane left to right: what changed, how it shows in the failure records, and who acts. None of them changes the contract; all of them change the data.

  1. The agent's producer
    1. New instructions: The underwriting agent starts writing a field differently.
    2. One field, one code: A new failure code on one path, starting on a release day.
  2. An outside producer
    1. Provider changes: The roof-data provider sends a value the enum does not know.
    2. Tool boundary: Failures at the tool result contract; the run degrades to request_info.
  3. The world
    1. New policy version: Reasoning that was right under the old rules is now wrong.
    2. Domain stage: Domain failures rise while schema failures stay flat.
Production pitfall. The repair loop is good at hiding drift. A producer that fails on attempt 1 and succeeds on attempt 2 costs a second model call and some seconds of waiting each time, and no output ever reaches the underwriter broken. Alert on first-attempt failure rates by field, not only on fallbacks.

Try it: find the drift#

The lab shows the 30 days of the synthetic log as daily failure rates. Filter by stage, by field and by schema version. The total jumps on day 18 (the producer changes); a smaller rise appears from day 22 for version 2 traffic, where the new evidence_ids field sometimes names evidence the run never gathered.

FilterDays 1 to 17Days 18 to 30
All fields29 of 680 runs (4.3 percent)86 of 520 runs (16.5 percent)
/stated_confidence0 of 68054 of 520 (10.4 percent)
/recommendation7 of 680 (1.0 percent)5 of 520 (1.0 percent)

Replay: keeping what is needed to check again#

Plain English. A replay artifact is everything needed to run the same validation again later and get the same result: the raw output, the exact schema, the facts and evidence the domain stage used, and the version of the validator. For example: the text "approved" plus recommendation.v1 as it was that day plus the facts of SUB-1004 plus policy 2026.2.

Why keep all of that, and not only the error? Three questions arrive weeks later, and each needs the original inputs:

  1. Was the validator right? An underwriter disputes a fallback. Replaying the raw output against the schema of that day answers whether the output really failed, or whether the validator had a bug.
  2. Would the new schema accept it? Before releasing version 2 of a contract, replay last month's outputs through the new validator and count what changes (the evolution page uses this as a compatibility test).
  3. Is the rule check reproducible? Domain failures depend on facts and policy version. Without them, a domain error cannot be explained, only repeated.

The example builds one with replay_artifact() and checks it with replay(). The real run on attempt 1 of UW-S04:

replay artifact keys: ['attempt_number', 'errors', 'evidence', 'facts', 'first_failure', 'policy_version', 'raw_output', 'run_id', 'schema', 'schema_name', 'schema_sha256_16', 'trace_id', 'validator_version']
replay: True

replay: True means the stored raw output, validated again with the stored schema and facts, produced exactly the same first failing stage and the same error list. The artifact stores the schema itself, not only its name, plus a short hash of it: schemas get edited in place more often than teams admit.

Plain English. A dead-letter store is where outputs go that could not be processed, with the reason, so that a person can look at them and they can be replayed. For example: every fallback record of the repair loop, with its replay artifact. In this example the recommendation record with status: "no_valid_recommendation" plays that role; the underwriter sees it in the cockpit and an engineer can replay it.

Before AI. A Kafka consumer that cannot process a message sends it to a dead-letter topic with headers saying why. The difference here is what "the message" is. With a model as producer, the raw text is the only copy of what was produced: the same call tomorrow returns something else. So the replay artifact must keep the raw output, not a parsed or repaired version of it.
What this is not. Collecting traces, metrics and logs, sampling them, and building dashboards is observability in general, owned by observe.sammuthu.com. Measuring whether outputs are right across many runs, with graders and golden datasets, is evaluation, owned by eval.sammuthu.com. This page is about what a contract failure should record.
For a coding agent. A coding agent's change_report is validated in CI. Record each failure with the pipeline run, the attempt and the field. When tests_passed starts arriving as "all" on one model setting, the cut by field and by agent version shows it the same week. CI-side detail is on diff-to-deploy.sammuthu.com.

What not to record#

A failure record and a replay artifact contain model output, and model output can contain whatever the model read: the broker's email, an applicant's details, an injected instruction. Three rules keep the evidence from becoming a liability:

Checklist#

Keeping artifacts for successes tooA sample of accepted outputs is the baseline drift is measured against

Failures alone cannot show drift that passes validation: a producer that starts writing shorter reasons, or always 0.9 as stated confidence, never fails. Keep replay artifacts for a sample of accepted outputs too, and track simple distributions per field (length of reasons, share of each recommendation word, the spread of stated confidence). A change in a distribution on a release day is drift even when every output is schema-valid. Deciding whether the new distribution is worse is evaluation, not validation.

Which versions to stampFive versions explain almost every failure

A failure can come from five things changing, and each needs its own version on the record: the schema (schema_version, schema_sha256_16), the validator code, the policy fixture (policy_version), the producer (prompt and model setting, here producer), and for tool results the provider's API. Missing any one of them turns some future investigation into guesswork. They are cheap to stamp and impossible to reconstruct.

In one breath#

Every validation failure should leave a record with stage, path, code and the identifiers that tie it to its run, its attempt and its trace, plus the versions of the schema and the producer. Cut failure rates by field and code, because the repair loop hides a drifting producer from everyone except the dashboard, and keep a replay artifact with the raw output, the exact schema and the facts, because a model's output cannot be produced again. Store the text carefully; let the counts travel.

You can now see what fails, where and since when. The next page asks how to find failures before production does: golden examples, generated inputs, hostile inputs and contract suites, in testing contracts.