The validation stages of the previous page have done their job on case UW-S16, one of this site's own scripted cases. On the facts of UW-004 (application says the roof is 9 years old, the roof-data provider says 22), the underwriting agent wrote "recommendation": "approve or escalate". The schema stage said no: that is not one of the four allowed words. Now what?
Three answers come to mind at once, and all three are tempting. Pick the first word, approve, and move on. Pick the safer word, escalate. Or ask the model again and keep asking until it gives a valid word. The first two are code deciding what the model meant, on a field that decides where a quote request goes. The third has no end: in this case the model answers approve/escalate the second time and escalate (or approve) the third.
Plain English. Repair is any action that turns a failed output into either an accepted output or a controlled hand-over to a person. For example: removing a markdown fence around correct JSON is a repair; so is giving up after three attempts and telling the underwriter there is no valid recommendation. This page sorts the possible actions by who acts, what they may change, and when to stop.
An error is a message for someone#
Every validation error has a reader who can do something about it. The question that picks the repair is: who can fix this, and can they fix it without changing what was meant?
| Repair | Who acts | May change | Fits errors like | In our cases |
|---|---|---|---|---|
| Deterministic repair | code | only the wrapping of the text | a code fence, a sentence before the JSON | UW-S02 |
| Re-ask | the model, given the errors | anything; the result is validated again | a wrong word, a missing field, a rule that does not apply | UW-S04 to UW-S14 |
| Clarification | a person outside the run (the broker, through the underwriter) | the facts | a fact nobody has: the roof age in UW-003 | a request_info recommendation |
| Fallback | code stops; a person takes over | nothing: it records what failed | anything still failing at the limit | UW-S13, UW-S16 |
Deterministic repair is a fix by code that cannot change meaning. In UW-S02 the model wrote a correct recommendation inside a markdown code fence, after the sentence "Here is the recommendation as requested:". Removing the fence and the sentence changes no value. Code can do that with certainty, so no re-ask is needed.
Re-ask means sending the errors back to the model and asking again. It raises the attempt number: attempt 1 failed, attempt 2 is the re-ask. In UW-S04 the model wrote "approved"; the re-ask says which value at which path is not allowed, and attempt 2 says "escalate".
Clarification means asking a person for a fact that nobody in the run has. It is not a reaction to a bad output; it is the right output. When the roof age is unknown (UW-003), the correct recommendation is request_info with missing: ["roof_age_years"], and the underwriter asks the broker. Code never "repairs" a missing fact.
Fallback is the bounded end: stop asking, store the errors, hand the case to a person.
Diagram: Who can fix this error · four repairs, from cheapest to most expensive
Read left to right. An error goes to the cheapest actor who can fix it without changing what was meant. Code may only unwrap. The model gets the errors. A missing fact goes to a person. Whatever still fails at the limit falls back to the underwriter.
- Code
- Deterministic repair: Removes a fence or prose around one complete JSON value; adds nothing.
- The model
- Re-ask: Sends the stage and each error with its path; the new output is validated from the start.
- A person, for a fact
- Clarification: A fact nobody in the run has is requested from the broker through the underwriter.
- A person, for the case
- Fallback: Code stops, stores no recommendation and the errors; the underwriter decides from the evidence.
What code must never fix#
Deterministic repair is the cheapest action, and that is its danger. A team that sees many failures of one kind is tempted to fix them in code. Some of those fixes are safe. Most are not.
| Failure | Tempting code fix | Why it is wrong |
|---|---|---|
Output cut off by the output limit (UW-S03) | close the open string and brackets | Code cannot know what was cut. A cut reasons list may have lost the sentence that carried the decision |
policy_version absent (UW-S05) | fill in the current version | The point of the field is that the model says which policy it read. Filling it in makes the check say nothing |
requires_human: false (UW-S13) | set it to true | The value is a symptom: the model behaved as if no person would decide. Flipping it hides that the output should not be trusted |
"approve or escalate" (UW-S16) | take the first word, or the safer word | Code picks the meaning. On the routing field of a quote request |
Extra field premium_usd (UW-S07) | drop the field and accept the rest | The model tried to quote a price, which it must not do. Dropping the field accepts the output that tried |
"stated_confidence": "high" (UW-S06) | map high to 0.8 | The number is invented by code and then looks like the model's |
The test is one question: could two careful engineers disagree about what the fixed value should be? If yes, code must not fix it. Unwrapping a fence passes the test. Every row of the table fails it.
unwrap() in examples/uw-contracts/repair.py is the only deterministic repair in the example. It takes the text inside a code fence, or from the first { to the last }, and accepts the result only if it parses on its own. It removes characters; it never adds one. On the cut-off output of UW-S03 it returns None: there is no closing brace to find, and it will not invent one.
Common confusion. "JSON repair" libraries that close brackets and fix quotes are useful when a person pastes broken JSON and checks the result by eye. Between a model and a program, nobody checks the result, so a repaired value enters the system as if the model had written it. Use them, if at all, to build a better error message, never to produce the accepted value.
The re-ask, and what it carries#
A re-ask is a new model call. What it sends matters as much as that it is sent. The example builds the message with reask_message() from the error records of the failed attempt. Here is the real message for UW-S04, produced by running the example in a scratch copy:
Your last output failed validation at the schema stage:
- /recommendation: "approved" is not one of "approve", "decline", "request_info", "escalate"
Return only the corrected JSON object. Do not add fields. Do not change the values that were accepted.
Three choices are visible in it. It names the stage, so the model knows whether the problem is shape or truth. It gives each error with its JSON Pointer path, so the model knows exactly which value to change. And it says what must not change: values that passed must stay as they were. Without that line a re-ask often returns a different recommendation with the original problem fixed and a new one introduced.
What the message leaves out is a choice too. It does not paste back the whole failed output, and it never quotes untrusted text from it. If a reason string carried an injected instruction, as in UW-S15, a re-ask that echoes the reason would put the instruction into the next prompt with the authority of the workflow. The security page explains why.
Before AI. A message consumer reading from a Kafka topic retries a failed message a fixed number of times, then sends it to a dead-letter topic for a person to inspect. A re-ask loop has the same shape with one difference that changes the design: each retry asks the producer to produce something new. The retry is not idempotent. The result of attempt 2 is a different output, so it goes through all four stages again from the start.
Bounds: attempt_number and the limit#
Plain English. A bound is a fixed limit on how many times the workflow tries. For example: at most 3 attempts per run, then stop. The example uses 3; the number is illustrative.
attempt_number is the counter. It starts at 1 for the first output of a run and rises by one with each re-ask. It is not the run: run_id (run-7f3a style) names one execution of the workflow for one submission, and all attempts of that run share it. On this site there is one trace per run, and each attempt is a span that carries its attempt_number, so a single trace shows the whole repair story. Tracing itself is explained on observe.sammuthu.com.
The loop in repair_loop() is short enough to read whole. For each attempt from 1 to the limit: get the output, run the stages; if the parse failed and deterministic repair is allowed, try unwrap() and run the stages again; if everything passed, accept; if attempts remain and re-ask is allowed, re-ask; otherwise, fall back. The real run over four cases:
UW-S02: accepted_after_repair [(1, 'parse', 'accept')]
UW-S03: accepted_after_reask [(1, 'parse', 're_ask'), (2, '-', 'accept')]
UW-S04: accepted_after_reask [(1, 'schema', 're_ask'), (2, '-', 'accept')]
UW-S16: fallback [(1, 'schema', 're_ask'), (2, 'schema', 're_ask'), (3, 'schema', 'fallback')]
Each tuple is one attempt: its number, the stage that failed first, and the action taken. The model here is a scripted stand-in, so these results show how the mechanism works, not how good any model is.
Why a limit at all? Three reasons, each sufficient. Cost: each re-ask is a full model call with the original context plus the errors. Time: the underwriter, or the broker waiting for a quote, is waiting. And evidence: when the same error comes back twice, the third call rarely differs. UW-S13 returns requires_human: false three times in a row. A model that keeps behaving as if no person will decide is telling you something a fourth attempt will not change.
Diagram: The bounded loop on UW-S16 · two re-asks, then a record with no recommendation
Read top to bottom. The workflow validates each attempt from the parse stage. Two re-asks carry the errors. At the limit of 3 the workflow stops and writes a record with no recommendation, which the underwriter sees.
- Model (stand-in) to Workflow (code): attempt 1: approve or escalate. The schema stage rejects the word: not one of the four.
- Workflow (code) to Model (stand-in): re-ask with the enum error. attempt_number becomes 2.
- Model (stand-in) to Workflow (code): attempt 2: approve/escalate. Still not one of the four words.
- Workflow (code) to Model (stand-in): re-ask with the enum error. attempt_number becomes 3, the limit.
- Model (stand-in) to Workflow (code): attempt 3: escalate (or approve). Still ambiguous; the limit is reached.
- Workflow (code): fallback record. recommendation null, status no_valid_recommendation, the errors, requires_human true.
- Workflow (code) to Underwriter: no valid recommendation. The underwriter decides from the submission and the evidence.
Try it: step through the loop#
The lab runs the real loop in your browser (the same code as the Python, checked row by row). Pick a failing case, set the limit and the allowed actions, and step through. Things to try: UW-S02 with deterministic repair switched off becomes a re-ask, one more model call for nothing. UW-S04 with the limit at 1 falls back even though attempt 2 would have been valid. UW-S13 falls back at any limit.
| Case | Attempt 1 fails at | Action per attempt (limit 3, both actions allowed) | Outcome |
|---|---|---|---|
UW-S01 | parse (prose) | re-ask, accept | accepted after re-ask |
UW-S02 | parse (fence) | unwrap, accept | accepted after deterministic repair |
UW-S03 | parse (cut off) | re-ask, accept | accepted after re-ask |
UW-S04 | schema (enum) | re-ask, accept | accepted after re-ask |
UW-S10 | domain (rule does not apply) | re-ask, accept | accepted after re-ask |
UW-S13 | domain (R-HUMAN-01) | re-ask, re-ask, fallback | fallback |
UW-S16 | schema (ambiguous word) | re-ask, re-ask, fallback | fallback |
The fallback record#
A fallback is not a failure of the workflow. It is the workflow working: a bounded attempt, an honest record, a person in charge. What makes it honest is what the record contains. Here is the real record code writes at the end of UW-S13 (the error list shortened to one entry):
{
"schema_version": 1,
"submission_id": "SUB-1004",
"run_id": "run-7f0d",
"attempt_number": 3,
"trace_id": "6863b07d0489a3e9d670f28baaa6fcb4",
"policy_version": "2026.2",
"evidence_ids": [],
"status": "no_valid_recommendation",
"recommendation": null,
"validation": [],
"errors": [
{
"stage": "domain",
"path": "/requires_human",
"code": "R-HUMAN-01",
"message": "every recommendation needs a decision by an underwriter"
}
],
"requires_human": true,
"human_decision": null
}
recommendation is null. Not the last attempt, not a "best effort". The last attempt failed validation, so it is not a recommendation. status says so in words the underwriter cockpit can show: no valid recommendation. The errors travel with the record, so the underwriter knows why and an engineer can count it later. requires_human is true, as for every record. human_decision is null until the underwriter acts. Every identifier in it was stamped by code, never taken from the model.
In our insurance agent. The underwriter sees the submission, the roof assessment and the message "no valid recommendation (validation failed 3 times at the domain stage)". They decide from the evidence, as they would have before any agent existed. The workflow cost them nothing but the time to read one line.
What this is not. Retries of a timed-out tool, backoff under load, idempotency keys and circuit breakers are reliability, owned by reliable.sammuthu.com. This page is about repairing an output that arrived and failed its contract. What runs next after a fallback (a queue, a notification, a different agent) is orchestration, owned by orchestrate.sammuthu.com.
For a coding agent. A coding agent'schange_reportfails validation becausetests_passedis"all"instead of a number. A re-ask with the error is right. Filling intests_passed = tests_runin code is the coding-agent version of flippingrequires_human: CI would then report tests as passed that nobody counted. The harness that runs such loops is described on harness.sammuthu.com.
Checklist#
- Allow exactly one kind of code repair: removing wrapping around one complete JSON value. Nothing else.
- Re-ask with the stage, each error with its path, and "do not change accepted values". Never echo untrusted text.
- Run all four stages again on every new attempt, from parse.
- Bound the loop; carry
attempt_numberon every attempt and on the record. - At the limit, write a record with
recommendation: null, the errors andrequires_human: true. - Never fill in, flip or pick a value on the model's behalf: not a version, not a boolean, not a word.
- Count outcomes per case kind (accepted, after repair, after re-ask, fallback); a rising fallback rate is a signal, not noise.
Choosing the limitMeasure the second-attempt success rate before picking a number
The right limit is an empirical question. Record, per error code, how often attempt 2 succeeds and how often attempt 3 succeeds after attempt 2 failed. If attempt 3 almost never succeeds for domain errors, a limit of 2 for domain errors and 3 for parse errors (where a larger output limit often fixes truncation) spends less for the same result. A limit per stage is a small change to repair_loop(). Measure on recorded runs, not on a guess; eval.sammuthu.com covers how.
Re-ask in the same conversation or a fresh oneThe failed output in context helps and hurts
A re-ask can append the error message to the conversation that produced the failed output, or start a fresh call with the original input and the error list. Appending is cheaper to write and lets the model see its own mistake, which often helps with schema errors. It also keeps the failed output, and anything injected into it, in the context of the next attempt. For outputs that carried untrusted text, a fresh call with only the original, trusted input plus the error codes is the safer design. What goes into the model's context is the subject of context.sammuthu.com.
Repair when the provider constrains the outputFewer parse and schema failures; the same domain failures
When the provider restricts decoding to your schema, most parse and schema failures disappear, and with them most re-asks. Truncation and refusals still happen, and no constrained decoding knows your policy fixture. The repair loop stays, with a different mix: fewer cheap fixes, the same expensive ones. The provider modes page describes what each mode guarantees.
In one breath#
Every validation error goes to whoever can fix it without changing what was meant: code may only unwrap, the model gets the errors and a re-ask, a missing fact becomes a request to a person, and anything still failing at the limit becomes a record with no recommendation and the errors attached. The limit is small and explicit, attempt_number counts against it, and every new attempt goes through all four stages. Code that fills in, flips or picks values on the model's behalf turns a visible failure into an invisible one.
You can now decide, for any error, the repair and its bound. What happens when the contract itself has to change, while old outputs, old readers and stored records still exist, is the next page: schema evolution.