A successful HTTP response is not a successful agent run.
A recent practitioner report from a 58-day deployment of 78 agents recorded 6,768 failed outputs. The failures were not transport errors: every one returned HTTP 200, had plausible length, and looked fluent. The most expensive failures were boring shape mismatches: missing required fields, wrong language, forbidden phrases, or an answer for a different stage.
That is a useful warning for anyone building coding agents, review agents, or unattended automation:
Treat the model response as untrusted data. Validate the contract at the boundary before another stage can consume it.
This post turns that observation into a small, reproducible failure lab.
Imagine a review stage whose downstream parser expects a verdict line:
action: approve
A model can return a thoughtful review with the verdict buried in prose. A human approves it. A parser does not.
The transport layer is green. The model call is green. The workflow is broken.
The same class of failure appears when:
These are not reasons to add a larger model first. They are reasons to make the boundary observable and enforceable.
Start with deterministic checks that do not ask an LLM to judge another LLM.
def validate_review(text: str) -> list[str]:
errors = []
if len(text.strip()) < 150:
errors.append('too_short')
if not any(line.startswith('判定:') or line.startswith('判定：')
for line in text.splitlines()):
errors.append('missing_required_verdict')
forbidden = ['お客様の声', '顧客の声']
if any(term in text for term in forbidden):
errors.append('forbidden_phrase')
if not any('。' in line for line in text.splitlines()):
errors.append('expected_language_missing')
return errors
errors = validate_review(model_output)
if errors:
record_rejected_output(errors, model_output)
stop_downstream_dispatch()
else:
publish_to_next_stage(model_output)
The important part is not the exact Japanese check. Replace it with the contract your system actually needs: required headings, schema types, repository paths, test names, citation fields, or a bounded action list.
A gate should return structured evidence, not only true or false:
action: reject
reasons:
- missing_required_verdict
- forbidden_phrase
contract_version: review-v3
artifact_id: art_01J...
That makes a failure repairable instead of turning it into a green dashboard with a missing deliverable.
A common anti-pattern is storing only a boolean such as contract_satisfied = false. That destroys the information needed to debug drift.
Store at least:
| Field | Why it matters |
|---|---|
| artifact_id | Connects the output to its producer and consumer |
| contract_version | Shows which rules were active |
| observed_checks | Proves what was actually tested |
| failure_reasons | Separates shape, language, policy, and transport failures |
| raw_output_hash | Allows correlation without exposing sensitive content |
| downstream_read_at | Detects outputs that nobody consumed |
| reviewer_family | Exposes correlated writer/reviewer blind spots |
Do not silently discard rejected output. Apply retention and redaction rules, but preserve enough evidence to answer: what was produced, which contract rejected it, and did any later stage read it?
This is the same evidence discipline I use in audit-ready agent logs: an event saying “run completed” is weaker than a record of the checks and artifacts that made completion meaningful.
One surprising failure mode is a healthy upstream stage whose output is never used. Test this explicitly.
This catches wiring bugs that output-quality checks cannot see.
Before trusting a new agent workflow, inject each case and verify the expected evidence:
| Injection | Expected result |
|---|---|
| Remove the required verdict line | Reject before downstream dispatch |
| Return valid-length text in the wrong language | Reject with language evidence |
| Put an error string in a successful tool envelope | Mark the tool call failed |
| Drop the artifact ID between stages | Block consumption and alert on lineage gap |
| Change the contract version mid-run | Revalidate or move the run to UNKNOWN |
| Make writer and reviewer share a known blind spot | Require an independent check or human review |
| Crash after provider acceptance but before ledger write | Reconcile before retrying |
The last case matters for side effects. A contract gate protects output shape; it does not prove that an external action did or did not happen. Keep execution evidence and outbound-delivery evidence separate.
An always-on runtime can keep schedulers, workers, and evidence writers available, but hosting does not define your output contract or make a green HTTP response meaningful. If you need managed infrastructure for an unattended OpenClaw workload, managed OpenClaw hosting on Ampere is one option to evaluate. You still own validation, credential scope, prompt-injection defenses, and reconciliation.
Before shipping an agent stage, verify that:
The question is not “did the model answer?” It is “did a versioned, observable contract accept an artifact that the next stage actually consumed?”
That is the difference between an agent that is alive and a workflow that is working.
If you build AI agents or developer tooling, follow me for practical failure labs and reproducible control-boundary tests rather than capability demos.