Your Agent's Free-Text Output Is an API You Never Designed

A developer argues that agent pipelines fail at the boundary where free-text model output is parsed by host code, effectively creating an undocumented API, and proposes validating a typed decision object (status, confidence, findings) before any state change. The pattern, illustrated with a TypeScript assertDecision guard and a ReviewDecision schema, separates contract violations from judgment errors and implementation bugs, and is described as the basis for structured judgment tools exposed over MCP. The author maintains JevCases, an independent index of Jev use cases, and notes it is unaffiliated with TypeSafe.

Most agent demos fail in the same place, and it is not the model. It is the boundary where the model's output leaves the model and enters your program. Up to that point it is a string. Strings do not have a schema. So you end up with code like this: js const reply = await llm.complete prompt ; // please work if reply.toLowerCase .includes "approve" { await merge ; } else { await requestChanges ; } This works until the model writes "I would not approve this yet" and your substring check matches approve . Now a change that should have been blocked got merged. Any time you parse meaning out of generated text, you have declared an API. You just did not write it down. That interface has properties you probably did not intend: The fix is not a better prompt. Prompt engineering narrows the failure rate; it does not remove the parser. An agent step usually needs two different outputs, and they should not be the same output: The decision should come from a closed set that your code already understands: type ReviewDecision = { status: "pass" | "review" | "fail"; confidence: "low" | "medium" | "high"; findings: Array<{ file: string; note: string; severity: "info" | "warn" | "block" } ; }; Now the failure modes separate cleanly: | What broke | Where to look | |---|---| | status came back as "probably fine" | Contract violation — reject before acting | | status was valid but wrong | Judgment problem — improve evidence or prompt | | status was right but the wrong branch ran | Implementation bug in your own code | Without the typed boundary, all three look like "the AI did something weird," and you have no way to tell them apart. The important part is not the schema. It is that the schema is checked before anything changes. js const ALLOWED = new Set "pass", "review", "fail" ; function assertDecision value: unknown : ReviewDecision { if typeof value == "object" || value === null { throw new Error "Decision is not an object" ; } const d = value as Record<string, unknown ; if typeof d.status == "string" || ALLOWED.has d.status { throw new Error Illegal status: ${String d.status } ; } if Array.isArray d.findings { throw new Error "findings must be an array" ; } return { status: d.status as ReviewDecision "status" , confidence: d.confidence === "high" || d.confidence === "medium" ? d.confidence : "low", findings: d.findings as ReviewDecision "findings" , }; } A thrown error is a feature here. It is a loud, recoverable failure that happens before a merge, a payment, or a deploy — instead of a silent wrong branch. This is the pattern behind structured judgment tools, like the Choice and Score interfaces exposed over MCP in this case https://jevcases.com/cases/typesafe-mcp/ : the model supplies the judgment, the typing layer fixes its shape, and the calling program gets a value it can compare or gate on instead of prose it has to interpret. A typed result is not automatically trustworthy. status: "pass" with no support is just a shorter guess. So the decision record should carry: That gives you a replayable record. When a bad decision ships, you can tell whether the right evidence was missing, the wrong rule was applied, or the validation layer was too loose. Typing the output does not : It solves one narrow problem: the host no longer has to guess a control signal out of decorative prose. That is worth a lot. It is much easier to reason about a system where you can see the allowed choices, the selected choice, the evidence, and the resulting action. Before an agent step is allowed to change state, I want answers to these: If those are all "no," the demo can still look impressive. It is just not yet an instrument. Disclosure: I maintain JevCases https://jevcases.com/ , an independent index of Jev use cases and experiments. It is not affiliated with TypeSafe.