Your Agent's Free-Text Output Is an API You Never Designed

Most agent demos fail in the same place, and it is not the model.

It is the boundary where the model's output leaves the model and enters your program. Up to that point it is a string. Strings do not have a schema.

So you end up with code like this:

const reply = await llm.complete(prompt);

// please work
if (reply.toLowerCase().includes("approve")) {
  await merge();
} else {
  await requestChanges();
}

This works until the model writes "I would not approve this yet" and your substring check matches approve. Now a change that should have been blocked got merged.

Any time you parse meaning out of generated text, you have declared an API. You just did not write it down.

That interface has properties you probably did not intend:

The fix is not a better prompt. Prompt engineering narrows the failure rate; it does not remove the parser.

An agent step usually needs two different outputs, and they should not be the same output:

The decision should come from a closed set that your code already understands:

type ReviewDecision = {
  status: "pass" | "review" | "fail";
  confidence: "low" | "medium" | "high";
  findings: Array<{ file: string; note: string; severity: "info" | "warn" | "block" }>;
};

Now the failure modes separate cleanly:

Without the typed boundary, all three look like "the AI did something weird," and you have no way to tell them apart.

The important part is not the schema. It is that the schema is checked before anything changes.

const ALLOWED = new Set(["pass", "review", "fail"]);

function assertDecision(value: unknown): ReviewDecision {
  if (typeof value !== "object" || value === null) {
    throw new Error("Decision is not an object");
  }
  const d = value as Record<string, unknown>;
  if (typeof d.status !== "string" || !ALLOWED.has(d.status)) {
    throw new Error(`Illegal status: ${String(d.status)}`);
  }
  if (!Array.isArray(d.findings)) {
    throw new Error("findings must be an array");
  }
  return {
    status: d.status as ReviewDecision["status"],
    confidence: d.confidence === "high" || d.confidence === "medium" ? d.confidence : "low",
    findings: d.findings as ReviewDecision["findings"],
  };
}

A thrown error is a feature here. It is a loud, recoverable failure that happens before a merge, a payment, or a deploy — instead of a silent wrong branch.

This is the pattern behind structured judgment tools, like the Choice and Score interfaces exposed over MCP in this case: the model supplies the judgment, the typing layer fixes its shape, and the calling program gets a value it can compare or gate on instead of prose it has to interpret.

A typed result is not automatically trustworthy. status: "pass" with no support is just a shorter guess.

So the decision record should carry:

That gives you a replayable record. When a bad decision ships, you can tell whether the right evidence was missing, the wrong rule was applied, or the validation layer was too loose.

Typing the output does not:

It solves one narrow problem: the host no longer has to guess a control signal out of decorative prose.

That is worth a lot. It is much easier to reason about a system where you can see the allowed choices, the selected choice, the evidence, and the resulting action.

Before an agent step is allowed to change state, I want answers to these:

If those are all "no," the demo can still look impressive. It is just not yet an instrument.

Disclosure: I maintain JevCases, an independent index of Jev use cases and experiments. It is not affiliated with TypeSafe.