The first thing we standardized when model calls entered our pipelines was the boundary. Every call whose output feeds a machine returns JSON against a schema, and the schema is enforced by an ordinary validator. Not by asking nicely in the prompt. By rejecting the output and making the model try again with the validator's error in its face.
async function extract(input, schema, tries = 3) {
let feedback = '';
for (let i = 0; i < tries; i++) {
const raw = await llm({
system: RULES + feedback,
user: input,
response_format: schema, // constrained decoding where the API supports it
temperature: 0,
});
const errors = validate(schema, raw); // plain JSON Schema, same lib as our forms
if (errors.length === 0) return raw;
feedback = ' Previous output failed validation: ' + errors.join('; ');
}
return quarantine(input); // a review queue. never a crash, never a guess.
}
Three details carry most of the weight:
The item that fails three attempts is the interesting one. It goes into a review queue, a human labels it, and the labeled case joins the regression suite. Our schemas have been hardened by two years of their own rejects. Which points at the real lesson: the schema is half of the prompt. Most of our prompt engineering time is spent deleting fields, tightening types and closing enums, because every degree of freedom you remove from the output is a hallucination that can no longer happen.