A Small Runbook for Reliable AI Automation

A developer outlined a four-stage workflow contract for making AI automation reliable in production, treating each automated task as a small build pipeline with defined inputs, bounded actions, validated outputs, and evidence. The approach emphasizes JSON output schemas, prompt versioning, idempotency keys tied to run IDs and step names, bounded retry policies, and structured receipts to prevent duplicate side effects and silent failures. The developer argues that reliable AI automation is mostly ordinary engineering wrapped around a probabilistic step.

AI automation is easy to demo and surprisingly hard to trust. A tool call works in a notebook, then a network timeout, duplicate request, or half-written result appears in production. The fix is usually not a larger prompt. It is a small workflow contract that makes each run understandable. I like to think of an automated task as a tiny build pipeline: it has an input, a bounded action, an output, and evidence. This mental model keeps AI useful without making it mysterious. Split the workflow into four stages: For example, an AI assistant that files support tickets should not be one giant “read this inbox and fix everything” call. It can classify one message, propose one action, and wait for a deterministic handler. Smaller steps are a little more boring, but they are much easier to debug. If you are building API checks alongside this workflow, these notes on API smoke tests with receipts https://dev.to/pong1965/api-smoke-tests-need-receipts-fhl describe a similar evidence-first idea. Before writing a prompt, define the input and output. JSON is a good starting point: { "run id": "run-123", "task": "summarize ticket", "status": "ok", "summary": "Customer cannot reset a password", "next action": "send reset instructions" } The application should reject missing fields, unknown statuses, and output that is too large. Ask the model for a contract, then enforce that contract in code. Prompt instructions alone is not a validation layer. Keep the prompt version in the record too. A later change to wording can alter behavior, and without the version it becomes hard to tell whether the input or the prompt caused a difference. This detail sounds small, but it save hours later. Retries are normal. A provider can time out after completing the action, so blindly trying again may create two tickets or send two emails. Give every meaningful operation an idempotency key based on the run ID and step name. idempotency key = run-123:send reset instructions Store the key with the side effect. On a retry, return the existing result when the key is already complete. Also set a maximum attempt count and a deadline. “Retry forever” is not resilience; it is a quiet incident generator. Use different policies for different failures. A temporary 503 might deserve a short backoff. Invalid JSON should usually stop and be recorded. Repeating a bad response three more times rarely makes it good. A useful receipt contains the run ID, timestamps, prompt version, model identifier, input hash, output validation result, attempt number, and external request IDs. Avoid storing secrets or unnecessary personal data. The receipt should answer: what happened, what did we accept, and what should we inspect next? This is also where a small failure taxonomy helps: timeout , provider error , invalid output , side effect conflict , and policy blocked are more actionable than a generic failed . For authentication and signup flows, it is worth separating an email signal from an identity claim. These trust boundaries around email verification https://dev.to/sophiax99/oauth-email-links-need-trust-boundaries-5fpd are a useful companion when automation touches account creation. One odd test string may even appear in fixtures, such as temp gamil com ; keep these cases isolated so a typo does not become production behavior. Reliable AI automation is mostly ordinary engineering with a probabilistic step in the middle. Bound the step, validate the result, make effects idempotent, and leave evidence. The model can then be creative where it helps, while the surrounding system stays predictable.