🚀 Technical Briefing: This tutorial is part of our deep-dive series on Agentic Workflows at Gate of AI. For the full technical breakdown, interactive code sandbox, and the native Arabic translation, visit the original article here.
Tutorial · Python AI workflow design
Learn how to separate AI intent interpretation from deterministic workflow execution, persist workflow history, add human approval gates, and reason about retries and idempotency. This guide presents an implementation blueprint rather than a claim that a short SQLite example provides production-grade distributed execution.
Long-running AI workflows have failure modes that a basic prompt-and-response application does not. A process can stop between steps. A request can be delivered again. A tool may succeed even though the caller never receives its response. An action may partly complete before a later action fails. For actions such as sending a message or changing a business record, blindly retrying can create duplicate side effects.
The verified material identifies durable checkpoints, persisted workflow state, structured action history, approval gates, and idempotency as relevant design concerns for long-running agents. It also describes an architecture that keeps natural-language interpretation separate from deterministic workflow generation: an LLM interprets an intent, validated logic produces a reproducible DAG, and domain-authored Skills provide vocabulary mappings, constraints, and optimization knowledge.
This tutorial translates those ideas into a small Python design. It uses SQLite as a simple local persistence example and an OpenAI client as an optional planning boundary. The example is not a distributed workflow engine. For workflows that must survive worker crashes, coordinate multiple workers, or reliably retry external actions, evaluate a durable execution system. Temporal and Azure Durable Functions are examples named in the verified context.
Use three conceptual layers:
Before execution, show a human-readable plan to a responsible person and allow approval or refinement. After approval, provision or prepare the required resources, record the resulting measurements or resolved estimates, and request execution approval when the workflow calls for it. This resembles the staged interaction described in the verified science-automation architecture: intent interpretation, planning, user validation, preparation, final workflow generation, and execution approval.
The distinction between a DAG and an agent's free-form plan matters. A DAG is an application-controlled representation of known steps and dependencies. Do not let model-generated text become executable code, arbitrary shell commands, or unrestricted tool calls.
Begin with a small state vocabulary. The exact statuses depend on the application, but separating planning, approval, and execution makes it easier to reason about which actions are allowed.
from enum import StrEnum
class Status(StrEnum):
RECEIVED = "received"
PLANNED = "planned"
AWAITING_APPROVAL = "awaiting_approval"
APPROVED = "approved"
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
REJECTED = "rejected"
Model a workflow run as a durable record containing a stable run identifier, an idempotency key, the validated intent, the workflow definition or version, the current status, and timestamps. Keep action attempts and action outcomes in a structured history rather than relying on console output. A history should help answer what ran, when it ran, and what result was recorded.
Do not infer that a status field alone makes a workflow durable. State updates must be persisted at meaningful boundaries, and the executor must be designed for interruption and repeat delivery. In a production system, define and test the permitted transitions. For example, a rejected run must not proceed to execution, and a completed run must not silently start over.
SQLite can demonstrate persistence in a single-process prototype. Keep the run record separate from its event history so a developer can inspect state changes. Use parameterized SQL and a unique constraint on the idempotency key. The following schema is a minimal starting point, not a replacement for a production migration or concurrency strategy.
import sqlite3
SCHEMA = """
CREATE TABLE IF NOT EXISTS runs (
run_id TEXT PRIMARY KEY,
idempotency_key TEXT NOT NULL UNIQUE,
status TEXT NOT NULL,
workflow_json TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS events (
event_id INTEGER PRIMARY KEY AUTOINCREMENT,
run_id TEXT NOT NULL REFERENCES runs(run_id),
event_type TEXT NOT NULL,
event_json TEXT NOT NULL,
created_at TEXT NOT NULL
);
"""
def initialize_database(path: str) -> None:
with sqlite3.connect(path) as db:
db.execute("PRAGMA foreign_keys = ON")
db.executescript(SCHEMA)
When accepting a request, create the run and its initial event in one database transaction. If the same idempotency key is submitted again, return the existing run rather than creating a second logical workflow. A unique constraint is important because checking for a key and inserting later, without a constraint, can allow concurrent submissions to race.
For a production service with multiple workers, a local SQLite database may not be the right coordination mechanism. Choose storage and transaction behavior based on the deployment, then test concurrent updates and recovery. The verified context discusses durable checkpoints and workflow state but does not establish that this particular SQLite schema is sufficient for distributed execution.
Ask the model for a structured intent, not an executable workflow. Validate its output in application code against a finite schema: recognized intent types, bounded parameter values, and a known set of action names. Reject missing required fields and unsupported actions. Keep authorization and policy decisions outside the model response.
If using the OpenAI Python SDK, use the current client-based interface and consult the SDK documentation for the installed version. Do not assume a specific model name or response capability without checking current provider documentation and account access. The verified context does not identify a model, model availability, pricing, or API response format.
from pydantic import BaseModel, Field
class Intent(BaseModel):
category: str = Field(min_length=1, max_length=80)
summary: str = Field(min_length=1, max_length=500)
parameters: dict[str, str | int | bool] = Field(default_factory=dict)
ALLOWED_CATEGORIES = {
"reporting",
"customer_communication",
"access_change",
"data_change",
}
def validate_intent(candidate: dict) -> Intent:
intent = Intent.model_validate(candidate)
if intent.category not in ALLOWED_CATEGORIES:
raise ValueError("unsupported intent category")
return intent
This is only a structural example. Production validation should be specific to the domain and should reject unknown parameters where appropriate. A model's statement that an action is safe is not a policy decision. The deterministic layer should decide which workflow template applies and whether human review is required.
Map validated intent to application-owned steps. For example, a reporting intent might map to prepare-inputs, generate-report, and record-result. A customer communication intent could map to draft-message, request-review, and send-after-approval. The example names are illustrative; they do not establish integration with any particular provider.
Store the workflow version along with the run. If a workflow definition changes later, a resumed run should not accidentally reinterpret an old plan using new rules. Generate a human-readable plan from the same validated representation that drives the DAG, then before sensitive or externally visible actions. Record the approver's decision and the point in the workflow at which it applies.
An approval gate is meaningful only if the execution path enforces it. Check the persisted state immediately before carrying out the gated action. Do not rely on a user interface to hide an execute button, or on the LLM to respect an approval instruction. For higher-risk actions, use approval rules appropriate to the action and organization; the verified context supports the general concept of approval gates but does not prescribe a particular policy.
Retries are necessary when transient failures occur, but a retry is not automatically safe. Suppose a remote service accepts a request and the workflow process stops before recording success. On recovery, the system may not know whether the action happened. Sending the same request again can duplicate a side effect.
Assign a stable idempotency key to each logical action, derived from the workflow run and action identity. Pass it to the external service when that service supports idempotency. Otherwise, maintain a durable action record and use a provider lookup or reconciliation procedure before repeating an uncertain operation. Record attempts, responses, and errors as structured events. Treat partial completion as its own recovery case rather than retrying every step indiscriminately.
A local append-only JSONL file is useful for a demonstration, but it cannot by itself guarantee exactly-once behavior across a crash and an external service. Nor does recording a workflow checkpoint atomically with a third-party action become possible merely by using SQLite. Design for duplicate delivery and uncertain outcomes. The verified context specifically highlights retries, partial failures, duplicate side effects, durable checkpoints, and action history as reliability concerns.
On startup, identify runs left in an in-progress state and apply an explicit recovery policy. Depending on the step, recovery may resume from a durable checkpoint, reconcile an external action, wait for a person, or mark the run for manual investigation. Do not convert every interrupted run into an automatic retry.
Expose an event history with clear event types such as request received, intent validated, plan presented, approval granted, action started, action outcome recorded, and run completed. Avoid logging secrets or unrestricted personal data. Add tracing and guardrails appropriate to the application; the verified context specifically points to tracing and guardrails as useful concerns and names the OpenAI Agents SDK as one possible tool. That mention should not be read as a requirement or as proof that it supplies a particular durability guarantee.
For longer-running, multi-step agents, consider a durable workflow engine instead of expanding a custom state machine indefinitely. The verified context names Temporal and Azure Durable Functions as examples. Compare their documented execution, persistence, retry, and operational behavior against your requirements before selecting one.
Write tests for the properties your workflow must preserve. At minimum, verify that duplicate submission with the same key returns the same logical run; a rejected run cannot execute; a gated action waits for approval; and repeating a completed action does not create a duplicate result where the executor can enforce that behavior.
Also simulate a worker stopping after an external action but before its outcome is recorded. Verify that recovery reconciles the uncertain result rather than blindly repeating it. Test partial completion, transient provider errors, invalid model output, duplicate event delivery, and an approval decision arriving after the run has changed state. These are design checks, not claims that the illustrative schema above already handles every case.
A dependable AI workflow is not just a prompt followed by a tool call. It separates semantic interpretation from deterministic execution, stores enough state to understand and resume work, and treats human approval and idempotency as explicit workflow concerns. A small Python and SQLite prototype can make those boundaries visible, but it should not be mistaken for a distributed exactly-once system.
For long-running automation, build around durable checkpoints, structured action history, careful recovery, and controls that prevent a model from directly authorizing side effects. Select an execution engine and storage design only after defining the failure modes and guarantees the application actually requires.