Building Fault-Tolerant AI Agent Workflows with Temporal and CrewAI

A developer built a fault-tolerant AI agent workflow using Temporal and CrewAI that survives process crashes, enforces retry policies, and blocks on human approval for unbounded periods. The reference implementation automates an SOP improvement pipeline where an LLM drafts documents in phases, a human approves or provides feedback, and a CrewAI agent loop validates and fixes the result before a final human gate approves a GitHub PR. The pattern uses Temporal as the durable orchestration engine and CrewAI as a stateless reasoning unit invoked from within Activities.

A reference pattern for running multi-agent LLM systems under strict human governance in production. Most "AI agent" demos share three properties that make them unfit for enterprise use: they hold state in process memory, they treat human approval as an optional UI affordance rather than a hard gate, and they have no answer for what happens when a call to the LLM provider times out at 2am. None of these are edge cases. In a real deployment: input call, or a LangGraph graph checkpointed to a database you manage yourself, does not survive a deploy, a container restart, or a crash during that wait — unless you build and maintain your own durability layer.The design question this article answers is narrow: how do you get a state machine that survives process death, retries failures according to an explicit policy, and blocks on human judgment for an unbounded period — without writing that infrastructure yourself? The reference implementation here is an SOP Standard Operating Procedure auto-improvement pipeline: an LLM drafts a document in three phases, a human approves or sends back feedback at each phase, an autonomous validation/fix loop CrewAI, two agents cleans up the result, and a final human gate approves a GitHub PR. The domain is incidental. The pattern — orchestration engine as the source of truth for state, agent framework as a stateless reasoning unit invoked from inside an Activity — is the point. The alternative most teams reach for first is a graph-based agent framework LangGraph and similar running as a long-lived process, or a hand-rolled script with a job queue. Both push the durability problem back onto the application: | Requirement | Plain script / job queue | LangGraph self-managed checkpointing | Temporal | |---|---|---|---| | Survives process crash mid-wait | No — in-memory state is gone | Depends on your checkpoint store's consistency guarantees | Yes — event history is the state; a new worker replays it | | Multi-day human approval wait | Requires a polling job + your own timeout handling | Requires your own checkpoint schema for "waiting" state | workflow.wait condition on a @workflow.signal , no polling | | Retry policy for LLM/API calls | Ad hoc, per call site | Ad hoc, per node | Declared once RetryPolicy , enforced by the runtime | | Deploying new code while workflows are in flight | Breaks in-flight state silently | Breaks in-flight state silently | workflow.patched — explicit, versioned branching | | Audit trail of what happened and when | You build it | You build it | Event history is authoritative by construction | The last two rows are the ones that actually matter at enterprise scale, and they're the ones demos never show. Determinism and replay. A Temporal Workflow function is not "running" in the conventional sense — it is replayed from its event history every time the worker needs to reconstruct its state after a crash, a deploy, a worker restart . This means the workflow function itself must be deterministic: no wall-clock time, no random numbers, no direct I/O. All non-deterministic work — LLM calls, GitHub API calls, validation — is pushed into Activities, which are allowed side effects and are retried independently by the runtime. sop workflow.py enforces this at the import level: with workflow.unsafe.imports passed through : from activities.sop activity import generate sop phase activity from activities.fix sop activity import fix sop with crew activity, ... from activities.github activity import GitHubActivity CrewAI itself imports os , makes network calls, and is not sandbox-safe — it cannot be imported inside the workflow's deterministic sandbox at all. It only ever runs inside an Activity. Versioning in-flight workflows. This is the detail most articles skip. A workflow that's been running for two days is mid-way through its event history when you ship new code. If you change the code path a running instance already took, replay diverges from history and the workflow breaks. sop workflow.py handles this with an explicit branch: if workflow.patched "split-writer-reviewer" : return await self. call fix decomposed sop text, failures, human feedback return await workflow.execute activity fix sop with crew activity, args= sop text, failures, human feedback, self. fix attempt , start to close timeout=timedelta minutes=7 , retry policy=LLM RETRY POLICY, This is a real production concern: the fix-loop was refactored from one monolithic CrewAI kickoff call into two separate Activities Writer, then Reviewer . Workflow instances already in flight when that code shipped keep using the old single-Activity path fix sop with crew activity ; new instances take the split path. Without patched , this refactor would non-deterministically break any workflow that was mid-execution at deploy time. There is no equivalent primitive in a hand-rolled state machine — you'd need to build and version your own replay-compatibility layer. CrewAI's contribution here is narrow and deliberate: it is a reasoning unit, not an orchestrator. It never owns state, never decides when to wait for a human, and never retries itself. Two agents — Writer and Reviewer — are defined per the standard CrewAI Agent / Task API: writer = Agent role="SOP 修正担当", goal= "SOP のバリデーション失敗項目を全て解消し、" "最小限の変更で明確・再現性の高い改善版を Markdown 形式で出力する。" , backstory="5 年以上のテクニカルライター経験を持つ専門家。...", llm=llm, verbose=True, allow delegation=False, Critically, in the current code path this pair is not invoked as a single CrewAI Crew.kickoff from inside one Activity. It's decomposed into two independent Activities, each with its own retry policy: python async def call fix decomposed self, sop text, failures, human feedback="" : self. active agent = "Writer" writer result = await workflow.execute activity writer task activity, args= sop text, failures, human feedback, self. fix attempt , start to close timeout=timedelta minutes=7 , retry policy=LLM RETRY POLICY, self. active agent = "Reviewer" reviewer result = await workflow.execute activity reviewer task activity, args= writer result.text , start to close timeout=timedelta minutes=7 , retry policy=LLM RETRY POLICY, ... The consequence: if the Reviewer's LLM call fails transiently, Temporal retries only the Reviewer Activity. The Writer's already-committed output is not re-run, re-billed, or re-generated which would risk non-identical output on a second pass . This is the practical argument for keeping the agent framework's unit of work small and pushing sequencing up into the orchestrator — multi-agent frameworks that bundle several agents into one opaque kickoff call give you all-or-nothing retry semantics, which is the wrong grain for cost- and latency-sensitive LLM pipelines. Retry policy itself is centralized and explicit, not scattered: LLM RETRY POLICY = RetryPolicy maximum attempts=3, initial interval=timedelta seconds=2 , backoff coefficient=2.0, maximum interval=timedelta seconds=30 , Every LLM- or API-backed Activity in the workflow — generation, validation, fix, GitHub PR creation — takes this same policy. There is one place to change backoff behavior, and it is enforced by the Temporal server, not by application code that might forget to call it. The product enforces four hard human judgment gates. None are advisory; the workflow physically cannot proceed past any of them without an external Signal. Gate 1–3: per-phase approval outline → draft → review . Each phase runs the same pattern: generate, then block on a Signal. php @workflow.signal def approve step self, feedback: str - None: """ "" 空文字 : 現フェーズを承認して次フェーズへ進む。 非空文字: フィードバックとして同フェーズを再生成する。 """ self. step feedback = feedback self. signal received = True for phase in PHASES: "outline", "draft", "review" self. current phase = phase self. attempt in phase = 0 last feedback = None while True: self. signal received = False reset BEFORE the Activity runs result = await self. call llm request self. current output = result.text self. status = "awaiting approval" await workflow.wait condition lambda: self. signal received feedback = self. step feedback self. step feedback = "" self. signal received = False if not feedback: self. approved phase = self. current output break move to next phase else: last feedback = feedback self. attempt in phase += 1 regenerate same phase One detail matters more than it looks: self. signal received = False is reset before await self. call llm ... , not after. Signals delivered while the Activity is still executing e.g., an eager reviewer clicking "approve" mid-generation are not lost — wait condition picks up the flag the instant control returns to the workflow. A polling-based approval mechanism would have a real race window here; Temporal's Signal delivery guarantees there isn't one. An empty-string feedback approves and advances; any non-empty string is treated as a rejection with revision instructions, and the same phase regenerates with that feedback injected into the next LLM request SOPRequest.feedback . This gives reviewers a single Signal type for two distinct actions, and gives the workflow author-level not binary revision control instead of just approve/reject. Gate 4: PR approval, with a reject path. The final gate — before creating a GitHub PR — supports both approval and a full round-trip back into the autonomous fix loop: php @workflow.signal def approve pr self - None: self. pr approved = True @workflow.signal def reject with feedback self, input data: dict - None: self. human feedback = input data.get "comment", "" if isinstance input data, dict else str input data if github params.require approval: self. status = "awaiting pr approval" self. pr approved = False await workflow.wait condition lambda: self. pr approved or bool self. human feedback if self. human feedback: continue loop back to Phase 4 autonomous fix with the feedback self. status = "creating pr" pr result = await self. call github pr ... wait condition here blocks on a disjunction of two Signals, which lets a single wait point resolve two different human outcomes without a secondary polling loop. A rejection doesn't just pause — it re-enters the fix loop with the human's comment injected as a first-class input alongside the validator's own failure list: if human feedback: fix result = await self. call fix final sop, , human feedback Why not just a database flag checked by a cron job? Because that reintroduces every problem Temporal exists to solve: you'd need your own polling interval latency vs. cost tradeoff , your own guarantee that a write isn't missed between poll cycles, and your own crash recovery for the poller itself. wait condition on a Signal is push-based, has no polling interval, and is backed by the same event history that makes the rest of the workflow durable — the wait is the crash-recovery mechanism, not a separate one. Observability without extra infrastructure. Every phase and fix attempt — approved or not — is appended to self. history , exposed via a @workflow.query : php @workflow.query def get history self - list: """全フェーズ・全試行の比較ログ。人間介入の Evidence として使用する。""" return list self. history This is the audit trail: every LLM output the workflow ever produced, whether a human approved or rejected it, and what feedback drove each regeneration — queryable at any point, including on a workflow that's still running, with no separate logging pipeline to maintain. Stripped of the SOP-specific domain logic, the pattern is: RetryPolicy . The workflow function itself does nothing non-deterministic. workflow.wait condition lambda: flag combined with @workflow.signal handlers gives push-based, race-free blocking that survives process restarts for however long a human takes — minutes or days, no difference to the implementation. kickoff , so retries are scoped to the agent that actually failed. workflow.patched , rather than silently breaking workflow instances that are mid-flight at deploy time.None of this is specific to SOP generation. The same shape applies to any pipeline where an LLM produces something a human must sign off on before it has real-world effect — code review automation, contract drafting, financial report generation — where "the process might be paused for two days waiting on a person" is not a corner case to be handled, but the normal operating condition the architecture has to be built around from the start. The code patterns discussed in this article were extracted from a production-ready SOP Standard Operating Procedure improvement platform built to demonstrate high-durability AI agents. Note: I am a hands-on Japanese engineer. While the technical architecture and code in this article are 100% mine, I use AI to assist with English translation for global communication. The complete, production-ready codebase with Docker Compose setup is fully documented and open-sourced here: 👉 GitHub: obataka/temporal-demo https://github.com/obataka/temporal-demo