A reference pattern for running multi-agent LLM systems under strict human governance in production.
Most "AI agent" demos share three properties that make them unfit for enterprise use: they hold state in process memory, they treat human approval as an optional UI affordance rather than a hard gate, and they have no answer for what happens when a call to the LLM provider times out at 2am.
None of these are edge cases. In a real deployment:
input()
call, or a LangGraph graph checkpointed to a database you manage yourself, does not survive a deploy, a container restart, or a crash during that wait — unless you build and maintain your own durability layer.The design question this article answers is narrow: how do you get a state machine that survives process death, retries failures according to an explicit policy, and blocks on human judgment for an unbounded period — without writing that infrastructure yourself?
The reference implementation here is an SOP (Standard Operating Procedure) auto-improvement pipeline: an LLM drafts a document in three phases, a human approves or sends back feedback at each phase, an autonomous validation/fix loop (CrewAI, two agents) cleans up the result, and a final human gate approves a GitHub PR. The domain is incidental. The pattern — orchestration engine as the source of truth for state, agent framework as a stateless reasoning unit invoked from inside an Activity — is the point.
The alternative most teams reach for first is a graph-based agent framework (LangGraph and similar) running as a long-lived process, or a hand-rolled script with a job queue. Both push the durability problem back onto the application:
| Requirement | Plain script / job queue | LangGraph (self-managed checkpointing) | Temporal |
|---|---|---|---|
| Survives process crash mid-wait | No — in-memory state is gone | Depends on your checkpoint store's consistency guarantees | Yes — event history is the state; a new worker replays it |
| Multi-day human approval wait | Requires a polling job + your own timeout handling | Requires your own checkpoint schema for "waiting" state | |
workflow.wait_condition on a @workflow.signal , no polling |
|||
| Retry policy for LLM/API calls | Ad hoc, per call site | Ad hoc, per node | Declared once (RetryPolicy ), enforced by the runtime |
| Deploying new code while workflows are in flight | Breaks in-flight state silently | Breaks in-flight state silently | |
workflow.patched() — explicit, versioned branching |
|||
| Audit trail of what happened and when | You build it | You build it | Event history is authoritative by construction |
The last two rows are the ones that actually matter at enterprise scale, and they're the ones demos never show.
Determinism and replay. A Temporal Workflow function is not "running" in the conventional sense — it is replayed from its event history every time the worker needs to reconstruct its state (after a crash, a deploy, a worker restart). This means the workflow function itself must be deterministic: no wall-clock time, no random numbers, no direct I/O. All non-deterministic work — LLM calls, GitHub API calls, validation — is pushed into Activities, which are allowed side effects and are retried independently by the runtime. sop_workflow.py
enforces this at the import level:
with workflow.unsafe.imports_passed_through():
from activities.sop_activity import generate_sop_phase_activity
from activities.fix_sop_activity import fix_sop_with_crew_activity, ...
from activities.github_activity import GitHubActivity
CrewAI itself imports os
, makes network calls, and is not sandbox-safe — it cannot be imported inside the workflow's deterministic sandbox at all. It only ever runs inside an Activity.
Versioning in-flight workflows. This is the detail most articles skip. A workflow that's been running for two days is mid-way through its event history when you ship new code. If you change the code path a running instance already took, replay diverges from history and the workflow breaks. sop_workflow.py
handles this with an explicit branch:
if workflow.patched("split-writer-reviewer"):
return await self._call_fix_decomposed(sop_text, failures, human_feedback)
return await workflow.execute_activity(
fix_sop_with_crew_activity,
args=[sop_text, failures, human_feedback, self._fix_attempt],
start_to_close_timeout=timedelta(minutes=7),
retry_policy=LLM_RETRY_POLICY,
)
This is a real production concern: the fix-loop was refactored from one monolithic CrewAI kickoff()
call into two separate Activities (Writer, then Reviewer). Workflow instances already in flight when that code shipped keep using the old single-Activity path (fix_sop_with_crew_activity
); new instances take the split path. Without patched()
, this refactor would non-deterministically break any workflow that was mid-execution at deploy time. There is no equivalent primitive in a hand-rolled state machine — you'd need to build and version your own replay-compatibility layer.
CrewAI's contribution here is narrow and deliberate: it is a reasoning unit, not an orchestrator. It never owns state, never decides when to wait for a human, and never retries itself. Two agents — Writer and Reviewer — are defined per the standard CrewAI Agent
/Task
API:
writer = Agent(
role="SOP 修正担当",
goal=(
"SOP のバリデーション失敗項目を全て解消し、"
"最小限の変更で明確・再現性の高い改善版を Markdown 形式で出力する。"
),
backstory="5 年以上のテクニカルライター経験を持つ専門家。...",
llm=llm,
verbose=True,
allow_delegation=False,
)
Critically, in the current code path this pair is not invoked as a single CrewAI Crew.kickoff()
from inside one Activity. It's decomposed into two independent Activities, each with its own retry policy:
async def _call_fix_decomposed(self, sop_text, failures, human_feedback=""):
self._active_agent = "Writer"
writer_result = await workflow.execute_activity(
writer_task_activity,
args=[sop_text, failures, human_feedback, self._fix_attempt],
start_to_close_timeout=timedelta(minutes=7),
retry_policy=LLM_RETRY_POLICY,
)
self._active_agent = "Reviewer"
reviewer_result = await workflow.execute_activity(
reviewer_task_activity,
args=[writer_result.text],
start_to_close_timeout=timedelta(minutes=7),
retry_policy=LLM_RETRY_POLICY,
)
...
The consequence: if the Reviewer's LLM call fails transiently, Temporal retries only the Reviewer Activity. The Writer's already-committed output is not re-run, re-billed, or re-generated (which would risk non-identical output on a second pass). This is the practical argument for keeping the agent framework's unit of work small and pushing sequencing up into the orchestrator — multi-agent frameworks that bundle several agents into one opaque kickoff()
call give you all-or-nothing retry semantics, which is the wrong grain for cost- and latency-sensitive LLM pipelines.
Retry policy itself is centralized and explicit, not scattered:
LLM_RETRY_POLICY = RetryPolicy(
maximum_attempts=3,
initial_interval=timedelta(seconds=2),
backoff_coefficient=2.0,
maximum_interval=timedelta(seconds=30),
)
Every LLM- or API-backed Activity in the workflow — generation, validation, fix, GitHub PR creation — takes this same policy. There is one place to change backoff behavior, and it is enforced by the Temporal server, not by application code that might forget to call it.
The product enforces four hard human judgment gates. None are advisory; the workflow physically cannot proceed past any of them without an external Signal.
Gate 1–3: per-phase approval (outline → draft → review). Each phase runs the same pattern: generate, then block on a Signal.
@workflow.signal
def approve_step(self, feedback: str) -> None:
"""
"" (空文字): 現フェーズを承認して次フェーズへ進む。
非空文字: フィードバックとして同フェーズを再生成する。
"""
self._step_feedback = feedback
self._signal_received = True
for phase in PHASES: # ["outline", "draft", "review"]
self._current_phase = phase
self._attempt_in_phase = 0
last_feedback = None
while True:
self._signal_received = False # reset BEFORE the Activity runs
result = await self._call_llm(request)
self._current_output = result.text
self._status = "awaiting_approval"
await workflow.wait_condition(lambda: self._signal_received)
feedback = self._step_feedback
self._step_feedback = ""
self._signal_received = False
if not feedback:
self._approved[phase] = self._current_output
break # move to next phase
else:
last_feedback = feedback
self._attempt_in_phase += 1 # regenerate same phase
One detail matters more than it looks: self._signal_received = False
is reset before await self._call_llm(...)
, not after. Signals delivered while the Activity is still executing (e.g., an eager reviewer clicking "approve" mid-generation) are not lost — wait_condition
picks up the flag the instant control returns to the workflow. A polling-based approval mechanism would have a real race window here; Temporal's Signal delivery guarantees there isn't one.
An empty-string feedback approves and advances; any non-empty string is treated as a rejection with revision instructions, and the same phase regenerates with that feedback injected into the next LLM request (SOPRequest.feedback
). This gives reviewers a single Signal type for two distinct actions, and gives the workflow author-level (not binary) revision control instead of just approve/reject.
Gate 4: PR approval, with a reject path. The final gate — before creating a GitHub PR — supports both approval and a full round-trip back into the autonomous fix loop:
@workflow.signal
def approve_pr(self) -> None:
self._pr_approved = True
@workflow.signal
def reject_with_feedback(self, input_data: dict) -> None:
self._human_feedback = (
input_data.get("comment", "") if isinstance(input_data, dict) else str(input_data)
)
if github_params.require_approval:
self._status = "awaiting_pr_approval"
self._pr_approved = False
await workflow.wait_condition(
lambda: self._pr_approved or bool(self._human_feedback)
)
if self._human_feedback:
continue # loop back to Phase 4 (autonomous_fix) with the feedback
self._status = "creating_pr"
pr_result = await self._call_github_pr(...)
wait_condition
here blocks on a disjunction of two Signals, which lets a single wait point resolve two different human outcomes without a secondary polling loop. A rejection doesn't just — it re-enters the fix loop with the human's comment injected as a first-class input alongside the validator's own failure list:
if human_feedback:
fix_result = await self._call_fix(final_sop, [], human_feedback)
Why not just a database flag checked by a cron job? Because that reintroduces every problem Temporal exists to solve: you'd need your own polling interval (latency vs. cost tradeoff), your own guarantee that a write isn't missed between poll cycles, and your own crash recovery for the poller itself. wait_condition
on a Signal is push-based, has no polling interval, and is backed by the same event history that makes the rest of the workflow durable — the wait is the crash-recovery mechanism, not a separate one.
Observability without extra infrastructure. Every phase and fix attempt — approved or not — is appended to self._history
, exposed via a @workflow.query
:
@workflow.query
def get_history(self) -> list:
"""全フェーズ・全試行の比較ログ。人間介入の Evidence として使用する。"""
return list(self._history)
This is the audit trail: every LLM output the workflow ever produced, whether a human approved or rejected it, and what feedback drove each regeneration — queryable at any point, including on a workflow that's still running, with no separate logging pipeline to maintain.
Stripped of the SOP-specific domain logic, the pattern is:
RetryPolicy
. The workflow function itself does nothing non-deterministic.workflow.wait_condition(lambda: flag)
combined with @workflow.signal
handlers gives push-based, race-free blocking that survives process restarts for however long a human takes — minutes or days, no difference to the implementation.kickoff()
, so retries are scoped to the agent that actually failed.workflow.patched()
, rather than silently breaking workflow instances that are mid-flight at deploy time.None of this is specific to SOP generation. The same shape applies to any pipeline where an LLM produces something a human must sign off on before it has real-world effect — code review automation, contract drafting, financial report generation — where "the process might be d for two days waiting on a person" is not a corner case to be handled, but the normal operating condition the architecture has to be built around from the start.
The code patterns discussed in this article were extracted from a production-ready SOP (Standard Operating Procedure) improvement platform built to demonstrate high-durability AI agents.
Note: I am a hands-on Japanese engineer. While the technical architecture and code in this article are 100% mine, I use AI to assist with English translation for global communication.
The complete, production-ready codebase with Docker Compose setup is fully documented and open-sourced here:
👉 GitHub: obataka/temporal-demo