Collapsing four hallucinating LLM orchestrators into zero tokens — and the two bugs the migration found

JobEmber's agentic pipeline had four batch orchestrators — each taking a list of N items, fanning out N parallel LLM worker agents, optionally reducing the workers' outputs. Three lived in TypeScript via Promise.allSettled with N orphan run-ids and no cascading cancel; one had been lifted into the runtime as an LLM agent with a careful system prompt instructing it to "fire all N spawns in ONE iteration." The orchestration work itself was deterministic — partition a list of job sites round-robin into N slices, chunk a list of matches into batches, wrap each item in a worker prompt, spawn one child per slice, collect results — yet job-search-batch was burning ~8,000 tokens per run to do round-robin partitioning, and the weak-tier model occasionally violated its own prompt and serialized the workers it was supposed to fan out. v0.20.0's inline code_body ingestion (the previous post) made the fix viable: replace all four orchestrators with deterministic code-js agents whose bodies ride through AgentDef like any other agent attribute. Result: zero tokens for the orchestration layer, replay-deterministic by construction (pure function of input.metadata + recorded tool results), one run-id to monitor and cancel as a unit (cancel cascades to children via loomcycle's multi-replica cancel primitive), Scheduler-fireable because the orchestrators now live in the runtime, and the "FIRE ALL N SPAWNS" hallucination goes away because JS doesn't have an opinion about how to dispatch — it calls Agent.parallel_spawn every time. The four orchestrators converted: employer-research-batch (was TS Promise.allSettled, now code-js fire-and-forget — children self-ingest via postResearchIngest); cv-cl-batch (was TS, now code-js fire-and-forget — children self-ingest via patchApplication); ats-filter-batch (was TS chunked map, now code-js map-reduce reducer — chunks matches, fans out filter workers, robustly parses each child's output via an ES5 port of parseAgentJSON, flattens + dedups verdicts); job-search-batch (the LLM agent, ~8K tokens/run → code-js — round-robin partition of jobSites[] into N slices, spawn one job-searcher per slice). Migration surfaced two latent loomcycle bugs, both shipped in v0.21.0. Bug #1 (PR #359): code-js wall-clock budget was 120 seconds, sized for CPU-bound JS that runs to completion in goja without I/O — but a fan-out orchestrator parks for minutes in Agent.parallel_spawn awaiting LLM children. The wall-clock budget keeps ticking; resume turns started already over-budget; the runtime interrupted the next interruptible bytecode (typically parseBatch or whatever was running when the deadline expired); the error message named that line, blaming entirely innocent code. Worse, interruptWatch's TIMER branch fired rt.Interrupt(context.DeadlineExceeded) WITHOUT cancelling the parent ctx, so classifyRunErr (which only special-cased ctx.Err()!=nil) fell to the default and emitted code_agent_threw instead of a distinct timeout error. Fix: replayState.timedOut atomic set before the interrupt; classifyRunErr emits a distinct code_agent_timeout class stating the budget and pointing at the override knobs, separate from code_agent_cancelled (ctx cancel) and code_agent_threw (real JS throw). Per-agent run_timeout_seconds on AgentDef (operational, not in content_sha256, mirrors retry_attempts) and per-run run_timeout_seconds on /v1/runs + /v1/sessions/{id}/messages. pickRunTimeout resolves per-run > per-agent > global default. Threaded through all four loop.Run sites including runSubAgent (caught in self-review — the one site that almost shipped without the override). JobEmber's batchRunTimeoutSeconds() scales the budget with the fan-out width (ceil(N/concurrency) waves, clamped to [180s, 1800s]). Bug #2 (PR #366): the replay model rebuilds the goja runtime each turn and re-converts the run's metadata (a Go map[string]any) into input.metadata via rt.ToValue. Go deliberately randomizes map iteration order per access. SAME metadata → JS object with DIFFERENT key order on each turn. An agent that does JSON.stringify(input.metadata.matches) into a tool_use input emitted byte-different bytes turn-1 vs replay, tripping a spurious code_agent_replay_divergence on tool call #0. Observed exactly once, in ats-filter-batch (the only orchestrator that serializes objects in JS — the other three pass pre-built prompt strings through metadata). LOOMCYCLE_CODE_AGENTS_DETERMINISTIC=1 does NOT fix this; it pins only the RNG seed + clock anchor, not Go-map key order. Fix: stableJSValue() recursively materializes every Go map as a JS object with sorted keys; arrays keep their order. JS objects are insertion-ordered, so sorted insertion yields sorted iteration AND sorted JSON.stringify. Reserved-key precedence (user_id/agent) unchanged. JobEmber had to ship a defensive fixed-key normalizeMatch workaround before the loomcycle fix landed; deleting normalizeMatch after #366 is itself the soundness test of the fix — replay staying divergence-free without the workaround is the test that the patch is correct. Why both bugs were hidden behind the LLM-orchestrator path: an LLM agent doesn't park in Agent.parallel_spawn synchronously (tool calls are issued by the model, the loop's outer ctx covers wall-clock — the code-js inner budget never fired), and an LLM agent never serializes input.metadata in JS (it has no JS at all). Both bugs were implied by the deployment shape the LLM-orchestrator path had been hiding. The pattern worth taking forward: if a step in your agentic pipeline can be expressed as a 30-line deterministic function, it should not be an LLM agent. Routing, partitioning, chunking, reducing, formatting — these are not language tasks. Asking an LLM to do them costs tokens, introduces non-determinism, and occasionally surfaces in a model "creatively" reinterpreting its instructions.