A demo agent has one user, one task, and a forgiving audience. A production agent has a queue, a bill, a security boundary, and a hostile network. The three signals I keep running into — sites actively blocking agents, research pipelines fanning out thousands of agents at once, and new filesystems built specifically for coding agents — are all symptoms of the same shift: agents are now network citizens with state, cost, and blast radius.
Here is the production view I wish I'd had before shipping my first one.
#
The web is now an adversarial environment
The Verge's piece on sites closing their doors to agents is not a policy story, it is an infrastructure story. Cloudflare bot management, robots.txt enforcement, pay-per-crawl, and login walls are all live. Your agent will get 403, 429, and a CAPTCHA where the demo got HTML.
Design for it:
- Prefer documented APIs with keys over scraping. If there is no API, ask whether the task is worth a terms-of-service fight.
- Handle
403and429differently.403is a policy stop; retrying makes you a bad actor.429is backoff with jitter, andRetry-Afteris a real header. - Put a per-domain circuit breaker in front of
fetch. One broken site should not stall a 500-item batch. - Cache aggressively and key the cache on the URL plus a content hash. Re-fetching the same page per agent is how you get blocked.
If you build on other people's content, assume the rules can change weekly. A tool that returns "blocked" is a valid result your loop must handle, not an exception to swallow.
#
Fan-out is a queueing problem, not a prompting problem
OpenAI's biological-data work with thousands of agents is impressive, but the engineering is unglamorous: a work queue, idempotency, deduplication, and a budget per unit of work. The same applies if you fan out 50 agents over your own backlog.
What I've seen break:
Duplicate work. Two workers claim the same item and both write the same row. Use an idempotency key derived from the input, not from a random UUID generated inside the agent. #
Retry storms. A non-idempotent side effect (sending email, creating a ticket) retried after a timeout. Make side effects idempotent at the tool boundary, or make the tool take a dedup key. #
Unbounded spawning. An agent that can call a "spawn sub-agent" tool will do it. Cap depth and total descendants per root task. #
No backpressure. A queue with no concurrency limit turns a provider rate limit into a wall of429 s. Setmax_workers and let the queue grow.
Temporal, Celery, and SQS all work here. The choice matters less than the invariant: every unit of work can be retried safely and has a hard cost ceiling.
#
Give agents a workspace, not a conversation
Coding agents are the clearest case. Context windows are a bad database. A filesystem gives you durable state, diffable artifacts, and tools the agent already understands: rg, git, pytest.
Projects like LayerFS are interesting because they treat the agent's workspace as a first-class primitive rather than a side effect of a chat transcript. You can get most of the benefit without a new product:
- One scratch directory per run, outside the repo.
- One
git worktreeper parallel agent so two agents never fight over the same files. - Artifacts (logs, diffs, generated files) written to disk and referenced by path in the transcript, not pasted into the prompt.
- A final
git diffas the review surface. If the diff is empty, the agent did nothing, regardless of what it said.
#
A bounded loop you can actually ship
The single most useful thing I've added to agent code is a hard stop: step limit, token budget, and per-tool error handling that feeds failures back into the loop instead of crashing it. This is complete enough to run, with your own tools swapped in.
Two details matter more than they look. The run_id comes from the caller, so a retried job reuses the same key. Tool errors are returned to the model as JSON, which is usually enough to recover; a thrown exception just kills a run that was one step from finishing.
#
Observe the trajectory, not the answer
Logging the final string tells you nothing when quality drops. Emit one structured event per step: model name, prompt template version, step index, tool name, arguments hash, latency, token count, and outcome. OpenTelemetry spans work; Langfuse, LangSmith, and Arize Phoenix are built for this shape of data.
What I actually query:
- Distribution of step counts. A spike means the prompt or a tool regressed.
- Tool error rate by tool name. A schema change upstream shows up here first.
- Token spend per successful task, broken down by tenant.
- Runs that hit the step limit. These are silent failures, not successes.
#
Pitfalls
Indirect prompt injection. Any fetched page, issue, or email can contain instructions. Treat tool output as untrusted data. Keep secrets out of the context window and require explicit approval for destructive tools. #
Tool schema drift. The model's tool definition and the real API diverge; the model keeps calling a field that no longer exists. Version your tool schemas and test them. #
Silent partial success. The agent summarizes three of five documents and calls it done. Verify with a deterministic check (row count, diff, schema validation), not with the model's own claim. #
Context rot. Long transcripts degrade instruction following. Compress or checkpoint at a fixed token boundary rather than waiting for the window to fill. #
Secrets in logs. Traces capture arguments. Redact before the span leaves the process. #
No kill switch. You need a way to stop all runs for one tenant or one tool without a deploy.
#
Monday checklist
Ship only when every line is true.
- [x] Every tool with a side effect takes an idempotency key, and retries reuse it.
- [x] Every run has a hard step limit and token budget, enforced in code.
- [x]
403,429, timeouts, and empty pages each have a distinct handler, not one generic retry. - [x] Sub-agent spawning is capped by depth and total descendants.
- [x] Parallel agents get isolated workspaces (
git worktreeor equivalent), not a shared directory. - [x] Traces log step index, tool name, latency, tokens, and outcome, with arguments redacted.
- [x] Success is verified by a deterministic check, not by the agent's summary.
- [x] Untrusted content is labelled as data in the prompt and cannot trigger destructive tools.
- [x] There is a per-tenant kill switch that does not require a deploy.
- [x] You have looked at the step-count and tool-error dashboards in the last seven days.