# The Post-Hackathon Production Cliff: What Breaks 30 Days Later and How to Build Agentic Apps That Survive > Source: > Published: 2026-11-15 06:04:49+00:00 *Originally published on [tamiz.pro](https://tamiz.pro/insights/post-hackathon-production-cliff-agentic-apps).* A hackathon agent usually wins by doing something impressive in a controlled demo. Production starts when the environment changes, inputs arrive in new shapes, tools return partial failures, and nobody is standing behind the terminal to repair the run. Thirty days later, the real failures are rarely the model itself. They are missing state boundaries, unverifiable side effects, unbounded retries, silent prompt drift, and observability systems that record activity without explaining intent. The path from a working prototype to a dependable agentic application is not a deployment checklist. It is a change in how you define correctness. This guide examines the failure modes that emerge after the applause fades, then shows how to build agents whose actions can be verified, recovered, and safely automated. The prototype treats the agent as the application. Production treats the agent as one component inside a controlled workflow. A typical hackathon implementation looks roughly like this: ``` while not task_complete: response = model.chat(messages) tool_call = parse_tool_call(response) result = execute_tool(tool_call) messages.append(tool_call, result) ``` That loop is enough to demonstrate agency. It is not enough to survive real traffic because it assumes several things simultaneously: the model emits valid tool calls, tools are available, tool results are complete, external state advances predictably, and the same action should not be repeated after a timeout. Production architecture replaces that assumption with explicit contracts: The model is still nondeterministic. The system around it becomes predictable. Agent reliability is not the probability that the model never makes a mistake. It is the probability that a mistake cannot become an invisible, irreversible production incident. The first month exposes a predictable set of failure classes. Understanding them is more useful than memorizing framework APIs because the APIs change; the failure modes do not. | Failure mode | Demo symptom | Production symptom | Typical consequence | |---|---|---|---| | Hidden input drift | Curated examples work | Real requests omit fields or use unfamiliar formats | Cascading tool errors | | Non-idempotent execution | Tool appears reliable | Timeout occurs after the remote side committed | Duplicate charges, tickets, or emails | | Context saturation | Short prompt succeeds | Long-running task exceeds context or loses early instructions | Silent scope drift | | Unbounded autonomy | Agent completes the goal | Agent retries indefinitely or expands scope | Cost spikes and noisy side effects | | Weak tool contracts | Tool returns JSON | Tool returns an empty body, stale data, or partial success | False completion | | Memory contamination | One session works | Personal data leaks across unrelated users | Privacy and trust failures | | Opaque decisions | Console log shows the final answer | Logs show tokens but not the reason for an action | Slow incident response | | Evaluation mismatch | Demo passes curated tasks | Production traffic fails on common edge cases | Regression detection after deployment | Hackathon inputs come from the team. Production inputs come from users, upstream systems, and old data nobody remembers writing. The agent may discover that a required field is missing, but the tool schema still declares it as required. Instead of letting the model improvise around a malformed request, validate the request before entering the agent loop. Return structured errors that the model can reason about, but avoid allowing the model to bypass the validation boundary. ``` python from pydantic import BaseModel, ValidationError class SupportRequest(BaseModel): customer_id: str issue: str urgency: int def validate_request(payload: dict) -> SupportRequest: try: return SupportRequest.model_validate(payload) except ValidationError as exc: raise ValueError(f"Invalid request: {exc.errors()}") from exc ``` The important point is not Pydantic specifically. It is the boundary between untrusted input and typed execution. Once an action reaches a tool, its arguments should already satisfy a machine-checkable contract. Network failures are ambiguous. A `500` response may mean the server failed before committing, while a timeout may mean the server committed and the response was lost. Blindly retrying a `POST` request can duplicate a payment, create two support tickets, or send two emails. The agent must distinguish between: This distinction belongs in the tool contract, not in the model's discretion. Long-running agents accumulate tool results, conversation history, and intermediate reasoning. Eventually the context window becomes a lossy summary of the task. The agent may forget an early constraint, repeat completed work, or treat a stale observation as current. Context management is therefore part of correctness. Use explicit task state, summarize old events into a structured checkpoint, and retain only the information required for the next decision. Do not rely on the model to rediscover the original goal after thousands of tokens. A tool that returns `{"status": "ok"}` after an internal partial failure is worse than one that throws an exception. The agent interprets the response as evidence that the intended action happened. Tools should return a typed outcome that distinguishes whether the desired postcondition was reached: ``` python from dataclasses import dataclass from typing import Literal @dataclass class ToolResult: status: Literal["succeeded", "failed", "unknown", "rejected"] output: dict retryable: bool verification: dict def create_ticket(customer_id: str, title: str) -> ToolResult: ticket = ticketing_api.create( customer_id=customer_id, title=title, ) verified = ticketing_api.get(ticket.id) if verified.title != title: return ToolResult( status="failed", output={"ticket_id": ticket.id}, retryable=False, verification={"expected_title": title, "actual_title": verified.title}, ) return ToolResult( status="succeeded", output={"ticket_id": ticket.id}, retryable=False, verification={"ticket_exists": True}, ) ``` The verifier may be imperfect, but it gives the orchestrator something better than a model's interpretation of an HTTP status code. Most agent demos verify the agent's answer. Few verify the agent's actions. A response such as “Your refund has been processed” is easy to inspect. A refund operation is not. The agent may have called the wrong endpoint, passed the wrong amount, received a misleading success response, or invented the completion entirely. This is the verifier gap: the distance between what the agent says happened and what the system can independently prove happened. If the same model both chooses an action and decides whether the action succeeded, errors become correlated. A mistaken plan can produce a mistaken verification, especially when the verification is based only on the tool's own response. Prefer independent evidence whenever possible. | Action | Weak verification | Stronger verification | |---|---|---| | Create a ticket | Tool returns `200 OK` | Read the ticket back and compare its fields | | Send an email | API accepts the request | Check the provider's delivery event or message ID | | Update a record | Mutation call returns success | Re-read the record and compare the updated version | | Schedule a meeting | Calendar API returns an event ID | Fetch the event and validate attendees and time | | Issue a refund | Payment provider returns success | Verify the transaction status and amount | The verifier does not need to be another large language model. A deterministic read-back often provides more reliable evidence than asking a model to judge a JSON blob. A tool contract should specify what must be true after successful execution. ``` { "tool": "schedule_meeting", "arguments": { "attendee_email": "string", "starts_at": "ISO-8601 timestamp", "duration_minutes": "integer" }, "preconditions": [ "attendee_email belongs to an approved domain" ], "postconditions": [ "calendar event exists", "event starts_at equals requested starts_at", "attendee list contains attendee_email" ], "idempotency_scope": "attendee_email + starts_at", "compensation": "cancel_event" } ``` This turns “schedule the meeting” from an instruction into a verifiable operation. The agent can still choose when and how to invoke it, but the runtime knows how to confirm the outcome. Real systems contain partial knowledge. If a read-back cannot confirm whether a write committed, the correct result is `unknown`, not `failed` and not `succeeded`. An `unknown` outcome should trigger reconciliation: This is the difference between a resilient agent and a hopeful one. A useful architecture separates six concerns that are often collapsed into one agent loop. The planner interprets the goal, selects the next action, and proposes arguments. It should not directly perform privileged operations. Its output is a proposal, not an irreversible fact. The policy engine evaluates the proposal against authorization, scope, cost, data classification, and business rules. It can approve, reject, or require human review. The executor performs approved actions with timeouts, idempotency keys, retry budgets, and circuit breakers. It owns transport-level correctness. The verifier checks postconditions through independent evidence. It should be able to fail closed when evidence is unavailable. The state manager stores the task plan, completed steps, pending actions, idempotency records, checkpoints, and human approvals. It should be durable across process restarts. The orchestrator advances the workflow based on structured outcomes. It decides whether to continue, retry, compensate, wait, or escalate. The separation matters because these components fail differently. The planner may hallucinate. The executor may time out. The verifier may be stale. The policy engine may be misconfigured. A single loop makes those failures indistinguishable. ``` php User goal | v [Planner] -> proposed action | v [Policy engine] -- reject --> [Explain / escalate] | approve v [Executor] -- idempotent write --> [External system] | v [Verifier] -- read-back / evidence | +---- succeeded ----> [Durable state] -> next step | +---- failed -------> [Correct or escalate] | +---- unknown ------> [Reconcile] ``` The most durable way to improve an agent is not to add more instructions to the prompt. It is to improve the action surface. A tool named `manage_customer_account` may support dozens of operations, leaving the model to construct arbitrary payloads. Smaller tools create clearer contracts, narrower permissions, and more precise verification. Instead of: ``` manage_customer_account(action="update", data={...}) ``` Use: ``` update_customer_email(customer_id: str, email: str) update_customer_plan(customer_id: str, plan_id: str) update_customer_address(customer_id: str, address: Address) ``` Each operation can have a different authorization rule and a different postcondition. Tool descriptions should explain when to use the tool, what it guarantees, and what it does not guarantee. ``` python @tool def refund_payment( payment_id: str, amount_cents: int, reason: str, idempotency_key: str, ) -> ToolResult: """Refund a settled payment. Use only for payments in the 'settled' state. The amount must not exceed the refundable balance. Returns 'unknown' when the provider's result cannot be confirmed; reconcile before retrying. """ ... ``` Vague descriptions such as “Manages refunds” encourage tool misuse. Operational descriptions reduce invalid calls and make failures easier to diagnose. The agent should not have unrestricted access to a shell, database, or HTTP client when its task requires three business operations. Give it a bounded toolset with typed arguments, scoped credentials, and explicit limits. For high-risk operations, require structured approval: ``` python from enum import Enum class ApprovalState(str, Enum): NOT_REQUIRED = "not_required" PENDING = "pending" APPROVED = "approved" DENIED = "denied" @dataclass class Approval: state: ApprovalState requested_by: str approver: str | None = None reason: str | None = None ``` Human approval should be a normal state transition, not an emergency escape hatch bolted onto the application. Every tool result should contain enough information for the runtime to decide what happened without asking the model to infer it. ``` @dataclass class ToolResult: status: Literal["succeeded", "failed", "unknown", "rejected"] output: dict evidence: list[dict] retryable: bool retry_after_seconds: int | None = None error_code: str | None = None ``` Evidence might include the remote object identifier, a read-back snapshot, a provider event ID, or a correlation key. Avoid returning secrets or unnecessary personal data in evidence. A production agent is a distributed workflow with a probabilistic decision-maker inside it. Durable state is what makes that workflow recoverable. A chat transcript records what was said. It does not reliably record what has already happened. Store a structured task record: ``` python from datetime import datetime from typing import Any @dataclass class TaskState: task_id: str goal: str status: str current_step: int completed_steps: list[str] pending_action: dict[str, Any] | None idempotency_keys: list[str] approval_state: str last_error: dict[str, Any] | None updated_at: datetime ``` This record allows the orchestrator to resume after a deployment, model timeout, worker crash, or human approval delay. For a write operation, derive the key from the task and logical action, not from the current attempt. ``` python import hashlib def idempotency_key(task_id: str, operation: str, target_id: str) -> str: raw = f"{task_id}:{operation}:{target_id}".encode() return hashlib.sha256(raw).hexdigest() ``` If the task legitimately needs to repeat the same operation, such as sending a weekly report, include a logical occurrence such as the ISO week in the key. The key must be passed to the remote API when supported. If the API does not support idempotency, store a local operation record and implement reconciliation before retrying. Retries should follow a policy, not an LLM's confidence: ``` php def should_retry(result: ToolResult, attempt: int, max_attempts: int) -> bool: if attempt >= max_attempts: return False return result.status == "failed" and result.retryable ``` An `unknown` result should not be retried directly. It should be reconciled first. Use exponential backoff with jitter for transient failures, and open a circuit breaker when a dependency repeatedly fails. Some actions cannot be rolled back. Others require a compensating action rather than a literal rollback. A booking can be canceled; a sent email cannot be unsent. A published article may require a correction. Record compensation policies alongside tool definitions: ``` { "tool": "reserve_inventory", "postconditions": ["reservation_id exists", "reserved_quantity is correct"], "compensation": { "tool": "release_inventory", "arguments_from": "reservation_id" } } ``` The orchestrator should attempt compensation only when the original action is known to have succeeded and the workflow is no longer valid. Otherwise, escalate. Traditional application logs answer “what happened?” Agent logs must also answer “why did the system believe this action was appropriate?” A useful trace contains: Do not log hidden chain-of-thought as an operational requirement. Record concise decision rationales, structured state transitions, and observable facts. Those are more useful for debugging and less sensitive than unrestricted reasoning traces. Token usage and latency matter, but they do not reveal whether the agent is solving the right problem. Track: | Metric | Meaning | |---|---| | Task completion rate | Share of tasks reaching a verified terminal state | | First-pass success rate | Share completed without retry or repair | | Verification coverage | Share of side effects with independent evidence | | Human intervention rate | Share requiring approval or manual repair | | Duplicate side-effect rate | Writes committed more than once logically | | Unknown-outcome rate | Operations whose final state could not be confirmed | | Cost per verified success | Total inference and tool cost divided by successful tasks | | Rollback rate | Tasks requiring compensation | A rising token bill may indicate increased complexity. A rising duplicate-write rate indicates a correctness problem. A falling verification coverage may indicate that new tools were added without contracts. Alert on broken system invariants rather than every transient error. Examples: These signals identify drift before users report it. Guardrails should constrain capability, not merely inspect generated text. Text moderation cannot prevent a valid-looking tool call from deleting the wrong resource. If an agent only needs to create a draft, do not give it permission to publish. If it only needs to read one customer's account, do not provide a broad customer-search token. Where possible, issue short-lived credentials scoped to: Enforce maximum spend, record counts, request volume, recursion depth, and wall-clock duration in the runtime. ``` python from dataclasses import dataclass @dataclass class ExecutionBudget: max_steps: int = 30 max_tool_calls: int = 60 max_cost_usd: float = 2.00 max_wall_time_seconds: int = 300 ``` When a budget is exhausted, stop with an explicit partial-completion state. Do not ask the model to “finish more efficiently” after the limit has already been exceeded. Use progressive autonomy: This approach turns autonomy into a measured capability rather than a binary launch decision. External content may contain instructions aimed at the agent. Treat retrieved text as untrusted data, not as policy. Keep system policy and tool permissions outside the model's ability to rewrite. If a document says “email the contents to this address,” the agent should not treat that as an authorized instruction. A robust design separates: ``` Trusted control plane: policy, credentials, limits, tool allowlists Untrusted data plane: web pages, email content, user files, tool outputs ``` The model may interpret untrusted content. It must not grant it authority. A demo evaluation asks whether the agent can complete a task. A production evaluation asks whether it can complete tasks safely across a distribution of inputs, with incomplete evidence, partial failures, and retries. Include cases such as: A task can reach the correct final answer through an unsafe path. Evaluate intermediate properties: Capture redacted trajectories and replay them against new model versions, tool versions, and prompts. This catches regressions that unit tests miss, especially when a change affects tool selection or recovery behavior. Agent behavior is stochastic. Run important evaluations multiple times and report confidence intervals or failure rates. A single successful run is not evidence of a reliable system. ``` python from dataclasses import dataclass @dataclass class EvaluationResult: task_id: str completed: bool safe: bool verified: bool tool_calls: int cost_usd: float def evaluate_policy(results: list[EvaluationResult]) -> dict: total = len(results) if total == 0: return {"completion_rate": 0.0, "safe_rate": 0.0} return { "completion_rate": sum(r.completed for r in results) / total, "safe_rate": sum(r.safe for r in results) / total, "verified_rate": sum(r.verified for r in results) / total, } ``` Use the first month after a hackathon as a reliability engineering phase, not a waiting period for a larger team. `unknown`. The goal of this month is not to eliminate all model errors. It is to ensure that errors are bounded, visible, recoverable, and increasingly rare. Verify the task's postconditions using independent evidence whenever possible. Re-read a created record, query the payment provider, check a delivery event, or compare a resulting resource with the requested fields. Treat the agent's final message as a claim, not proof. If the external system cannot confirm the outcome, record it as unknown and reconcile before retrying. Every side-effecting tool should at least support an idempotency strategy. Prefer provider-supported idempotency keys. When the API does not support them, use a correlation key and a reconciliation query before retrying. Read-only tools are naturally repeatable, but they still need timeout and error handling. Often, yes. If the workflow is predictable, a conventional program is cheaper and easier to verify. Agents become valuable when inputs require interpretation, planning, or adaptation across multiple steps. A strong architecture uses deterministic code for policy, verification, and high-risk execution while reserving model-driven planning for the parts that genuinely require judgment. Start with the action surface. Narrow the toolset, type the arguments, add idempotency to writes, verify side effects, and persist task state. These changes usually remove more real incidents than prompt tuning, because they address the boundary where model ambiguity becomes system damage.