{"slug": "the-post-delivery-gap-nobody-s-talking-about", "title": "The Post-Delivery Gap Nobody's Talking About", "summary": "A senior engineer with two years of experience running AI agents on personal infrastructure warns that the gap between a working demo and a production system is where AI projects fail, citing multiple incidents including a hallucination that changed a network device name during a bridge setup and an agent that skipped an entire directory during a 3TB media transfer without error. The engineer prescribes three operational safeguards: sandboxes with controlled blast radius, state validation beyond uptime, and human-in-the-loop as a breakpoint rather than a checkbox.", "body_md": "May 2026\n\nEvery AI project has a moment where the demo works, the stakeholders nod, and someone says \"great, now operationalize it.\"\n\nThat's where things fall apart.\n\nDemos work because they're small. A single task, a single input, a single output. No state to manage, no files to sync, no approvals to enforce. You're watching it work in a controlled environment where nothing else touches the result.\n\nProduction is the opposite of that.\n\nI've been running AI agents on my own infrastructure for two years. Not demos. Not POCs. Production systems that scrape data, manage knowledge bases, and run autonomously across multiple hosts. And the pattern I keep hitting is the same one every organization will face: **the gap between \"it works\" and \"it runs\" is where AI projects go to die.**\n\nHere's what I mean.\n\nAI changed a network device name during a bridge setup. Not a bug — a hallucination. It decided the name should be different, made the change, and moved on. I caught it by accident. If I hadn't been watching the logs, I would have spent hours debugging a problem that didn't exist until the agent invented it. The lesson: you need to be an active approver of every change, not a passive observer. If you're not in the loop, things go south.\n\nI asked AI to automate file transfers between hosts. It requested SSH access to a system already sharing via NFS and SMB. It reached for a working solution, not the simplest one. When I pointed out the logic flaw, it used the share correctly. But I had to be there to catch it.\n\nI was tailoring job applications. My agent altered the master CV and committed it to one repo without syncing the others. Two repos, two versions. My CV targeted \"sales\" roles for weeks before I noticed. That incident generated a new guardrail: every AI agent now has an \"update all repos\" rule. File sync between sandboxes isn't optional — it's a requirement you have to enforce.\n\nI captured a database state during a VM migration and never refreshed it. Two weeks of stale responses from my RAG pipeline before I caught it. AI did its part with the migration, but the data was in use by other agents and processes in flight. Testing isn't just up/down — state matters.\n\nAI consolidated 3TB of media from four hosts. Almost everything transferred. But it skipped an entire directory. No error. No warning. When I asked directly, the answer was: \"I thought it had already been copied.\" A hallucination — it invented a completion that never happened. No way to know without checking every folder manually.\n\nThese aren't hypotheticals. They're Tuesday.\n\nThe pattern is clear: **AI is good at the plan, good at the execution, but bad at the details that only matter after it ships.** And when you introduce hallucinations into a production environment with a loose approval process, the failure modes multiply. File paths, hostnames, states, sync — the operational reality that separates a demo from a production system.\n\nSo what do we do?\n\n**Three things I've learned:**\n\n1. **Sandboxes aren't optional.** Every AI agent needs a controlled blast radius. File system, container, or VM — the framework changes with use cases, but the principle doesn't. No AI model has ever been granted root access to bare metal on my infrastructure.\n\n2. **State matters more than uptime.** The VM came up. The migration succeeded. But the database was stale, the files were out of sync, and the agents were working from different versions of the truth. Testing needs to validate state, not just function.\n\n3. **Human-in-the-loop isn't a checkbox — it's a breakpoint.** When a switch drops a connection mid-deployment, AI asks me to pull up the console. The break-glass scenario always involves direct human interaction. Plan for it. Be an active approver, not a passive observer.\n\nThe organizations that will win with AI aren't the ones that build fastest. They're the ones that figure out the post-delivery gap before it becomes a production incident.\n\nWhat's the one operational detail your AI stack isn't telling you about?\n\n*Andrew Katana is a cloud transformation and AI infrastructure leader. He writes about the operational reality behind AI adoption — from RAG pipelines to production failures to what actually works. Follow at [atkatana.com/blog](https://atkatana.com/blog).*\n\nBuilt on a home lab, powered by local models, and owned by [Andrew Katana](https://linkedin.com/in/andrew-katana-615294).", "url": "https://wpnews.pro/news/the-post-delivery-gap-nobody-s-talking-about", "canonical_source": "https://atkatana.com/blog/post-delivery-gap.html", "published_at": "2026-07-24 08:54:53+00:00", "updated_at": "2026-07-24 09:22:37.554379+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-infrastructure", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/the-post-delivery-gap-nobody-s-talking-about", "markdown": "https://wpnews.pro/news/the-post-delivery-gap-nobody-s-talking-about.md", "text": "https://wpnews.pro/news/the-post-delivery-gap-nobody-s-talking-about.txt", "jsonld": "https://wpnews.pro/news/the-post-delivery-gap-nobody-s-talking-about.jsonld"}}