{"slug": "the-ai-demo-worked-heres-what-production-will-ask-for", "title": "The AI Demo Worked. Here’s What Production Will Ask For.", "summary": "A developer's account of moving AI projects from demo to production argues that the questions a demo answers (\"Can this work?\") differ from those a production system must answer, and lays out the checks that separate the two: testing against unfiltered real data with failures grouped by type, maintaining a fixed evaluation suite rerun after every prompt and model change, enforcing the same access rules as the underlying data store, and defending against prompt injection architecturally rather than with filters. For AI agents touching financial or tokenized assets, the piece contends compliance cannot be probabilistic, pointing to standards such as ERC-3643 that enforce KYC/AML eligibility at the smart contract level, scoped session keys with spending limits, and master keys held in HSMs or MPC custody.", "body_md": "The Demo Answered a Different Question\n\nAlmost every AI initiative has a moment where the demo lands. The assistant answers the question, the extraction pulls the right fields, the agent completes the task. The room is convinced.\n\nThen the project stalls somewhere between that meeting and a system people rely on every day.\n\nThis usually isn’t because the demo was misleading. A demo answers “Can this work?” A product has to answer “Will this keep working, for everyone, on every input, when someone is actively trying to break it?” Those are different questions, and the second one is where most of the engineering lives.\n\nIf you’re evaluating an AI build, or evaluating a vendor proposing one, these are the questions production will ask that the demo never had to.\n\nDoes It Work on Your Real Data?\n\nDemos run on examples someone picked. Production runs on whatever your systems contain: scanned PDFs, half filled forms, records in three languages, fields that contradict each other.\n\nThe fix is unglamorous. Before approving a build, ask for the prototype to be run against an unfiltered sample of your actual data. Then read the failures, grouped by type. One accuracy number hides the problem; a list of failure categories tells you what the build will really involve.\n\nDoes It Behave the Same Way Tomorrow?\n\nLanguage models are nondeterministic, and model providers update and retire versions on their own schedules. A prompt that behaves well this week can drift after an upgrade you didn’t choose.\n\nProduction AI needs an evaluation suite: a fixed set of cases with expected behavior, rerun after every prompt change and every model change. If a team or vendor can’t describe how they’ll detect a regression, they’re planning to find out from your users.\n\nDoes It Respect Who Is Allowed to See What?\n\nA demo usually has one user and full access to everything. A real system has roles, departments, and documents some people must never see.\n\nIf an AI assistant can summarize a file, it must apply the same access rules as the system that stores the file. Otherwise, you’ve built a polite way around your own permissions. This is one of the most common gaps in early AI builds, and one of the most damaging when found late.\n\nWhat Happens When Someone Attacks It?\n\nDemo audiences are friendly. Production users include people probing for weaknesses.\n\nPrompt injection is the clearest example. Instructions hidden in an email, a web page, or an uploaded document can redirect what a model does. Because instructions and data share the same channel, no filter fully solves this. The dependable defense is architectural:\n\nGive the AI only the permissions it needs for the task.\n\nRequire human approval for anything consequential or irreversible.\n\nLog every action it takes, so incidents can be traced.\n\nTreat model output as untrusted input to every downstream system.\n\nWhen the AI Can Move Assets, Compliance Cannot Be Probabilistic\n\nThis matters most when AI agents interact with financial systems or tokenized assets.\n\nA model can be asked to only transfer tokens to verified investors. It will usually comply. “Usually” is not a compliance posture. Rules like KYC/AML checks and transfer restrictions belong in deterministic code. Standards such as ERC-3643 enforce eligibility at the smart contract level, so a transfer to an unverified wallet fails regardless of what the agent decides. The AI proposes; the contract enforces.\n\nThe same thinking applies to keys. An agent should never hold an unrestricted private key. Use scoped session keys, spending limits, and allowlisted contract calls, with master keys held in proper custody (HSMs or MPC setups) and controlled by the asset owner. Any contract an agent can call deserves an independent security audit, because the agent widens who and what can reach that contract.\n\nWhen Blockchain Is the Wrong Tool Here\n\nA common suggestion is to put AI decisions on-chain for transparency. Usually, this is unnecessary.\n\nIf your goal is an audit trail of what an AI system did, an append-only log with signed entries in a database you control is cheaper, faster, and easier to query. A blockchain earns its place only when multiple parties who don’t trust each other need to verify the same record without relying on one operator.\n\nThere is also a deeper limitation: writing a model’s output to a chain records that the output existed. It does not make the output correct. An AI-generated valuation stored on-chain is still an AI-generated valuation. If anything, permanence raises the cost of being wrong.\n\nCount the Cost at Real Volume\n\nA demo that takes a few seconds and costs a fraction of a cent per request feels fine. Multiply by real users and real workflows, and latency and spend become design constraints.\n\nPlan for caching repeated context, routing simple requests to smaller models, and setting budgets per workflow. These decisions are much cheaper before launch than after the first surprise invoice.\n\nScope the First Phase Honestly\n\nThe most useful thing a development partner can do is tell you which parts of the demo are close to production and which are not. A good first phase is narrow: one workflow, real data, an evaluation suite, defined permissions, and a named owner for quality after launch.\n\nSometimes the honest answer is that a demo is enough. Internal tools, low-stakes tasks, and workflows where a person reviews every output often don’t need a full production build. And if an off-the-shelf product already covers your use case under acceptable data terms, buying it is often the better decision.\n\nOwn What You Build\n\nIf you do build, make sure you own the result. That means the code, the prompts, the evaluation suite, the integration logic, and every key that controls assets. Evaluation suites in particular are easy to overlook, but they are what lets you switch models or vendors later without starting over.\n\nA build-to-own model keeps your options open as AI tooling keeps changing.\n\nRWaltz is a technology partner building blockchain, AI, and enterprise software, including smart contracts, tokenization platforms, and custom integrations. Our approach is security first and build-to-own: honest scoping, independent audits, and IP and keys that stay with the client.\n\n📖 Read the full blog: [https://www.rwaltz.com/blogs/why-ai-demos-work-and-ai-products-dont-what-changes-in-production](https://www.rwaltz.com/blogs/why-ai-demos-work-and-ai-products-dont-what-changes-in-production)\n\nConnect with RWaltz:\n\nLinkedIn: [https://www.linkedin.com/company/rwaltzsoftware](https://www.linkedin.com/company/rwaltzsoftware)\n\nX (Twitter): [https://twitter.com/rwaltzsoftware](https://twitter.com/rwaltzsoftware)\n\nFacebook: [https://www.facebook.com/RWaltz-Software-PvtLtd-255590135349493](https://www.facebook.com/RWaltz-Software-PvtLtd-255590135349493)\n\nTelegram: [https://t.me/RWaltzCrypto](https://t.me/RWaltzCrypto)\n\nGitHub: [https://github.com/rwaltzsoftware](https://github.com/rwaltzsoftware)\n\nClutch: [https://clutch.co/profile/rwaltz-software](https://clutch.co/profile/rwaltz-software)\n\nWebsite: [https://www.rwaltz.com](https://www.rwaltz.com)", "url": "https://wpnews.pro/news/the-ai-demo-worked-heres-what-production-will-ask-for", "canonical_source": "https://dev.to/rwaltzsoftware/the-ai-demo-worked-heres-what-production-will-ask-for-1njk", "published_at": "2026-10-15 13:12:40+00:00", "updated_at": "2026-10-15 13:20:20.261515+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "large-language-models", "mlops"], "entities": ["ERC-3643"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-ai-demo-worked-heres-what-production-will-ask-for", "markdown": "https://wpnews.pro/news/the-ai-demo-worked-heres-what-production-will-ask-for.md", "text": "https://wpnews.pro/news/the-ai-demo-worked-heres-what-production-will-ask-for.txt", "jsonld": "https://wpnews.pro/news/the-ai-demo-worked-heres-what-production-will-ask-for.jsonld"}}