Four questions to ask before you build an AI product

An engineer outlined four pre-build questions for teams taking generative AI from demo to production: assemble a version-controlled test set of roughly 50 real cases and score every change against it in CI, gate risky agent tool calls behind human approval and audit logging, measure cost per task rather than the monthly pilot bill, and assign a weekly owner to catch model and prompt drift. The guidance cites MIT NANDA's 2025 "The GenAI Divide" study, which found 95% of organizations saw no measurable return from generative AI pilots, and argues the model is rarely the reason.

Building an AI demo takes a weekend now. Getting it into production is where most projects stop. A demo runs on a few clean examples, chosen by the people who built it. A product runs on whatever your customers send it, at three in the morning. MIT NANDA's 2025 study, "The GenAI Divide: State of AI in Business 2025", found that 95% of organisations were getting no measurable return from their generative AI pilots. The model was rarely the reason. The gap is everything around it. We made a short film on this for founders and product leads deciding whether to build: This post is the engineering version. Four questions, and for each one the thing to build before you write the feature. Not on a public benchmark, and not on the vendor's examples. On your documents, your odd cases, your mess. So, before any prompt or model choice, collect real cases. Fifty is enough to start. Take them from support tickets, past documents, the requests that made someone sigh. Write down what a good answer looks like for each, and keep the set in version control next to the code. Then score every change against it. Here is the smallest harness that does the job: python import json def load cases path : with open path as f: return json.loads line for line in f def score case, output : Start with checks you can automate: required facts present, forbidden content absent, valid JSON, within length. required = all fact.lower in output.lower for fact in case "must include" forbidden = any bad.lower in output.lower for bad in case.get "must not include", return required and not forbidden def run eval system, cases : results = case "id" , score case, system case "input" for case in cases passed = sum ok for , ok in results print f"{passed}/{len results } passed" for case id, ok in results: if not ok: print "FAIL", case id return passed / len results system is whatever you are testing: a prompt, a chain, an agent. Run it in CI. A change that lowers the pass rate does not ship, however good the demo looks. Every model is wrong some of the time. That is fine for a draft a person reads. An agent connected to real systems is different. Its mistakes act. Decide three things before it gets access: A simple pattern is a tool wrapper that checks a policy before it runs anything: python APPROVAL REQUIRED = {"refund", "delete record", "send email"} def call tool name, args, approved=False : if name in APPROVAL REQUIRED and not approved: return queue for review name, args a person decides result = TOOLS name args audit log name, args, result enough to undo it return result A pilot that costs pennies can cost more than the work it replaces once everyone uses it. The pilot bill hides this. Twenty testers never hit the long documents, the retries or the follow-up questions. Measure the cost of each task from day one, not the monthly invoice: python def cost per task tokens in, tokens out, price in per m, price out per m, calls=1 : return calls tokens in price in per m + tokens out price out per m / 1 000 000 Log it per request, then multiply by the volume you expect in a year. Then use the smallest model that still passes your test set from question 1. The test set is what lets you switch models with confidence instead of hope. Your data changes. Providers update their models, and sometimes withdraw them. Prompts that passed in March can fail in June with no change on your side. Someone must own this every week, for as long as the system runs: Buy when the job is common and an existing tool already does it well enough. Build when the AI is the product, or when it depends on data and decisions only you have. Either way, budget for the gap, not for the demo. The four pieces above a test set, limits on what it can do, cost per task, and an owner are most of that gap. At Fountech, that gap is our work, for the AI we build for clients and for our own ventures. If you are taking something from demo to production, you can start a project with us here: https://fountech.ai/?utm source=devto&utm medium=post&utm campaign=fountech-2026-10 https://fountech.ai/?utm source=devto&utm medium=post&utm campaign=fountech-2026-10