{"slug": "four-questions-to-ask-before-you-build-an-ai-product", "title": "Four questions to ask before you build an AI product", "summary": "An engineer outlined four pre-build questions for teams taking generative AI from demo to production: assemble a version-controlled test set of roughly 50 real cases and score every change against it in CI, gate risky agent tool calls behind human approval and audit logging, measure cost per task rather than the monthly pilot bill, and assign a weekly owner to catch model and prompt drift. The guidance cites MIT NANDA's 2025 \"The GenAI Divide\" study, which found 95% of organizations saw no measurable return from generative AI pilots, and argues the model is rarely the reason.", "body_md": "Building an AI demo takes a weekend now. Getting it into production is where most projects stop.\n\nA demo runs on a few clean examples, chosen by the people who built it. A product runs on whatever your customers send it, at three in the morning. MIT NANDA's 2025 study, \"The GenAI Divide: State of AI in Business 2025\", found that 95% of organisations were getting no measurable return from their generative AI pilots. The model was rarely the reason. The gap is everything around it.\n\nWe made a short film on this for founders and product leads deciding whether to build:\n\nThis post is the engineering version. Four questions, and for each one the thing to build before you write the feature.\n\nNot on a public benchmark, and not on the vendor's examples. On your documents, your odd cases, your mess.\n\nSo, before any prompt or model choice, collect real cases. Fifty is enough to start. Take them from support tickets, past documents, the requests that made someone sigh. Write down what a good answer looks like for each, and keep the set in version control next to the code.\n\nThen score every change against it. Here is the smallest harness that does the job:\n\n``` python\nimport json\n\ndef load_cases(path):\n    with open(path) as f:\n        return [json.loads(line) for line in f]\n\ndef score(case, output):\n    # Start with checks you can automate: required facts present,\n    # forbidden content absent, valid JSON, within length.\n    required = all(fact.lower() in output.lower() for fact in case[\"must_include\"])\n    forbidden = any(bad.lower() in output.lower() for bad in case.get(\"must_not_include\", []))\n    return required and not forbidden\n\ndef run_eval(system, cases):\n    results = [(case[\"id\"], score(case, system(case[\"input\"]))) for case in cases]\n    passed = sum(ok for _, ok in results)\n    print(f\"{passed}/{len(results)} passed\")\n    for case_id, ok in results:\n        if not ok:\n            print(\"FAIL\", case_id)\n    return passed / len(results)\n```\n\n`system` is whatever you are testing: a prompt, a chain, an agent. Run it in CI. A change that lowers the pass rate does not ship, however good the demo looks.\n\nEvery model is wrong some of the time. That is fine for a draft a person reads. An agent connected to real systems is different. Its mistakes act.\n\nDecide three things before it gets access:\n\nA simple pattern is a tool wrapper that checks a policy before it runs anything:\n\n``` python\nAPPROVAL_REQUIRED = {\"refund\", \"delete_record\", \"send_email\"}\n\ndef call_tool(name, args, approved=False):\n    if name in APPROVAL_REQUIRED and not approved:\n        return queue_for_review(name, args)   # a person decides\n    result = TOOLS[name](**args)\n    audit_log(name, args, result)             # enough to undo it\n    return result\n```\n\nA pilot that costs pennies can cost more than the work it replaces once everyone uses it. The pilot bill hides this. Twenty testers never hit the long documents, the retries or the follow-up questions.\n\nMeasure the cost of each task from day one, not the monthly invoice:\n\n``` python\ndef cost_per_task(tokens_in, tokens_out, price_in_per_m, price_out_per_m, calls=1):\n    return calls * (tokens_in * price_in_per_m + tokens_out * price_out_per_m) / 1_000_000\n```\n\nLog it per request, then multiply by the volume you expect in a year. Then use the smallest model that still passes your test set from question 1. The test set is what lets you switch models with confidence instead of hope.\n\nYour data changes. Providers update their models, and sometimes withdraw them. Prompts that passed in March can fail in June with no change on your side.\n\nSomeone must own this every week, for as long as the system runs:\n\nBuy when the job is common and an existing tool already does it well enough. Build when the AI is the product, or when it depends on data and decisions only you have.\n\nEither way, budget for the gap, not for the demo. The four pieces above (a test set, limits on what it can do, cost per task, and an owner) are most of that gap.\n\nAt Fountech, that gap is our work, for the AI we build for clients and for our own ventures. If you are taking something from demo to production, you can start a project with us here: [https://fountech.ai/?utm_source=devto&utm_medium=post&utm_campaign=fountech-2026-10](https://fountech.ai/?utm_source=devto&utm_medium=post&utm_campaign=fountech-2026-10)", "url": "https://wpnews.pro/news/four-questions-to-ask-before-you-build-an-ai-product", "canonical_source": "https://dev.to/nikkairinos/four-questions-to-ask-before-you-build-an-ai-product-39mf", "published_at": "2026-10-22 08:31:19+00:00", "updated_at": "2026-10-22 08:40:26.595209+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "artificial-intelligence"], "entities": ["MIT NANDA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/four-questions-to-ask-before-you-build-an-ai-product", "markdown": "https://wpnews.pro/news/four-questions-to-ask-before-you-build-an-ai-product.md", "text": "https://wpnews.pro/news/four-questions-to-ask-before-you-build-an-ai-product.txt", "jsonld": "https://wpnews.pro/news/four-questions-to-ask-before-you-build-an-ai-product.jsonld"}}