Building an AI demo takes a weekend now. Getting it into production is where most projects stop.
A demo runs on a few clean examples, chosen by the people who built it. A product runs on whatever your customers send it, at three in the morning. MIT NANDA's 2025 study, "The GenAI Divide: State of AI in Business 2025", found that 95% of organisations were getting no measurable return from their generative AI pilots. The model was rarely the reason. The gap is everything around it.
We made a short film on this for founders and product leads deciding whether to build:
This post is the engineering version. Four questions, and for each one the thing to build before you write the feature.
Not on a public benchmark, and not on the vendor's examples. On your documents, your odd cases, your mess.
So, before any prompt or model choice, collect real cases. Fifty is enough to start. Take them from support tickets, past documents, the requests that made someone sigh. Write down what a good answer looks like for each, and keep the set in version control next to the code.
Then score every change against it. Here is the smallest harness that does the job:
import json
def load_cases(path):
with open(path) as f:
return [json.loads(line) for line in f]
def score(case, output):
required = all(fact.lower() in output.lower() for fact in case["must_include"])
forbidden = any(bad.lower() in output.lower() for bad in case.get("must_not_include", []))
return required and not forbidden
def run_eval(system, cases):
results = [(case["id"], score(case, system(case["input"]))) for case in cases]
passed = sum(ok for _, ok in results)
print(f"{passed}/{len(results)} passed")
for case_id, ok in results:
if not ok:
print("FAIL", case_id)
return passed / len(results)
system is whatever you are testing: a prompt, a chain, an agent. Run it in CI. A change that lowers the pass rate does not ship, however good the demo looks.
Every model is wrong some of the time. That is fine for a draft a person reads. An agent connected to real systems is different. Its mistakes act.
Decide three things before it gets access:
A simple pattern is a tool wrapper that checks a policy before it runs anything:
APPROVAL_REQUIRED = {"refund", "delete_record", "send_email"}
def call_tool(name, args, approved=False):
if name in APPROVAL_REQUIRED and not approved:
return queue_for_review(name, args) # a person decides
result = TOOLS[name](**args)
audit_log(name, args, result) # enough to undo it
return result
A pilot that costs pennies can cost more than the work it replaces once everyone uses it. The pilot bill hides this. Twenty testers never hit the long documents, the retries or the follow-up questions.
Measure the cost of each task from day one, not the monthly invoice:
def cost_per_task(tokens_in, tokens_out, price_in_per_m, price_out_per_m, calls=1):
return calls * (tokens_in * price_in_per_m + tokens_out * price_out_per_m) / 1_000_000
Log it per request, then multiply by the volume you expect in a year. Then use the smallest model that still passes your test set from question 1. The test set is what lets you switch models with confidence instead of hope.
Your data changes. Providers update their models, and sometimes withdraw them. Prompts that passed in March can fail in June with no change on your side.
Someone must own this every week, for as long as the system runs:
Buy when the job is common and an existing tool already does it well enough. Build when the AI is the product, or when it depends on data and decisions only you have.
Either way, budget for the gap, not for the demo. The four pieces above (a test set, limits on what it can do, cost per task, and an owner) are most of that gap.
At Fountech, that gap is our work, for the AI we build for clients and for our own ventures. If you are taking something from demo to production, you can start a project with us here: https://fountech.ai/?utm_source=devto&utm_medium=post&utm_campaign=fountech-2026-10