{"slug": "how-i-build-ai-projects-from-scratch-six-decisions-one-rule", "title": "How I Build AI Projects From Scratch (Six Decisions, One Rule)", "summary": "A developer who has shipped four AI products in three years outlined a repeatable process for building AI projects, centered on a single rule: pick a job with a checkable outcome, write the eval before the prompt, and invest in the data layer rather than the intelligence layer. In the most recent project, a travel agent, replacing roughly 1,900 lines of custom API adapters with an MCP server cut the codebase to 240 lines and let the agent return real prices instead of plausible ones, while a 20-case eval written in two hours lifted the pass rate from 11/20 to 18/20 in the first week. The developer also described keeping agents to a simple loop of about 300 lines with an eight-step and 45-second budget, and argued that products survive when users can see and recover from the model's wrongness.", "body_md": "I have shipped four AI products in three years, two of which earned money and two of which I shut down. The difference was never the model.\n\nThe pattern I trust now fits in one sentence: pick a job with a checkable outcome, write the eval before the prompt, and spend your effort on the data layer instead of the intelligence layer.\n\nHere is the process, in the order I actually do it, with the numbers from the last project.\n\nBad framing: \"an AI assistant for travel.\" There is no way to know if it worked.\n\nGood framing: \"given a city and a date range, return three hotels that exist, with prices within 10% of the live rate, in under six seconds.\"\n\nThe second version tells me what to build, what to measure, and when to quit. It also rules out a lot of features I would otherwise build because they demo well.\n\nI write the outcome sentence before I write any code, and I keep it at the top of the README. When the project drifts, which it always does, that sentence is the thing I compare against.\n\nThis is the step people skip, and it is the one that decides whether you are doing engineering or vibes.\n\nBefore the first prompt, I collect 20 real cases and write down the pass condition for each. Not \"looks good\". Something a script can check or a stranger can verify in five seconds.\n\nFor the travel project the eval had four columns:\n\n| Case | Must return · Must not | Pass condition | \n|---|---|---|\n| Kyoto, 2 nights, 4-star | 3 hotels · invented names | all IDs exist in fixtures | \n| Lisbon, past dates | error message · a hotel list | mentions the date problem | \n| Tokyo, budget 8,000 JPY | 3 hotels · 5-star results | max price under budget | \n| \"somewhere warm in March\" | clarifying question · a guess | asks for a city | \n\nTwenty cases took two hours to write and saved me from shipping three broken versions. Every time I changed a prompt, I re-ran them. Pass rate went from 11/20 to 18/20 in the first week, mostly from fixing the data layer rather than the wording.\n\nThe eval is also the only honest answer to \"is it better now?\". Without it, every change feels like an improvement.\n\nMy agents are a while loop: call the model, run a tool, store the result, repeat until a final answer or a step budget runs out. No planner, no graph, no memory layer.\n\nThe loop is around 300 lines including retries and logging. I have written about the mechanics separately, so here is only the design constraint: every step must be reconstructible from the log. If I cannot replay a session and see the exact tool arguments, the loop is too clever.\n\nStep budgets and timeouts are not polish. They are what turns a demo into something that can run unattended. Mine are eight steps and 45 seconds, and they are the first two constants in the file.\n\nThe single highest-value decision in every project was to stop writing integrations.\n\nMy first travel agent had 1,900 lines of adapters: auth wrappers, response mappers, retry logic, and a fuzzy name matcher that reconciled three sources. It broke whenever a supplier changed a field. The second version replaced that layer with an MCP server and kept the loop.\n\n```\n{\n  \"mcpServers\": {\n    \"rollinggo-hotel\": {\n      \"type\": \"http\",\n      \"url\": \"https://mcp.rollinggo.ai/mcp\",\n      \"headers\": { \"Authorization\": \"Bearer YOUR_API_KEY\" }\n    }\n  }\n}\n```\n\nWhy this specific one, since \"use an MCP server\" is generic advice:\n\n`searchHotels`, `getHotelDetail`, `getHotelSearchTags` are defined server-side, so validation and error shapes stop being my problem.\nThe adapter deletion took the codebase from 1,900 lines to 240. More importantly, it changed what the agent could claim: real prices instead of plausible ones.\n\nEvery AI product I built had a wrongness rate. The ones that survived were the ones where users could see it and recover.\n\nThree mechanisms, in increasing order of usefulness:\n\n**Cite the source of a fact.** A hotel ID, a price timestamp, a document name. If the model cannot name where a number came from, the number should not be on screen.\n\n**Show the failure state.** \"No hotels match those dates\" is a feature. Empty space is a bug report waiting to happen.\n\n**Log the correction.** When a user says \"that is wrong\", write down what they corrected and which tool produced it. After 40 users I had a list of six recurring failures, and four of them were data problems, not prompt problems.\n\nThe wrongness rate on the travel project went from 26% (model-only) to under 1% after the data layer swap, with four percent of prices still drifting between quote and click. I know those numbers because the eval measured them, not because it felt better.\n\nI give every project three weeks and one metric.\n\nIf the eval pass rate is under 60% after three weeks of real work, I stop. Not because the idea is bad, but because the gap is usually structural: the data does not exist, the task is not checkable, or the user does not care enough to correct the output.\n\nTwo of my four projects died on that rule. One was an AI meeting summarizer where the output was fine and nobody needed it. The other was a document Q&A tool where the retrieval quality plateaued at 55% and every fix cost a week.\n\nKilling them early is the reason the other two got the time they needed.\n\n**Fine-tuning.** Every problem I had was a data or eval problem. Fine-tuning moved the failure around without removing it.\n\n**Agent frameworks.** I tried four. Each added a layer between me and the behavior I was debugging. The loop is 300 lines and I can print it.\n\n**Prompt libraries.** A prompt that works on one model version is a liability on the next. What survives a model upgrade is a clear tool schema and a checkable outcome.\n\n**Big launches.** The project that worked had 40 users for a month before anyone outside heard about it. The one that failed had a landing page and no working eval.\n\nThe rest, UI, billing, marketing, can be added later. None of it saves a product whose facts are wrong.\n\nIf you want the data-layer shortcut I used, the [RollingGo Hotel MCP repository](https://github.com/RollingGo-AI/RollingGo-Hotel-MCP-Global) has the config, the tool schemas, and the response examples in one place. Read the response schema first; it tells you more about the product than the marketing page does.", "url": "https://wpnews.pro/news/how-i-build-ai-projects-from-scratch-six-decisions-one-rule", "canonical_source": "https://dev.to/iamthedev/how-i-build-ai-projects-from-scratch-six-decisions-one-rule-10k8", "published_at": "2026-11-01 17:41:00+00:00", "updated_at": "2026-11-01 17:52:17.456863+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "agent-protocols", "developer-tools", "ai-products"], "entities": ["RollingGo", "MCP"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-build-ai-projects-from-scratch-six-decisions-one-rule", "markdown": "https://wpnews.pro/news/how-i-build-ai-projects-from-scratch-six-decisions-one-rule.md", "text": "https://wpnews.pro/news/how-i-build-ai-projects-from-scratch-six-decisions-one-rule.txt", "jsonld": "https://wpnews.pro/news/how-i-build-ai-projects-from-scratch-six-decisions-one-rule.jsonld"}}