{"slug": "your-first-ai-architecture-project-what-changes-and-what-stays-the-same", "title": "Your First AI Architecture Project: What Changes and What Stays the Same", "summary": "In a webinar titled \"Same Job, New Rules,\" Matthias Bohlen argued that AI does not remove the architect's job but adds new risks the design must control, since system behavior now comes from data, models, prompts, search, and test rules rather than code alone. The accompanying guide walks through planning a small first AI project, using an incident-review assistant as an example, and stresses versioning models and prompts, building a test set from the start, and keeping the model's permissions narrow.", "body_md": "**Your first AI project can make architecture work feel old. It is not.**\n\nThe architect still owns the system’s structure, reliability, security, observability, and hard choices. AI does change what drives the system. Some behavior now comes from data, models, prompts, search, and test rules. Code is only one part.\n\nThis was a key point in [Matthias Bohlen’s webinar, “Same Job, New Rules”](https://www.youtube.com/watch?v=9keRB3vjMQw). **AI does not remove the architect’s job.** It adds new risks that the design must control.\n\n**This guide shows what to keep, what to add, and how to plan a small first AI project.**\n\nAn AI feature is still part of a software system. The basic design questions do not go away:\n\nSuppose you build an assistant for an operations team. It helps them review incidents. It reads an incident note, searches trusted runbooks, and drafts a short summary. It also lists possible next checks.\n\nYou still need goals for speed, uptime, privacy, cost, and support. You still need clear APIs and a safe fallback. You must record key choices and explain their trade-offs.\n\nAI can expose weak design.\n\nThree changes affect the design from the start.\n\nMost normal functions should return the same result for the same input. An AI model may return new words, new advice, or a new mistake each time. This type of result is called probabilistic.\n\nThis changes how you test. An exact text match is rarely enough. You need a test set, output rules, source checks, and a safe fallback. Use the fallback when you cannot trust the result.\n\nThe system must also catch answers that look right but lack support from the given sources.\n\nIn a normal service, code and settings define most behavior. In an AI service, the result may also depend on:\n\nThe code may stay the same while the feature starts to act in a new way. A model, prompt, or set of source files may have changed. These parts need version control and review too.\n\nA first diagram may show one box called “AI.” The real flow has more steps:\n\nEach step can fail in its own way. One black box makes faults hard to trace and explain.\n\nDo not begin with a product name. **First ask what type of result you need.**\n\nUse machine learning, or ML, when you need a score, rank, flag, or class. Your team may train its own model. That work needs data checks, tests, a build, a release path, and a way to spot drift.\n\nUse GenAI when you need new text, code, or other content. Teams often start with a base model. The system may add prompts, RAG, tool calls, agents, safety rules, and model tests.\n\nSome apps use both. An incident service may use ML to rank events. It may then use a language model to explain the events with the highest risk. The architect must set the point where the two parts meet. The design must also show how errors can move through the flow.\n\nKeep the first use case small. Make sure a person can check the result.\n\nOur incident assistant has one job. It prepares a draft for an engineer. It cannot restart a service. It cannot change the system or send a message to a customer. The first release is still useful, but the model gets no extra power.\n\nThe flow can be split into clear parts:\n\nAn early test should answer design questions, not just prove that a model can return text.\n\nTest it with real examples and ask:\n\nThe answers may change the design. A slow model may need an async flow. Poor search may need better tags or smaller chunks. Private data may need a private model API or stricter filters.\n\nTest the risky parts before the design becomes hard to change.\n\nDo not wait until release to test the AI. Make a small test set at the start.\n\nFor the incident assistant, each case can include:\n\nRun this set after each change to the model, prompt, search rules, or output checks. Do not ship a new version just because a few demo prompts looked good.\n\nThe application code can make the boundary visible:\n\ntype ReviewResult =\n\n  | {\n\n      status: \"ready_for_review\";\n\n      summary: string;\n\n      checks: string[];\n\n      sourceIds: string[];\n\n    }\n\n  | {\n\n      status: \"fallback\";\n\n      reason: string;\n\n    };\n\nasync function createIncidentDraft(input: Incident): Promise {\n\n  const safeInput = sanitize(input);\n\n  const sources = await runbooks.search(safeInput.summary);\n\nif (sources.length === 0) {\n\n    return { status: \"fallback\", reason: \"No trusted context found\" };\n\n  }\n\nconst prompt = buildPrompt({ incident: safeInput, sources });\n\n  const draft = await model.generate(prompt);\n\nconst checked = validateDraft(draft, sources);\n\n  await recordTrace({\n\n    modelVersion: model.version,\n\n    promptVersion: PROMPT_VERSION,\n\n    sourceIds: sources.map((source) => source.id),\n\n    result: checked.ok ? \"ready_for_review\" : \"fallback\"\n\n  });\n\nreturn checked.ok\n\n    ? { status: \"ready_for_review\", ...checked.value }\n\n    : { status: \"fallback\", reason: checked.reason };\n\n}\n\nNo check can prove that every claim is true. It can check the format and reject source IDs that do not exist. It can block banned content and send weak results to the fallback. The engineer still owns the final action.\n\n*(PS: The following simplified TypeScript example applies these architecture principles. It was created for this article and was not presented in the webinar)*\n\nA green health check does not mean that users get good results.\n\nTrack speed, errors, token use, and cost. Also track the fallback rate, rejected drafts, edited drafts, missing sources, and failed searches.\n\nLogs may hold private data. Do not save raw prompts and answers by default. Decide what is safe to keep. Set who can see it and when it must be removed.\n\nYour logs should answer two questions. Is the service running? Is its output still useful?\n\nAI work asks for new skills. Architects must learn about model limits, data quality, search, tests, safety rules, and AI ops. Yet the base still matters. Teams need clear goals, boundaries, APIs, trade-offs, records, and clear communication.\n\nTeams can use the [tecnovy iSAQB board](https://tecnovy.com/en/isaqb) to compare core courses with AI-focused modules such as [SWARC4AI](https://tecnovy.com/en/isaqb/advanced-swarc4ai)or [AGENTA](https://tecnovy.com/en/isaqb/agenta). It shows both Foundation and Advanced Level options.\n\nCourses do not replace hands-on work. Shared terms and methods can help the team learn before a serious fault puts it under pressure.\n\nYour first AI system does not need an agent. It does not need a large model stack.\n\nIt does need one clear task. It needs trusted input, visible steps, repeatable tests, useful logs, a safe fallback, and one person who owns the result.\n\nThe architect’s job stays the same. Make system risks easy to see. Make design choices clear. AI changes where those risks come from and how the team must test them.\n\n**If you were starting this incident assistant tomorrow, which risk would you test first: retrieval quality, unsafe output, latency, cost, or human review?**", "url": "https://wpnews.pro/news/your-first-ai-architecture-project-what-changes-and-what-stays-the-same", "canonical_source": "https://dev.to/tecnovy_academy/your-first-ai-architecture-project-what-changes-and-what-stays-the-same-942", "published_at": "2026-09-25 17:29:56+00:00", "updated_at": "2026-09-25 18:00:41.857666+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "mlops", "generative-ai"], "entities": ["Matthias Bohlen"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/your-first-ai-architecture-project-what-changes-and-what-stays-the-same", "markdown": "https://wpnews.pro/news/your-first-ai-architecture-project-what-changes-and-what-stays-the-same.md", "text": "https://wpnews.pro/news/your-first-ai-architecture-project-what-changes-and-what-stays-the-same.txt", "jsonld": "https://wpnews.pro/news/your-first-ai-architecture-project-what-changes-and-what-stays-the-same.jsonld"}}