# Your First AI Architecture Project: What Changes and What Stays the Same

> Source: <https://dev.to/tecnovy_academy/your-first-ai-architecture-project-what-changes-and-what-stays-the-same-942>
> Published: 2026-09-25 17:29:56+00:00

**Your first AI project can make architecture work feel old. It is not.**

The architect still owns the system’s structure, reliability, security, observability, and hard choices. AI does change what drives the system. Some behavior now comes from data, models, prompts, search, and test rules. Code is only one part.

This was a key point in [Matthias Bohlen’s webinar, “Same Job, New Rules”](https://www.youtube.com/watch?v=9keRB3vjMQw). **AI does not remove the architect’s job.** It adds new risks that the design must control.

**This guide shows what to keep, what to add, and how to plan a small first AI project.**

An AI feature is still part of a software system. The basic design questions do not go away:

Suppose you build an assistant for an operations team. It helps them review incidents. It reads an incident note, searches trusted runbooks, and drafts a short summary. It also lists possible next checks.

You still need goals for speed, uptime, privacy, cost, and support. You still need clear APIs and a safe fallback. You must record key choices and explain their trade-offs.

AI can expose weak design.

Three changes affect the design from the start.

Most normal functions should return the same result for the same input. An AI model may return new words, new advice, or a new mistake each time. This type of result is called probabilistic.

This changes how you test. An exact text match is rarely enough. You need a test set, output rules, source checks, and a safe fallback. Use the fallback when you cannot trust the result.

The system must also catch answers that look right but lack support from the given sources.

In a normal service, code and settings define most behavior. In an AI service, the result may also depend on:

The code may stay the same while the feature starts to act in a new way. A model, prompt, or set of source files may have changed. These parts need version control and review too.

A first diagram may show one box called “AI.” The real flow has more steps:

Each step can fail in its own way. One black box makes faults hard to trace and explain.

Do not begin with a product name. **First ask what type of result you need.**

Use machine learning, or ML, when you need a score, rank, flag, or class. Your team may train its own model. That work needs data checks, tests, a build, a release path, and a way to spot drift.

Use GenAI when you need new text, code, or other content. Teams often start with a base model. The system may add prompts, RAG, tool calls, agents, safety rules, and model tests.

Some apps use both. An incident service may use ML to rank events. It may then use a language model to explain the events with the highest risk. The architect must set the point where the two parts meet. The design must also show how errors can move through the flow.

Keep the first use case small. Make sure a person can check the result.

Our incident assistant has one job. It prepares a draft for an engineer. It cannot restart a service. It cannot change the system or send a message to a customer. The first release is still useful, but the model gets no extra power.

The flow can be split into clear parts:

An early test should answer design questions, not just prove that a model can return text.

Test it with real examples and ask:

The answers may change the design. A slow model may need an async flow. Poor search may need better tags or smaller chunks. Private data may need a private model API or stricter filters.

Test the risky parts before the design becomes hard to change.

Do not wait until release to test the AI. Make a small test set at the start.

For the incident assistant, each case can include:

Run this set after each change to the model, prompt, search rules, or output checks. Do not ship a new version just because a few demo prompts looked good.

The application code can make the boundary visible:

type ReviewResult =

  | {

      status: "ready_for_review";

      summary: string;

      checks: string[];

      sourceIds: string[];

    }

  | {

      status: "fallback";

      reason: string;

    };

async function createIncidentDraft(input: Incident): Promise {

  const safeInput = sanitize(input);

  const sources = await runbooks.search(safeInput.summary);

if (sources.length === 0) {

    return { status: "fallback", reason: "No trusted context found" };

  }

const prompt = buildPrompt({ incident: safeInput, sources });

  const draft = await model.generate(prompt);

const checked = validateDraft(draft, sources);

  await recordTrace({

    modelVersion: model.version,

    promptVersion: PROMPT_VERSION,

    sourceIds: sources.map((source) => source.id),

    result: checked.ok ? "ready_for_review" : "fallback"

  });

return checked.ok

    ? { status: "ready_for_review", ...checked.value }

    : { status: "fallback", reason: checked.reason };

}

No check can prove that every claim is true. It can check the format and reject source IDs that do not exist. It can block banned content and send weak results to the fallback. The engineer still owns the final action.

*(PS: The following simplified TypeScript example applies these architecture principles. It was created for this article and was not presented in the webinar)*

A green health check does not mean that users get good results.

Track speed, errors, token use, and cost. Also track the fallback rate, rejected drafts, edited drafts, missing sources, and failed searches.

Logs may hold private data. Do not save raw prompts and answers by default. Decide what is safe to keep. Set who can see it and when it must be removed.

Your logs should answer two questions. Is the service running? Is its output still useful?

AI work asks for new skills. Architects must learn about model limits, data quality, search, tests, safety rules, and AI ops. Yet the base still matters. Teams need clear goals, boundaries, APIs, trade-offs, records, and clear communication.

Teams can use the [tecnovy iSAQB board](https://tecnovy.com/en/isaqb) to compare core courses with AI-focused modules such as [SWARC4AI](https://tecnovy.com/en/isaqb/advanced-swarc4ai)or [AGENTA](https://tecnovy.com/en/isaqb/agenta). It shows both Foundation and Advanced Level options.

Courses do not replace hands-on work. Shared terms and methods can help the team learn before a serious fault puts it under pressure.

Your first AI system does not need an agent. It does not need a large model stack.

It does need one clear task. It needs trusted input, visible steps, repeatable tests, useful logs, a safe fallback, and one person who owns the result.

The architect’s job stays the same. Make system risks easy to see. Make design choices clear. AI changes where those risks come from and how the team must test them.

**If you were starting this incident assistant tomorrow, which risk would you test first: retrieval quality, unsafe output, latency, cost, or human review?**