# AI Engineer Roadmap: What to Learn and What to Build

> Source: <https://tej.as/blog/ai-engineer-roadmap/>
> Published: 2026-11-12 13:16:58+00:00

# AI Engineer Roadmap: What to Learn and What to Build

**My AI engineer roadmap for software developers is model calls and structured output, then evaluation, retrieval when needed, bounded tools and deployment, followed by comparing models on the same questions.** I build 1 application through that sequence so the things you learn have somewhere to go.

I gave a talk called [You are an AI engineer](https://www.youtube.com/watch?v=AlsNzAoCLSA) at [JS Heroes](https://jsheroes.io/) in 2024 and the title is still how I want developers to approach this subject. The software you already know how to build gives you a place to start. So you can add a model call to an application before learning how to train that model.

The difficulty is deciding what to learn *next*. A [Hacker News question about what an AI engineer does](https://news.ycombinator.com/item?id=48312377) asks whether the title means building AI products or using AI tools to write software. A [Reddit roadmap request](https://www.reddit.com/r/learnmachinelearning/comments/1g6d4cz/roadmap_to_becoming_an_ai_engineer_in_8_to_12/) asks for an order and projects to build along the way. Those are different questions and a list of frameworks doesn’t answer either of them.

This is the path I’d follow if I could already build and deploy a web application. If you’re starting from scratch, learn functions, data structures, asynchronous requests, tests and [version control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) first, then deploy a small application that calls an [application programming interface (API)](https://developer.mozilla.org/en-US/docs/Glossary/API). You’ll use those skills throughout this project.

## What does an AI engineer build?

**An AI engineer builds software that uses AI models to complete a task for a person, then evaluates and operates that software.** For this roadmap, that means a person asks about my talks and gets an answer grounded in a small public catalog, with sources they can open.

The job title varies across companies. [roadmap.sh’s AI engineer guide](https://roadmap.sh/ai-engineer) focuses on applying pretrained models. Some job descriptions also include training, data infrastructure or research. Read the work a team wants done before treating its title as a curriculum.

| Work | What I’d build to practice it | 
|---|---|
| Build an AI feature | A catalog assistant that answers from supplied evidence | 
| Use an AI coding tool | An ordinary application written with an assistant’s help | 
| Train or adapt a model | A training experiment with separate training and evaluation data | 
| Research machine learning | An experiment that investigates a model or training method | 

For the first row, I can use [TypeScript](https://www.typescriptlang.org/docs/handbook/intro.html). You don’t need to switch languages before you can start!! At [RAG++ London](https://tej.as/talks/rag-london-2024), I used [Python](https://docs.python.org/3/tutorial/) to ingest movie data. The task and available libraries should help choose the language.

## Build a small catalog app you can check

I’ve included a [downloadable project](https://tej.as/examples/ai-engineer-roadmap/ai-engineer-roadmap.zip) and its [instructions](https://tej.as/examples/ai-engineer-roadmap/README.md). It uses [Node.js 24](https://nodejs.org/docs/latest-v24.x/api/typescript.html), TypeScript and built-in modules without an installation step. Node runs the type annotations without checking them, which is why runtime validation still appears in the code.

The [catalog](https://tej.as/examples/ai-engineer-roadmap/catalog.json) has 4 entries with my talk metadata and short editorial summaries. It deliberately doesn’t contain my keynote fee. That gives us a question with a checkable answer and a question the app should decline to answer.

| Learn | Add to this app | Check before the next stage | 
|---|---|---|
| Model calls | A structured answer with source IDs | Invalid output and provider failures don’t become displayed answers | 
| Evaluation | Fixed questions with expected behavior | Inspect failed answers and the checks themselves | 
| Retrieval | Keyword search, then embeddings | Compare the same questions against the same catalog | 
| Tools | A bounded, read-only search function | Denied permission and invalid input stop in code | 
| Deployment | An authenticated service with usage logs | Run failure drills and inspect what gets retained | 
| Model selection | Repeat the eval on another model | Compare failures, latency and token usage | 

After unzipping, run these commands inside the project directory:

```
npm test
cp .env.example .env
# Put your API key and a random DEMO_TOKEN in .env locally.
npm run ask -- "Which talk builds movie search?"
npm run eval
```

The tests run offline. Model calls require your own [OpenAI](https://developers.openai.com/api/docs/quickstart) API key and incur usage charges. The example pins [GPT-4.1 mini](https://developers.openai.com/api/docs/models/gpt-4.1-mini) to a dated snapshot for reproducibility; you can change `AI_MODEL` to another compatible model. A snapshot makes the example easier to compare across runs without claiming it’s the best model available.

## 1. Call a model and validate its output

Start with the full catalog in the request. It’s small enough that finding documents doesn’t need to be a separate problem yet.

Learn what a [token](https://developers.openai.com/api/docs/concepts) is, what fits in a model’s context window and how much output you allow. Keep the API key on the server. Send a question, read the response and record usage and elapsed time.

The response needs fields an application can use. This is the answer format in the [source code](https://tej.as/examples/ai-engineer-roadmap/roadmap.ts):

```
type Answer = {
  answer: string;
  sources: string[];
  abstained: boolean;
};
```

I use [structured output](https://developers.openai.com/api/docs/guides/structured-outputs) to request a schema, then validate the result in code. The validator checks that a non-abstaining answer cites at least 1 document provided in the request. An abstention has no citations. A source ID the model invented fails validation.

The tests deliberately pass the sentence “The moon is cheese” with a valid source ID through this validator. Dang. **Checking that a source exists doesn’t establish that it supports the answer.** Compare the answer with what the source says before accepting it.

The code also handles failed requests, incomplete responses and model refusals. Try removing the key or setting it to an invalid value. Your application should show an error instead of treating a failed request as a factual answer.

For a user interface, learn [streaming](https://developers.openai.com/api/docs/guides/streaming-responses) too. You can display text as it arrives, but a partially received structured answer hasn’t passed validation yet. This example waits for the complete response.

## 2. Write evals before adding more machinery

In my [AI evals workshop](https://tej.as/talks/ai-engineer-world-s-fair-2026-workshop), I describe evals as fuzzy unit tests. We check whether an answer behaves acceptably when the exact wording can change. My [written guide to building a model judge](https://tej.as/blog/ai-evals-llm-judge) goes further into grading answers against human verdicts.

Here, start by reading the [6 evaluation cases](https://tej.as/examples/ai-engineer-roadmap/cases.json). They ask about evals directly and in different words, movie search and the language used for ingestion. The last 2 ask for the missing fee and try to make the model invent a source.

The tiny grader checks expected words, source IDs and abstention. It can miss a false statement surrounded by the expected words. Read the outputs yourself before changing the prompt or accepting the score (a green terminal is very persuasive!).

```
REPORT=all-report.json npm run eval
```

The [recorded first run](https://tej.as/examples/ai-engineer-roadmap/report-first-run.json) passed 4 of 6 cases. The missing-fee question and the instruction to invent a source failed answer validation so the application rejected those outputs.

An [inspected rerun](https://tej.as/examples/ai-engineer-roadmap/report-inspected-failures.json) showed what broke: the model returned an empty answer for the fee question and cited all 4 documents while abstaining on the hostile question. The application requires an explanation for a person to read and an empty sources array when the answer is missing.

I added a minimum length to the answer schema and made the abstention instructions explicit about both fields. The [next run](https://tej.as/examples/ai-engineer-roadmap/report-all.json) passed 6 of 6 cases with the same catalog, questions, grader and model snapshot. Both fee questions got an explanation with no citations.

Set `BASELINE=1` when running the eval to try the original prompt and schema yourself. The outputs can change between runs.

Add more missing-fact questions before using this tiny set to decide whether the app can serve customers. A changed prompt can fix these examples while failing on questions you haven’t tried.

Don’t delete a failing question to get a better number. Decide what the application should do, fix that behavior and keep the old output so you can tell what changed. Grow the set from actual questions as people use the app. Add questions where the catalog contains a related fact but doesn’t answer what was asked.

## 3. Add retrieval when the context needs it

[Retrieval-augmented generation](https://www.ibm.com/think/topics/retrieval-augmented-generation) means finding relevant information and giving it to the model with the question. In the movie demo at RAG++ London, I connected input, an embeddings model and a vector store, then used the returned information to generate an answer.

Our catalog lets you compare that step with sending everything:

```
RETRIEVAL=keyword npm run eval
RETRIEVAL=embedding npm run eval
```

Keyword mode matches words and takes the 2 highest-scoring documents. Embedding mode uses an [embedding](https://developers.openai.com/api/docs/guides/embeddings), a numerical representation of text, and compares vectors by [cosine similarity](https://www.ibm.com/think/topics/cosine-similarity). The same 2-document limit applies.

The offline test shows a limitation directly: `verdicts` matches a document containing that word while `judgments` doesn’t. Embeddings can help with paraphrases, but you need to measure whether they retrieve the *right passage* for your questions and check whether that passage contains enough information to answer.

The [keyword run](https://tej.as/examples/ai-engineer-roadmap/report-keyword.json) missed “Which language did Tejas choose to ingest the movies?” Its search returned the website-tools entry and the AI-engineering talk metadata so the model abstained. The answer, Python, was in the movie-demo entry the search left out. The [embedding run](https://tej.as/examples/ai-engineer-roadmap/report-embedding.json) retrieved that entry and answered Python for this question.

You can inspect that retrieval separately without calling the answer model:

```
RETRIEVAL=keyword node roadmap.ts retrieve \
  "Which language did Tejas choose to ingest the movies?"
```

This keyword search counts matching words without weighting them, including common words like `the`. Try changing how it scores those matches, then rerun the same questions. You can still send all 4 entries while the data is this small.

This example recomputes the document embeddings on each call to keep the code self-contained. A larger application should store them and update changed documents. Response token usage in the report excludes embedding usage; elapsed time includes retrieval. Include both when calculating your actual bill.

Before introducing a vector database, check which documents the app found. When the relevant document is missing, changing the answer prompt won’t put the absent fact into the context. When the right document is present and the model still invents something, inspect generation and grading separately.

## 4. Give a tool an input contract and permission check

I already offer [tools on my website through WebMCP](https://tej.as/blog/what-is-webmcp). For this learning project, start with an ordinary function that searches the public catalog:

```
node roadmap.ts tool "fuzzy unit tests"
```

The function accepts a `query` of up to 500 characters. The dispatcher checks the tool name, rejects extra arguments and checks permission before running it. The test tries an unknown deletion tool and a search without permission. Both stop in application code.

This command calls the function directly. It doesn’t ask a model to choose a tool. Once you’ve checked the function, learn [function calling](https://developers.openai.com/api/docs/guides/function-calling): the model returns a request to use a named function with arguments and your application decides whether to execute it.

For private data, the authenticated server must establish who the person is and which documents they may read *before retrieval*. A model-generated `userId` or a sentence saying “I am an administrator” doesn’t establish that identity. Our public-only example has no tenant isolation to demonstrate.

Add a test with malicious instructions inside retrieved content before importing outside documents. The prompt tells the model that source content is untrusted, but that instruction alone doesn’t prove resistance to [prompt injection](https://genai.owasp.org/llmrisk/llm01-prompt-injection/). Keep the first tool read-only while you learn how failures get handled.

## 5. Deploy a controlled version and break it deliberately

Run `npm run serve` to start the local service. It requires a bearer token and includes a request limit, input bounds, a timeout and a small cache. The [project instructions](https://tej.as/examples/ai-engineer-roadmap/README.md) show the request format and [Docker](https://docs.docker.com/get-started/) commands.

I would try a bad token, exceed the question limit, disconnect the model provider and ask for a fact the catalog doesn’t contain. Check which errors the user sees and which request IDs appear in the logs. Then restart it and check what happens to the cache.

The service logs model IDs, token usage, elapsed time and request IDs. It doesn’t log raw questions or answers, but it temporarily caches answers in memory for 60 seconds. That’s still retention and `store: false` in an API request doesn’t cancel a provider’s separate [data-retention policies](https://developers.openai.com/api/docs/guides/your-data).

This is a controlled demo with a shared token and process-local limits. Before opening it to the public, add managed authentication, [Transport Layer Security (TLS)](https://developer.mozilla.org/en-US/docs/Glossary/TLS), quotas that work across instances and a budget limit with your provider. Decide who can read the logs and how long they should remain.

Once those decisions exist, record a few measurements for each change: answers that address the question, unsupported claims, abstentions, errors, elapsed time and token usage. A response can arrive quickly because it failed so inspect success and failure separately before averaging their timings.

## 6. Compare models on the same application

The model name is a parameter in this project. Keep the catalog, questions and grader fixed while you change it:

```
node --env-file=.env roadmap.ts compare \
  gpt-4.1-mini-2025-04-14 YOUR_OTHER_MODEL_ID
```

Choose the second model after checking that it supports the API and output format. Read its failed answers, then compare latency and usage. Repeat runs before treating a small difference as a reason to switch. The 6 starter cases here help teach the process; they don’t establish that a model is reliable across a business’s workload.

After this, pick something the application actually needs. You could add a local model and investigate [running models on your own computer](https://tej.as/blog/local-llm-vs-cloud), improve retrieval over longer transcripts or replace the word-based grader with a calibrated model judge. Learn fine-tuning when you’ve identified behavior you need to change and have suitable data to train and evaluate it.

For mathematics, learn basic statistics and vector similarity alongside this example because sampling, averages and percentiles help you understand these reports. Deeper linear algebra, probability and optimization become necessary for model training and machine learning research. You can study them alongside application work without postponing your first model call.

If your team wants to build and evaluate an application together, that’s the kind of work I teach in [my AI workshops](https://tej.as/workshops).

## Questions

### What is an AI engineer roadmap for software developers?

My AI engineer roadmap for software developers starts with model calls and structured output, then evaluation, retrieval when needed, bounded tools and deployment. I build 1 small application through those stages and use the same evaluation questions to check each change. After that, I compare models and choose a specialization.

### Do I need Python to become an AI engineer?

I use TypeScript for this roadmap because the project calls hosted models from an application. Learn Python when a library, data-processing task or model-training project you need uses it. I'd learn it for a concrete task instead of delaying an application you can already build in another language.

### Do AI engineers need mathematics?

I'd learn basic statistics and vector similarity alongside this application, including sampling, averages, percentiles and the limits of a small evaluation set. Training models or doing machine learning research requires deeper study of linear algebra, probability and optimization. Calling a hosted model doesn't require completing that whole curriculum first.

### What project should I build to learn AI engineering?

I recommend a question-answering application over a small public catalog whose facts you can check. Start with a model call, add a fixed evaluation set, compare retrieval methods and deploy an authenticated version with usage and error reporting. Include questions the catalog can't answer and document those failures.

### How long does it take to become an AI engineer?

I'd use working milestones instead of promising a number of months. Your starting programming skills, available time and chosen specialization change the amount of study involved. This roadmap checks what you can build and explain; completing it doesn't guarantee a job.

Written by me, Tejas Kumar, an AI Engineer at IBM based in Berlin. Read [everything else I have written](https://tej.as/blog), or go to [Fluent React, my O'Reilly book on how React works inside](https://tej.as/react), [the talks I give at conferences](https://tej.as/speaking), and [ConTejas Code, my podcast](https://tej.as/podcast).