Here's something I've been thinking about while building AI systems:
Developers are obsessed with testing code.
We write unit tests.
Integration tests.
End-to-end tests.
CI pipelines.
Code reviews.
Linting.
Type checking.
But then we build an AI feature and suddenly the testing strategy becomes:
"I tried it three times and it seems pretty good."
That's not testing.
That's optimism.
And I think this is becoming one of the biggest weaknesses in AI development.
AI Applications Are Software
Consider a simple AI coding assistant.
The workflow might look like this:
User Request
↓
Context Retrieval
↓
Prompt
↓
LLM
↓
Generated Code
↓
Validation
Every component can fail.
The retrieval can return the wrong files.
The context can be incomplete.
The prompt can be ambiguous.
The model can hallucinate.
The generated code can contain bugs.
Yet many AI applications have no automated way of detecting these failures.
We wouldn't accept that standard from a normal API.
Why should AI be different?
"It Worked Once" Means Almost Nothing
Suppose you're building a system that converts natural language into SQL.
You test:
"Show me the top 10 customers by revenue."
The model generates:
SELECT customer_name, SUM(revenue) AS total_revenue
FROM sales
GROUP BY customer_name
ORDER BY total_revenue DESC
LIMIT 10;
Looks good.
You ship it.
Then a user asks:
"Show me the top 10 customers by revenue in 2025,
excluding cancelled orders."
Suddenly your system may produce completely different behavior.
AI outputs are probabilistic.
That means testing one input isn't enough.
Build an Evaluation Dataset
One of the simplest things an AI builder can do is create a small evaluation dataset.
For example:
test_cases = [
{
"input": "Find the top 10 customers by revenue.",
"expected_contains": ["GROUP BY", "ORDER BY", "LIMIT"]
},
{
"input": "Find revenue for 2025 excluding cancelled orders.",
"expected_contains": ["2025", "cancelled"]
},
]
Now you can run your AI system against the same cases whenever you change:
That changes everything.
You're no longer asking:
"Does this feel better?"
You're asking:
"Did performance improve?"
Prompts Need Tests Too
This is one reason I don't think prompt engineering is disappearing.
A production prompt isn't just something you write once.
It is part of the system.
And if you change it, you should know whether the change improved the output.
I discussed the importance of this broader discipline in The Real Reason Prompt Engineering Isn't Going Away.
The next step is to connect prompt engineering with evaluation.
Think of it like software:
Prompt v1
↓
Evaluation
↓
Results
↓
Prompt v2
↓
Evaluation
↓
Compare
That's much more reliable than changing prompts based on intuition.
Context Needs Testing Too
Here's another problem.
You can have a perfect prompt and still get a terrible answer because the AI received the wrong context.
Imagine a coding assistant receives:
Prompt:
"Fix the authentication bug."
Context:
5 unrelated files
outdated documentation
wrong configuration
The model may generate perfectly reasonable code based on completely incorrect information.
This is why I believe context engineering is becoming just as important as prompt engineering.
I wrote about this in Why Context Engineering Is More Important Than Prompt Engineering.
The lesson is simple:
Don't only test what you ask the model. Test what you give the model.
Workflows Need Evaluation
This becomes even more important when AI is part of a larger workflow.
Consider:
User
↓
Retriever
↓
LLM
↓
Tool Call
↓
Validation
↓
Final Response
Where did the failure happen?
You need to know.
Was the retrieval wrong?
Did the model select the wrong tool?
Did the API return bad data?
Did validation fail?
This is one reason I've argued that workflows often matter more than agents.
A well-defined workflow gives you clear places to measure and debug.
I explored that argument in Why I Think Workflows Matter More Than Agents.
Start Small
You don't need an expensive AI evaluation platform to begin.
Start with 20–50 representative test cases.
For each case, record:
Then run the dataset whenever you make a significant change.
Over time, your evaluation dataset becomes one of the most valuable assets in your AI project.
It captures what "good" actually means.
My AI Evaluation Rule
I've started thinking about AI systems in three layers:
Basic functionality.
Reliability.
Engineering maturity.
The third question is where many AI projects struggle.
If you cannot measure improvement, you're mostly guessing.
Evaluation Is the Missing Layer
The AI industry has spent enormous effort improving:
But evaluation deserves the same attention.
Because eventually every AI system needs to answer one uncomfortable question:
"How do you know it works?"
Not:
"The demo looked impressive."
Not:
"The model is highly capable."
Not:
"Users seem to like it."
Show me the evaluation.
That's the engineering mindset I want to see more often in AI.
Build This Into Your GitHub Workflow
I'm also adding a simple AI evaluation starter to my companion AI Builder resources.
A useful structure is:
ai-evaluation/
├── README.md
├── test_cases.json
├── evaluate.py
├── results.csv
└── prompts/
├── v1.txt
└── v2.txt
The idea is straightforward:
Prompt changes → Run tests → Record results → Compare versions.
This turns experimentation into an engineering process.
Final Thoughts
I don't think AI development should be:
Prompt
↓
Looks good
↓
Ship
It should look more like:
Build
↓
Evaluate
↓
Measure
↓
Improve
↓
Evaluate Again
↓
Deploy
That's how we build reliable software.
And I believe that's how we need to start building reliable AI.
The future of AI engineering won't belong only to people who know how to make models produce impressive outputs.
It will belong to developers who can measure, reproduce, debug, and improve those outputs.
Because the most important question in AI isn't:
"Can the model do it?"
It's:
"Can I prove that my system does it reliably?"