Most Developers Test Their Code. Why Don't They Test Their AI?

A developer argues that AI applications are software and should be tested with the same rigor as traditional code, advocating for evaluation datasets, prompt testing, and context testing. The developer highlights that probabilistic AI outputs require systematic evaluation rather than relying on intuition or 'it worked once' optimism.

Here's something I've been thinking about while building AI systems: Developers are obsessed with testing code. We write unit tests. Integration tests. End-to-end tests. CI pipelines. Code reviews. Linting. Type checking. But then we build an AI feature and suddenly the testing strategy becomes: "I tried it three times and it seems pretty good." That's not testing. That's optimism. And I think this is becoming one of the biggest weaknesses in AI development. AI Applications Are Software Consider a simple AI coding assistant. The workflow might look like this: User Request ↓ Context Retrieval ↓ Prompt ↓ LLM ↓ Generated Code ↓ Validation Every component can fail. The retrieval can return the wrong files. The context can be incomplete. The prompt can be ambiguous. The model can hallucinate. The generated code can contain bugs. Yet many AI applications have no automated way of detecting these failures. We wouldn't accept that standard from a normal API. Why should AI be different? "It Worked Once" Means Almost Nothing Suppose you're building a system that converts natural language into SQL. You test: "Show me the top 10 customers by revenue." The model generates: SELECT customer name, SUM revenue AS total revenue FROM sales GROUP BY customer name ORDER BY total revenue DESC LIMIT 10; Looks good. You ship it. Then a user asks: "Show me the top 10 customers by revenue in 2025, excluding cancelled orders." Suddenly your system may produce completely different behavior. AI outputs are probabilistic. That means testing one input isn't enough. Build an Evaluation Dataset One of the simplest things an AI builder can do is create a small evaluation dataset. For example: test cases = { "input": "Find the top 10 customers by revenue.", "expected contains": "GROUP BY", "ORDER BY", "LIMIT" }, { "input": "Find revenue for 2025 excluding cancelled orders.", "expected contains": "2025", "cancelled" }, Now you can run your AI system against the same cases whenever you change: That changes everything. You're no longer asking: "Does this feel better?" You're asking: "Did performance improve?" Prompts Need Tests Too This is one reason I don't think prompt engineering is disappearing. A production prompt isn't just something you write once. It is part of the system. And if you change it, you should know whether the change improved the output. I discussed the importance of this broader discipline in The Real Reason Prompt Engineering Isn't Going Away https://dev.to/jaideepparashar/the-real-reason-prompt-engineering-isnt-going-away-2koo . The next step is to connect prompt engineering with evaluation. Think of it like software: Prompt v1 ↓ Evaluation ↓ Results ↓ Prompt v2 ↓ Evaluation ↓ Compare That's much more reliable than changing prompts based on intuition. Context Needs Testing Too Here's another problem. You can have a perfect prompt and still get a terrible answer because the AI received the wrong context. Imagine a coding assistant receives: Prompt: "Fix the authentication bug." Context: 5 unrelated files + outdated documentation + wrong configuration The model may generate perfectly reasonable code based on completely incorrect information. This is why I believe context engineering is becoming just as important as prompt engineering. I wrote about this in Why Context Engineering Is More Important Than Prompt Engineering. The lesson is simple: Don't only test what you ask the model. Test what you give the model. Workflows Need Evaluation This becomes even more important when AI is part of a larger workflow. Consider: User ↓ Retriever ↓ LLM ↓ Tool Call ↓ Validation ↓ Final Response Where did the failure happen? You need to know. Was the retrieval wrong? Did the model select the wrong tool? Did the API return bad data? Did validation fail? This is one reason I've argued that workflows often matter more than agents. A well-defined workflow gives you clear places to measure and debug. I explored that argument in Why I Think Workflows Matter More Than Agents https://dev.to/jaideepparashar/why-i-think-workflows-matter-more-than-agents-3p82 . Start Small You don't need an expensive AI evaluation platform to begin. Start with 20–50 representative test cases. For each case, record: Then run the dataset whenever you make a significant change. Over time, your evaluation dataset becomes one of the most valuable assets in your AI project. It captures what "good" actually means. My AI Evaluation Rule I've started thinking about AI systems in three layers: Basic functionality. Reliability. Engineering maturity. The third question is where many AI projects struggle. If you cannot measure improvement, you're mostly guessing. Evaluation Is the Missing Layer The AI industry has spent enormous effort improving: But evaluation deserves the same attention. Because eventually every AI system needs to answer one uncomfortable question: "How do you know it works?" Not: "The demo looked impressive." Not: "The model is highly capable." Not: "Users seem to like it." Show me the evaluation. That's the engineering mindset I want to see more often in AI. Build This Into Your GitHub Workflow I'm also adding a simple AI evaluation starter to my companion AI Builder resources. A useful structure is: ai-evaluation/ ├── README.md ├── test cases.json ├── evaluate.py ├── results.csv └── prompts/ ├── v1.txt └── v2.txt The idea is straightforward: Prompt changes → Run tests → Record results → Compare versions. This turns experimentation into an engineering process. Final Thoughts I don't think AI development should be: Prompt ↓ Looks good ↓ Ship It should look more like: Build ↓ Evaluate ↓ Measure ↓ Improve ↓ Evaluate Again ↓ Deploy That's how we build reliable software. And I believe that's how we need to start building reliable AI. The future of AI engineering won't belong only to people who know how to make models produce impressive outputs. It will belong to developers who can measure, reproduce, debug, and improve those outputs. Because the most important question in AI isn't: "Can the model do it?" It's: "Can I prove that my system does it reliably?"