There is a tempting assumption when building AI-powered applications: if an LLM can make a decision, why not ask it to make that decision every time?
It sounds reasonable. The model is already part of the application, it can interpret context, and it can handle a wide range of situations.
But there is a catch.
Not every problem needs reasoning. Some problems just need code.
If an AI agent repeatedly solves the same problem, it may be time to turn that solution into a deterministic function. Doing so can reduce inference costs, improve response times, and make the system more predictable.
This is a small architectural decision with potentially significant benefits.
Imagine an AI agent that investigates failed CI/CD pipelines.
Whenever a build fails, the agent:
This is a useful application of an LLM. Build logs can be long, unfamiliar, and sometimes ambiguous. The agent may need to connect several clues before identifying the root cause.
Now imagine that the same known error appears repeatedly:
error NU1301: Unable to load the service index for source.
After investigating the issue, the agent determines that the package feed is temporarily unavailable and recommends retrying package restore.
The next time the same failure occurs, should the application ask the LLM to investigate everything again?
What about the hundredth time?
Probably not. Once the pattern is understood, we can handle it with ordinary application code.
For example, we could introduce a small C# function that recognizes known build errors:
public string AnalyzeBuildFailure(string logs)
{
if (logs.Contains("NU1301", StringComparison.OrdinalIgnoreCase))
{
return "NuGet feed may be unavailable. Check connectivity and retry package restore.";
}
if (logs.Contains("CS0246", StringComparison.OrdinalIgnoreCase))
{
return "A type or namespace could not be found. Check using directives and project references.";
}
if (logs.Contains("401 Unauthorized", StringComparison.OrdinalIgnoreCase))
{
return "Authentication failed. Check the credentials or token used by the pipeline.";
}
return "Unknown failure. Further investigation is required.";
}
This is a simplified example; a production implementation should account for log structure, error context, false matches, and the possibility that one error message has multiple causes.
The important idea is that known patterns can be handled without an LLM call.
Instead of sending every failure to a model, the application can first check whether it recognizes the issue. If it does, it follows a tested path. If it does not, it can escalate the problem to the LLM.
That gives us a useful division of responsibility:
Deterministic code produces the same result for the same input, assuming the same relevant state and execution environment.
For example:
public decimal CalculateTotal(
decimal subtotal,
decimal taxRate)
{
return subtotal + (subtotal * taxRate);
}
Given the same values, this function returns the same result. There is no need for an LLM to interpret the request or generate a calculation.
Compare that with asking an LLM to calculate an invoice total. The model may produce the correct answer, but it introduces unnecessary uncertainty into a task that should be handled by precise, testable business logic.
This distinction matters:
| Use deterministic code when... | Use an LLM when... |
|---|---|
| The rules are known and explicit | The rules need to be inferred from context |
| The result must be precise and repeatable | The input is ambiguous or unstructured |
| The task involves calculations or validation | The task involves interpretation or summarization |
| The same decision is made frequently | The situation is unfamiliar or requires flexible reasoning |
An LLM is not a replacement for every function in an application. It is a tool for the parts of a problem where language understanding and flexible reasoning add value.
There are two practical principles worth keeping in mind.
Consider a small business invoicing application.
Some features might include:
These are ordinary business rules. They belong in application code, supported by unit tests.
Other features may benefit from an LLM:
The goal is not to make the application less intelligent. It is to use the right tool for each responsibility.
Suppose an agent repeatedly investigates API failures and discovers that a particular class of failures is caused by expired access tokens.
Initially, the LLM may be useful because it needs to examine logs and understand the situation.
But once the pattern is confirmed, checking token expiration is straightforward:
public bool IsTokenExpired(DateTime expiresAtUtc)
{
return expiresAtUtc <= DateTime.UtcNow;
}
The application can now perform this check directly.
The same idea applies to repeated decisions in many domains:
A repeated LLM decision is an opportunity to ask: Can we capture what we have learned as a reliable software rule?
A practical AI agent does not need to choose between traditional code and an LLM. It can use both.
Consider a simplified workflow for our CI failure investigation agent:
CI Pipeline Failure
|
v
Read and collect logs
|
v
Check known patterns
|
+---------+---------+
| |
Recognized Unknown
| |
v v
Deterministic Send context
handler to LLM
| |
v v
Apply known fix Investigate
or recommendation and reason
| |
+---------+---------+
|
v
Validate the result
|
v
Report the outcome
The deterministic path handles situations the system already understands. The LLM path handles cases that need further investigation.
There is an important engineering detail here: recognizing a known pattern is not the same as proving its cause.
For example, an error code may have multiple possible causes. A rule-based handler should only take an automated action when the evidence and risk level justify it. Otherwise, it should provide a safe recommendation or escalate to the LLM or a human.
This is why validation, tests, observability, and clear fallback behavior remain essential.
There is another interesting part of this idea: AI coding tools can help us create the deterministic code that reduces future LLM usage.
For example, a developer can ask an AI coding assistant to:
The generated implementation still needs to be reviewed and tested. An AI-generated function is not automatically correct just because it is deterministic.
But AI can reduce the effort required to turn a discovered pattern into maintainable, tested software.
This creates an interesting feedback loop:
Over time, the system can handle a larger share of familiar situations without inference.
Potentially, yes.
A more capable agent does not necessarily need to make more model calls. In a well-designed system, repeated reasoning can gradually become reusable software.
That does not mean every LLM interaction should be replaced with a rule. Real-world inputs change, edge cases appear, and some decisions remain inherently ambiguous. The system still needs a way to handle cases that do not match known patterns.
The objective is not to eliminate LLM calls. It is to make each call worthwhile.
A useful way to think about the architecture is:
One of the biggest opportunities in AI-assisted development may not be adding AI to more parts of an application. It may be using AI to build better software around the places where AI is genuinely needed.
When an LLM solves a problem once, it provides an answer.
When we understand that solution, verify it, and turn it into tested code, we create something reusable.
That code can run repeatedly, predictably, and without another inference call.
The best AI architecture may not be the one that calls an LLM for everything. It may be the one that knows exactly when not to.
And perhaps one of the most valuable uses of AI is helping us write the code that makes future AI calls unnecessary.