Complex AI workflows can become expensive surprisingly quickly. Not necessarily because the models themselves are expensive, but because small design decisions are repeated hundreds or thousands of times.
A system message that is too short to benefit from input caching, a model that explains every decision when nobody needs those explanations, a task that gives the model too much freedom, or a single step that gets stuck repeating the same function calls can have a surprisingly large effect once the workflow grows.
I wanted to look at this using a real process rather than a synthetic benchmark.
The experiment is based on a text-analysis workflow that splits a document into sentences, tokens and multi-word terms, and then performs several syntactic, secondary and free-form classifications. Each trial involved several thousand API requests and several million tokens. I ran four trials using the same two source texts under different conditions, and compared token usage, caching, estimated cost and preliminary result quality.
The whole experiment was built and executed inside AIDBDeveloper, the platform I introduced in a previous article. If you want some background on the application, its modular dashboards and its provider-independent context management, you can find that introduction here.
The video attached to this article contains the complete walkthrough. What follows is a shorter overview of the experiment and some of the results that I found most interesting.
The first design decision was not to ask one model to analyse the whole document.
The application remains responsible for orchestration, data storage and everything that can be done deterministically. AI is used only for the individual tasks that actually require interpretation.
The process starts by extracting sentences from the source text. For these trials I used GPT 5.6 Sol with low reasoning effort. Each sentence is then split into words, numbers and punctuation marks using GPT 5.4 mini. Multi-word units — which I call terms — are extracted with GPT 5.6 Terra at medium reasoning effort, and Terra is also used for the subsequent classification steps.
Token classifications are processed in batches of five, with ten model instances working in parallel on different sentences. Later steps reuse information produced by previous ones whenever that can reduce the model's decision space.
This was one of the ideas I wanted to test throughout the experiment: instead of asking the model to be as intelligent as possible, try to make each individual task simple enough that it needs as little intelligence as possible.
I used two short articles about logical fallacies that I had written years ago. Each document was processed twice.
The first trial on TEXT 1 used shorter system messages in an attempt to save input tokens. That turned out not to be particularly successful. Some steps did not use the input cache, term extraction was far too permissive, and the workflow produced many more classifications than necessary.
For the second trial, the system messages were made considerably more explicit. Cache usage improved substantially and the number of extracted terms and classifications fell. TEXT 2 was then processed under essentially the same improved configuration. In its second trial, however, I deliberately removed one apparently trivial instruction: after finishing its function calls, the model was no longer told to answer with only a single full stop.
That provided a useful demonstration of how expensive unnecessary politeness can become.
The infographic above summarizes some of the structural differences between the four runs.
One result remained completely stable: tokenization. TEXT 1 produced exactly 1,650 tokens in both trials, while TEXT 2 produced 1,762 in both.
Term extraction was a different story. TEXT 1 went from 1,114 terms in the first trial to 431 in the second, after improving the instructions. The total number of classifications also fell from 15,673 to 9,580.
That is not simply a token-saving exercise. In the first trial the model was identifying almost any group of words as a term, so the larger result was actually a symptom of poorer analysis.
The two TEXT 2 trials were much closer structurally because the system messages and classification functions were essentially unchanged.
The individual source documents were short, but the resulting workload was not.
Across the relevant trials, total usage ranged from approximately 3 to 8 million tokens per trial, with roughly 2,000 to 3,000 requests each. Output remained around 600,000 tokens in most runs, except for the final trial where it rose to roughly 800,000.
The usage screenshot gives some idea of the scale involved. Once a process reaches this size, looking only at a final API bill is not particularly useful. You need to know which step generated those tokens, which context was used and what the model was doing at that moment.
For that reason, every model-backed execution instance in my workflow records its configuration, start and end times, input and output data, token usage and a copy of the context that produced the result. That made it possible to analyse the workflow internally instead of treating the whole experiment as one large anonymous API workload.
One of the main goals of the TEXT 1 comparison was to improve input-cache usage.
It worked.
Between the two TEXT 1 trials, the estimated cost of uncached input fell by almost 73%. When all three input-related categories — uncached input, cached input and cache writes — were combined, the reduction was approximately 18%.
But something else happened at the same time.
The output cost also fell by almost 15%.
That mattered much more than I initially expected because output tokens represented around 64% of the total estimated cost in that comparison. The total theoretical cost therefore fell from $11.39 to $9.59, approximately 16% overall.
In other words, I had been trying to optimize input caching, and caching did improve, but a large part of the final saving actually came from producing less output.
The chart also shows the two TEXT 2 trials. The last one is deliberately worse, because the model was allowed to generate explanatory responses after completing its function calls.
That apparently small change was enough to push the estimated cost from $11.67 to $14.97.
Function calling introduces a slightly annoying detail. After the model has finished making all the requested function calls, it normally returns a final assistant message. In an interactive conversation that makes sense. In an automated classification process, it often does not.
The useful information has already been delivered through the function calls. A final message saying what the model has done is just additional output.
In most of the trials, the system instructions explicitly told the model to finish with a single full stop.
When I removed that instruction in the second TEXT 2 trial, the model began explaining its work again and again. Nobody was going to read those explanations, but they were still generated and billed.
One classification step gives a particularly clear example: its output increased from roughly 234,000 tokens to 426,000 tokens.
Almost twice as much output for no useful additional result.
That was one of the simplest lessons from the whole experiment: in an automated workflow, a natural-language answer can be pure overhead.
The second TEXT 2 trial also produced another interesting problem.
During one classification step, a model instance became stuck repeatedly calling the same functions for the same batch of five tokens, using exactly the same parameters.
I managed to get it unstuck without stopping the complete process, but its token usage had already increased dramatically.
The interesting part was that cached-input percentage barely changed. The repeated context was still being cached efficiently; the process was simply repeating far too much work.
This is a useful reminder that good cache statistics do not necessarily mean that a workflow is efficient.
You can cache an error very efficiently.
I also added a parameterized cost calculation that allowed me to apply a different model's rates to exactly the same recorded token usage.
As an experiment, I replaced GPT 5.6 Terra with GPT 6 Astra in the calculation.
The trials that cost roughly $10–15 using the actual model mix rose to approximately $42–65 with Astra pricing — around 4.5 times as much.
This is only a cost simulation. It does not imply that Astra would actually consume the same number of tokens or produce identical results.
But it illustrates the scale of the decision.
Choosing a model substantially more capable than the task requires can outweigh many of the smaller optimizations applied later.
There is little value in making a workflow cheaper if the results become worse.
The preliminary quality review produced a very uneven picture.
Sentence extraction was extremely consistent. Tokenization was even more stable: it produced identical results across equivalent trials.
Word-level syntactic classification still needs some refinement, but the general quality was reasonably good.
The harder part was multi-word analysis. Term extraction remained weak, even after improving the instructions. Syntactic classification of terms was considerably poorer than classification of individual words, and the secondary classification of terms was clearly inadequate. The free-form word tags looked more promising, although those are inherently more subjective.
This changed the optimization question again.
A cheap and stable step is probably not where more engineering effort should go. A step that is expensive and still produces poor results is a much better candidate for redesign.
In some cases, the right optimization is not a better prompt. It is a different process.
The original purpose of these trials was mainly to explore context management and caching, but the broader conclusion ended up being more useful.
Optimizing an AI workflow is not simply about reducing token counts.
It means deciding which work genuinely requires a model, breaking that work into sufficiently small tasks, constraining the model's choices whenever possible, providing known information instead of making it infer it again, removing context that is no longer useful, avoiding unnecessary natural-language output, choosing the least expensive model that can perform each task reliably, and recording enough information to know where the cost is actually coming from.
The application should do everything it already knows how to do.
The model should be used for the uncertain parts.
And both cost and result quality need to be measured together.
The video goes much deeper into all of this. It includes the context structure, input caching, function calling and structured responses, the complete workflow, execution logs, examples of actual contexts, detailed token and cost charts, some of the errors I found, and the preliminary quality review.
Watch the full video on YouTube This is still only the preliminary stage of the larger text-analysis experiment. The next work will focus much more heavily on improving and validating the linguistic results themselves.
Thanks for reading.