A few months ago we had to work on a system that turned free text — sales reps’ notes, call transcripts, forwarded emails — into structured CRM records: clients, opportunities, next steps, cumulative summaries. The business logic was well thought out, and the prompt had been written by someone who knew the sales domain inside out.
The problem was elsewhere: it took between 44 and 60 seconds per entry.
Fifty-four seconds on average. Nearly a minute of spinner every time a sales rep saved a note from their phone. Today that same process responds in 12-17 seconds: up to four times faster, with the same functionality, the same base model and not a single field dropped from the result.
This article explains how we brought that process under 20 seconds without removing a single feature. There is no prompt trick involved. The change was architectural, and the conclusion — which we already suspected, but which here got measured — is that a giant prompt is not an architecture, it is a postponement of the architecture.
The project belongs to a client, so everything below is anonymised: no names, no literal prompts, no business data. What is real is the shape of the problem, the measurements and the decisions.
Step 1: everything in a single call #
The starting point was the classic one, and I do not say that with contempt: it is how 90% of AI features that get validated in a chat window first begin. A single model call, with a system prompt of several thousand words, doing all of this at once:
- Detecting which commercial events appeared in the text
- Classifying them by type
- Interpreting the subtext and deciding which was the most important move
- Building a JSON per event with more than thirty fields
- Generating narrative: state snapshots, tactical insights, briefings, call scripts
- Consolidating the cumulative summaries of every client and every opportunity, respecting history
It worked. In a chat, with a nice example, it worked rather well. But in production, at temperature 0.7 and ten consecutive requests, these were the timings:
| Metric | Time |
|---|---|
| Minimum | 44.2 s |
| Maximum | 60.7 s |
| Average | 54.1 s |
| P95 | 60.7 s |
Dropping the temperature to 0.2 — the usual suspect — changed nothing: 55.0 s average. At that point we knew the problem was not configuration.
The starting point: one call, five responsibilities, an enormous output and 54.1 seconds of waiting.
Why a single call was the problem
There are three reasons, and it is worth separating them because they are fixed in different ways.
Latency is driven by output, not input. An LLM generates tokens one at a time. You can feed it 8,000 tokens of context and it will process them quickly, but every token it writes costs time. A prompt asking for a thirty-field JSON plus five narrative texts for each detected event is not slow because it thinks hard: it is slow because it writes a huge amount. And if the input text contains three commercial events, the output triples.
Attention gets diluted. When you put formatting rules, field definitions, tone instructions, prioritisation criteria and conditional logic into the same prompt, they all compete for the same attention budget. A naming rule can steal reasoning capacity from event detection. It is the same effect as asking a person to, in a single pass, read a text, classify it, fill in a thirty-field form, write three paragraphs of strategic analysis and update a history log. They can do it. They will do it worse than if you ask for it in parts.
Reasoning is opaque. If the system misdetects an event, the error propagates into structuring, narrative and consolidation. There is no way to know where it failed, or to retry just that part. And you cannot optimise anything in isolation either: if the narrative is slow, you cannot remove it without breaking the rest of the output.
That third point is the one that gets expensive in the medium term, and the least visible in a demo.
Step 2: split the prompt to see, not to run faster #
We did not make the first change to optimise. We made it to understand.
We separated the single call into phases with explicit inputs and outputs, measured each one independently and left everything else untouched. No parallelisation, no different models, no prompt trimming. Just splitting and timing.
The result was a modest improvement — from 54 to 31.5 seconds, essentially because each call generated less text at once — but that was not the point. The point is that we went from a black box to an observable system. For the first time we could answer “where are the seconds going?”.
Same work, three measurable phases. The time gain was modest; the visibility gain was total.
And the answer was clear: the structuring step was still a complete mini-brain. It detected, structured, prioritised, wrote narrative and consolidated entities. We had split the monolith into pieces, but one of the pieces was still a monolith.
This intermediate step looks skippable when you tell the story afterwards. It is not. Had we jumped straight to the final design, we would have optimised blind — and probably optimised the wrong step.
Step 3: specialise, parallelise and get things off the critical path #
With the measurements in front of us, the redesign wrote itself. Three decisions:
One responsibility per call. The big step was split again, separating the operational (data going into the database) from the narrative (text a human reads).Whatever depends on nothing, in parallel. Several steps only needed the output of the initial detection, not each other’s. There was no reason to run them in series.Whatever does not block, to the background. Narrative is the slowest thing to generate and the least urgent: nobody reads a meeting briefing in the same second they save a note.
The pipeline ended up like this:
| Step | What it does | Model | Time | Blocks the user? |
|---|---|---|---|---|
| 1. Detection | Identifies and classifies events in the text | Light | 3-5 s | Yes |
| 2. Structuring | Turns each event into an operational record | Main | 5-10 s | Yes |
| 3. Consolidation | Updates the cumulative summaries of each entity | Main | 5-10 s | Yes, in parallel with 2 |
| 4. Automatic actions | Detects explicit data changes that need no judgement | Light | 2-4 s | Yes, in parallel with 2 and 3 |
| 5. Enrichment | Generates narrative, insights and calendar content | Main | 10-15 s | No |
Steps 2, 3 and 4 all start the moment step 1 finishes, because all three consume the same thing: the list of detected events. Step 5 is launched after responding to the user and updates the records when it completes.
The final design: one light step opening the graph, three steps in parallel, and all the narrative off the critical path.
The numbers
| Before | Sequential pipeline | Parallel pipeline | |
|---|---|---|---|
| Total time | 54.1 s | 31.5 s | 18.6 s |
| Perceived time | 54.1 s | 31.5 s | 12-17 s |
In one sentence: the process ended up almost 3 times faster end to end, and up to 4 times faster in what the user perceives. The wait went from 54.1 seconds — nearly a minute in front of a spinner — to 12-17 seconds. Seventy-two per cent less, with the same functionality and the same base model.
And that is not even the harshest comparison. The worst case measured on the original system was 60.7 seconds: over a minute to save a note. Today the slowest stretch of the whole process — the narrative, 10-15 seconds — is no longer on the critical path. The user never waits for it.
What 39 seconds less actually means
Percentages get forgotten; accumulated time does not. At 100 entries a day, a modest volume for a mid-sized sales team:
| Before | Now | |
|---|---|---|
| Accumulated waiting per day | 1 h 30 min | 25 min |
| Per month (20 working days) | 30 hours | 8 hours |
That is 22 hours a month — almost three full working days — that a team stopped spending in front of an unresponsive screen.
But the important change is behavioural, not arithmetic. Above 40 seconds, people fire the request and leave: they open another tab, get in the car, forget. Below 20, they stay and look at the result — and if something came out wrong, they fix it there and then, while they still remember the conversation they just had. That difference shows up on no latency chart, and it is the one that decides whether the feature gets used or abandoned.
And there is a part that does not show up in the table either: we now know which step fails when something fails, we can retry only that one, we can change the model of one step without touching the others, and we can add a new step without rewriting a 4,000-word prompt.
The rules we took away #
If you have an AI feature doing too much in a single call, this is the order we would attack it in today: 1. Separate data from narrative. It is the most profitable split of all and it almost always exists. Operational fields are short, deterministic and validatable against a schema. Narrative text is long, subjective and benefits from a prompt with personality. Mixing them forces the model to switch register mid-generation, and makes every regeneration expensive: if all you want is to tweak the tone of a briefing, you should not have to re-extract the data.
2. Send everything the user does not read now to the background. The criterion is not importance, it is moment of consumption. A summary that will be read tomorrow before a meeting can take 20 seconds to generate. A field that appears on screen at save time cannot.
3. Parallelise by real dependencies, not by logical order. It is easy to chain the steps in the order you thought of them. Draw what each step actually needs: in our case, three of the five only depended on the first.
4. Use the cheap model where there is no judgement involved. Classifying, detecting and extracting explicit changes is mechanical work. It does not need the most expensive model: it needs clear instructions and a low temperature. Save the good model for where interpretation happens.
5. Start by measuring, not optimising. Splitting in order to observe is a step with value of its own even if it does not improve timings. Without per-phase measurements, optimising is guessing.
What this architecture charges you #
Anyone selling you agent pipelines without mentioning this part is selling you half the story.
More failure points. Five calls are five things that can time out, return broken JSON or stop halfway. You need per-step retries, schema validation on every output, and a clear policy for what happens when a background step fails (retry? leave it empty? notify?). None of that code existed when everything was one call.
Repeated input tokens. Each step needs its share of context, and some context travels in several calls. Input cost goes up. In our case it was more than offset by the drop in output tokens — the expensive ones — and by prompt caching, but that is a calculation you have to run, not assume.
Voice consistency. Once you split the narrative into its own step, that step no longer sees the full prompt with all of the assistant’s personality. If you do not pass it enough tone context, the result comes out correct but flat. It took us a couple of iterations to tune.
Real orchestration. Someone has to handle parallelism, timeouts, database write ordering and what happens if the background step finishes after the user has manually edited the record. It is ordinary engineering work, but it is work.
None of these trade-offs made us doubt the decision. All of them cost us days we would not have spent maintaining the monolith.
The most important change was not technical #
There is one benefit that shows up in no metric and that, over time, turned out to be the most valuable.
A project like this holds two different kinds of knowledge. There is the knowledge of whoever defines what counts as a relevant commercial event and how it is structured, which is domain and data knowledge. And there is the knowledge of whoever defines how the assistant speaks, which is product, tone and brand knowledge. With a monolithic prompt, those two people edit the same 4,000-word block of text and constantly get in each other’s way. Any tone adjustment forces a revalidation of data extraction, and any extraction adjustment can wreck the voice.
Splitting the steps gives each one an owner. The person who defines the voice iterates on the enrichment prompt without touching anything operational. The technical team adjusts extraction without rewriting the personality. Each can be versioned, compared and reverted separately.
Put another way: we stopped trying to make the model do everything and started designing how we wanted it to think. And the side effect was that it also became clear who decides what.
How to replicate this in your project #
If you are at the starting point — an AI feature that behaves well in the demo and so-so in production — this is the order we would follow: Instrument before touching anything. Measure latency, input tokens and output tokens per request. If you cannot say how many tokens your worst case generates, you do not yet know what your problem is.List the responsibilities of the current prompt. If enumerating them yields more than three distinct verbs (detect, classify, write, consolidate…), you have a pipeline hidden inside a prompt.Split and measure, without optimising. Accept that the first split version will improve things only slightly. The goal is to see.Draw the real dependency graph and parallelise whatever does not depend on anything else.Get everything narrative off the critical path and return it when it is ready.Set up a per-step evaluation before optimising further: a set of sample inputs with expected outputs, for each phase. Without it, every speed improvement is a bet placed against quality.
Point 6 is the one most people skip and the one that costs the most. A modular architecture without per-module evaluation is a modular architecture you do not know works.
None of this required a better model, a bigger token budget or an agent framework. It required deciding what the model had to think about at each moment — and that change, measured, was worth 39 seconds per entry.
If you are building AI features on top of LLMs and running into response times that do not add up, or a prompt nobody on the team dares to touch any more, let’s talk. It is a pattern we have seen often enough to recognise quickly. And if the economics of this decision interest you, what running LLMs in production actually costs breaks down token, infrastructure and optimisation numbers with current data.