Most systems that sort, route, or categorise work arrive at the same question eventually: “Which bucket does this belong in?”
A support request needs to reach the right team. An invoice needs a cost centre. A document needs a queue. At first, this sounds like a textbook supervised classification problem: collect examples, train a classifier, and let the model assign the label.
That approach works well. But it carries an assumption that is easy to miss: the set of labels is relatively stable.
In practice, it often isn’t.
Different customers use different terminology. Teams reorganise. Categories are renamed, split, merged, or retired. New products introduce new kinds of requests. The system is expected to adapt, but the underlying model only knows the world it saw when it was trained.
That is where a simple shift in perspective helps: instead of asking a model to memorise a customer’s past, keep the past available and search it when a new decision arrives.
A classical ML classifier is usually built from labelled examples: this type of ticket goes to that team; this kind of transaction belongs to that cost centre.
For one stable problem, that is a sensible arrangement. But in a multi-customer product, the work quickly multiplies. Each customer has its own labels, language, and distribution of work. In effect, you are not maintaining one classification problem. You are maintaining many of them, each changing independently.
The cost is not only the first training run.
A new customer has little or no labelled history, and there is no model until there is. The categories are lopsided: a few carry most of the volume while dozens have a handful of examples each — and those rare ones are exactly where a classifier is weakest. A model trained today cannot predict a category added tomorrow. And as the language of incoming work shifts, and categories change meaning, accuracy decays quietly between training runs (data and concept drift).
None of these issues is unusual on its own. Together, they turn retraining into a permanent operational burden: data preparation, evaluation, versioning, deployment, monitoring, and repeat.
The useful question isn’t “How can we retrain faster?”
It’s “Do we need to train at all?”
People make decisions this way all the time.
When an unusual request lands in a shared inbox, an experienced operator doesn’t reach for a rulebook. They remember — or look up — a similar request from last month and see how it was handled. The earlier case provides context, a likely answer, and often a reason for that answer.
The same principle can be applied to automated classification.
Rather than compressing a customer’s labelled history into a custom-trained model, keep that history searchable:
For example, if eight of the ten closest historical support tickets were sent to the billing team, that is strong evidence that the new ticket should go there too — especially if those examples are very similar.
Under the hood, this is a k-nearest-neighbours (kNN) classifier: one shared, general-purpose embedding model does the representing, and each customer gets its own index of history. It’s an old idea. What makes it practical now is that embeddings capture similarity in meaning, not just shared keywords.
from collections import defaultdictdef predict(record, index, k=10): neighbours = index.search(embed(record), k=k) scores = defaultdict(float) for neighbour in neighbours: scores[neighbour.label] += neighbour.similarity label = max(scores, key=scores.get) confidence = scores[label] / sum(scores.values()) return label, confidence, neighbours
The important part is not the few lines of code. It is the decision model: a prediction is based on comparable precedents, and those precedents remain visible.
A retrieval-based approach changes the shape of the problem.
A customer becomes predictable as soon as there is enough labelled history to search. There is no separate training cycle to wait for. If a new category is introduced, it becomes available once relevant examples are labelled and added to the index. As terminology changes, newer examples start steering decisions.
It shortens the delay between a change in the real world and a change in the system’s behaviour.
It also creates a simpler operating model. Instead of maintaining a specialised classifier for every customer, you run one shared embedding model and one serving path. The customer-specific knowledge lives in that customer’s history.
And unlike most model predictions, the result comes with its own evidence: “We routed this here because it closely resembled these earlier cases.”
That matters in any setting where people need to trust, review, or override an automated decision.
There is one obvious limitation: precedent only helps when precedent exists.
A new customer with almost no historical records cannot benefit much from retrieval. That gap needs a bridge. One that works well is an LLM asked to classify into a small set of broad categories — a coarse call it can make reliably with no customer history at all. Simple rules the customer already uses, or human review while examples accumulate, can fill the gap too.
The goal is not to pretend the system knows more than it does. It is to make the uncertainty explicit, use a temporary fallback, and gradually hand decisions over to retrieval as trustworthy history grows.
Retrieval is not a universal replacement for trained classifiers.
It reproduces bad labels just as faithfully as good ones. Old examples can point toward categories that no longer exist, so predictions should still be checked against the labels currently in use. And records that sound similar are not always supposed to be treated the same way; a general representation of meaning does not automatically understand the particular boundaries that matter to a business.
There are also practical trade-offs. Searching a growing collection of past records has a cost, and maintaining that collection well matters. A conventional classifier is the better choice when one customer has a stable label set, abundant high-quality data, and clear boundaries between categories.
The point is not that classifiers are obsolete. It is that retraining should not be the reflex.
When categories change faster than you can retrain, stop trying to freeze them in place.
Keep the history. Search it. Let precedent help make the next decision.
The catch, of course, is that history becomes part of the model. And if that history is incomplete, stale, or unreliable, the system will inherit those flaws too.
Route by Precedent, Not by Retraining was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.