# Why Your LLM Classifier Doesn't Need the Taxonomy: Hypothetical Classification with Embeddings > Source: > Published: 2026-08-14 14:47:39+00:00 Classifying free-text queries into a fixed product taxonomy is one of the most common LLM workloads in production, and one of the most quietly expensive ones. A typical e-commerce catalog ships with 400 to 800 legal categories, each spelled out as a fully qualified path like `Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables` . Every time a search query needs a label, the whole vocabulary has to travel with the prompt: as a giant Pydantic `Literal` , as a JSON schema, or as a few hundred lines of enum text. Tokens are not free, and neither is latency. The prompt grows, the small model you wanted to use starts misbehaving, and the request that should have cost a fraction of a cent now needs the biggest model on the roster just to keep the output valid. There is a cheaper pattern that most teams never consider. Instead of forcing the model to choose from the real taxonomy, you let a small, cheap model invent fake categories that sound plausible, and then resolve those hallucinations back into the real vocabulary with an embedding lookup. It sounds backwards. It works surprisingly well. This article walks through the idea, the implementation, the failure modes, and the measurements you should collect before trusting it. The conventional approach is structured outputs. You define the legal vocabulary as a type, hand it to the provider, and ask for a constrained decode: ``` python from typing import Literal from pydantic import BaseModel, Field FullyQualifiedClassifications = Literal[ "Furniture / Bedroom Furniture / Beds & Headboards / Beds", "Furniture / Living Room Furniture / Chairs & Seating / Accent Chairs", "Rugs / Area Rugs", # ... times 500 ] class QueryClassification(BaseModel): """Structured representation of a search query.""" classifications: list[FullyQualifiedClassifications] = Field( description="Possible classifications for the product." ) ``` Constrained decoding guarantees that the answer is drawn from the legal set, which is a strong property. But it carries a hidden cost that grows with the catalog. The schema has to be sent with every request, so every call pays the full vocabulary in input tokens. Classification is a high-volume, low-value-per-request workload, which is exactly the wrong place to spend tokens. The second problem is that the guarantee is only as good as the model's willingness to stay inside the schema. Small models choke on a 500-element `Literal` ; they either truncate it, or they start hallucinating categories that were never in the list, which defeats the entire purpose. The teams I have watched hit this wall usually respond by upgrading the model, and the unit economics of the whole pipeline follow them upward. The key realization is that the model does not need to know the taxonomy to know what kind of thing a query is talking about. A query like "brown coffee table" is obviously a piece of living room furniture. The hard part is not understanding the query; it is mapping that understanding onto a specific node in a tree with hundreds of leaves. So stop sending the tree. Send an example of the *shape* of a classification, and ask the model to invent new ones: ``` hallucination_prompt = f""" Your task is to create novel, never seen before furniture, home goods, or hardware classifications that best fit a search query. Product classifications look like: Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables Décor & Pillows / Decorative Pillows & Blankets / Throw Pillows Furniture / Bedroom Furniture / Dressers & Chests Kitchen & Tabletop / Kitchen Organization / Food Storage & Canisters Baby & Kids / Toddler & Kids Bedroom Furniture / Kids Beds Here is the query to generate classifications for: brown coffee table """ ``` The prompt now fits in a few hundred tokens instead of several thousand. The model has no legal vocabulary to violate, so it will happily produce a made-up path like `Furniture / Living Room / Tables / Coffee` . That output is useless as a label by itself. But it is a precise description of the query's meaning, written in the same style as the real taxonomy. Now the problem becomes: given a hallucinated path, find the real category it points at. Embeddings make this a nearest-neighbor search instead of a string-matching problem, which matters because the fake path and the real path share almost no literal characters. The setup is small enough to live in memory. Embed every real category path once at startup with a compact sentence model, keep the vectors in a NumPy array or a tiny vector store, and embed the hallucinated path at request time. The label is the real category whose vector has the highest dot product with the fake one: ``` python import numpy as np from sentence_transformers import SentenceTransformer model = SentenceTransformer("all-MiniLM-L6-v2") real_paths = [...] # the full taxonomy, loaded once real_vectors = model.encode(real_paths, normalize_embeddings=True) def resolve(hallucinated_path: str) -> str: v = model.encode([hallucinated_path], normalize_embeddings=True)[0] idx = int(np.argmax(real_vectors @ v)) return real_paths[idx] ``` For a catalog of a few hundred categories, this is a single matrix-vector product, well under a millisecond even on a laptop CPU. The expensive part of the pipeline is now the tiny LLM call, and that is the whole point: you have moved the cost from a big model with a giant schema to a small model with a short prompt plus a local lookup that costs nothing. The pattern sounds like it should fail, so it is worth being precise about why it does not. Small models are bad at constrained generation over large vocabularies: their attention degrades as the constraint set grows, and their sampling drifts into invalid outputs. But they are good at open-ended paraphrase and at writing text in the style of an example. Inventing a plausible category path for "brown coffee table" is a style-matching task, not a constraint-satisfaction task. It is exactly the kind of fluent generation that even a 1B-parameter model does reliably. The embedding model, meanwhile, is the component that actually holds the taxonomy. Sentence embeddings map both the fake path and the real paths into a space where meaning, not surface form, determines distance. `Furniture / Living Room / Tables / Coffee` and `Furniture / Living Room Furniture / Coffee Tables & End Tables / Coffee Tables` share almost no tokens, but they land close together because the embedding space was trained on paraphrases. Each component does the part it is good at, and neither needs to be strong at the other's job. The pattern is not a universal replacement for structured outputs; it is a trade. You should measure three failure classes before committing. The first is vocabulary drift. A small model inventing categories will sometimes produce a path that is semantically generic, like `Furniture / Tables` , which resolves to whichever real node is closest in the embedding space. If your taxonomy has many tables, the nearest neighbor may be wrong even though the query was specific. This is the dominant failure mode, and its rate is a property of your taxonomy, not of the model. The second is resolution ambiguity. Some queries are genuinely under-specified: "black stand" could be a phone stand, a monitor stand, or a plant stand. The hallucinated path will be confidently wrong because the model had no way to know. This is not a bug in the pattern; structured outputs fail the same way, but they fail by returning a confidently wrong label with a schema-valid appearance. The third is taxonomy churn. If your catalog changes often, the embedding index must be rebuilt, and a newly added category is invisible until then. Rebuilding a few hundred embeddings takes seconds, so this is an operational detail rather than a blocker, but it needs to be part of the deploy pipeline. The pattern shines when the vocabulary is large, the queries are short, and you want the classification step cheap enough to run on every request without thinking about the bill. It is a poor fit when the taxonomy is tiny, when labels must be exact strings with no tolerance, or when you cannot tolerate any rate of misclassification and prefer the deterministic failure of constrained decoding. In practice you want a confidence floor on the resolution step, not just an argmax. The dot product between the hallucinated vector and its nearest real neighbor tells you how convinced the system is. A path that resolves at 0.45 cosine similarity is a guess; one that resolves at 0.85 is a lock. The cheap fix is a threshold with two fallback tiers: ``` php def classify(query: str, threshold: float = 0.6) -> tuple[str, str]: fake = invent_categories(query) # small LLM, short prompt real, score = resolve(fake) # embedding lookup if score >= threshold: return real, "auto" return run_structured_classification(query), "fallback" ``` Below the threshold you can escalate to the structured-output path on a bigger model, or route to a human review queue. Because the cheap path handles the bulk of traffic, the escalation path only fires on the ambiguous fraction, and your average cost stays low even though the worst case still uses the expensive machinery. A second guardrail worth adding is a short denylist of generic leaves that are known to swallow unrelated queries, so a generic hallucination can be rejected before it resolves to a useless bucket. The objection to this pattern is usually "how do I know it is accurate?" You can get surprisingly far without a hand-labeled test set. Sample a few hundred real search queries from your logs, run both the cheap path and the structured-output path on the same queries, and diff the labels. Disagreement does not mean the cheap path is wrong, but every disagreement is a case to eyeball. In practice the disagreements cluster into two piles: cases where the cheap path is semantically right and the schema path was forced into a wrong but valid leaf, and cases where the cheap path resolved to a generic neighbor. The first pile is evidence the pattern is working; the second pile tunes your threshold. Track the resolution score distribution as a metric. If the mean score drifts down over weeks, the model vendor changed something or your taxonomy drifted; either way you want to know before the misclassification rate moves. A small dashboard with three numbers, mean score, escalation rate, and disagreement rate against the structured path, is enough to run this pattern in production with confidence. The complete pipeline is short enough to hold in your head: a tiny model invents a category path from a prompt of a few hundred tokens, an embedding lookup resolves that path to a real leaf in under a millisecond, and a similarity threshold decides whether the answer ships or escalates. The taxonomy never travels with the request, the cheap model stays cheap, and the expensive machinery only sees the genuinely ambiguous cases. Hypothetical classification is one of those ideas that looks like a hack until you measure it. The vocabulary is the expensive part of classification, and the embedding model lets you carry it once, locally, instead of shipping it on every request. If your classification workload is currently paying the taxonomy tax on a big model, it is worth a weekend experiment: replace the schema with a hallucination prompt, add the lookup, and look at the disagreement rate. The numbers will tell you quickly whether the backwards idea is the right one. Originally published on [Dispatch](https://dispatch-blog.hashnode.dev/why-your-llm-classifier-doesn-t-need-the-taxonomy-hypothetical-classification-with-embeddings).