Model Cascade: making LLM classification cheaper

A developer introduced Model Cascade, a technique that uses a cheap model's confidence scores to decide when a large, expensive LLM is needed for classification tasks, potentially reducing costs significantly. The approach, based on the BARGAIN paper, calibrates a confidence threshold offline and routes most records to the small model, reserving the oracle for low-confidence cases. The developer also highlighted the follow-up Task Cascades paper, which adds optimizations for further cost savings.

Many LLM workloads are classification tasks. This can get expensive, and I believe it is going to become more and more important, especially with the proliferation of software factories. So what is Model Cascade ? In short, it is a way to make a deterministic system around a cheap model and make it give us the same results as the expensive model. The LLM we use gives us the probability of every token in the output, same probability model used to generate the response. We put all the tokens of the response together, and we get the probability of the response. Now the smart part of the Model Cascade: flowchart TB subgraph CAL "Calibrate once, offline" S "Sample ~500 records" -- O1 "Label sample with oracle" O1 -- T "Try every observed confidence <br/ value as a threshold" T -- P "Pick cheapest threshold that<br/ meets the accuracy target" O1 -- G "Check that proxy confidence agrees <br/ with oracle labels" end subgraph ROUTE "Route every record, at scale" R "Record" -- PX "Proxy: small, cheap model" PX -- L "Label + confidence score,<br/ from logprob" L -- D{"Confidence above threshold?"} D -- |"yes, most records"| K "Keep proxy label" D -- |"no, few records"| O2 "Oracle: large, expensive model" K -- OUT "Final labels" O2 -- OUT end P -. "sets threshold" .- D Below is a summary of the BARGAIN paper I used to learn about this principle. It is more detailed than the first part, so if you want to learn more, read on. Or read the full paper here: https://github.com/ucbepic/BARGAIN https://github.com/ucbepic/BARGAIN Across eight datasets, the BARGAIN paper reports up to 86% more cost reduction than competing methods. The follow-up Task Cascades paper adds three optimizations: rewriting prompts into simpler surrogate questions, reading only the most relevant document chunks, and searching over candidate cascades for the cheapest sequence. These cut costs a further 48.5% on average. Unlike FrugalGPT, BARGAIN gives statistical guarantees. Unlike SUPG, they hold at any sample size, and it uses adaptive sampling and better estimation. pip install bargain Dependencies are numpy, pandas, tqdm, and openai. You can swap providers by defining your own proxy and oracle. Examples live in examples/ https://github.com/ucbepic/BARGAIN/tree/main/examples . Run the Supreme Court one from that directory; it loads court opinion.csv by relative path.The Supreme Court numbers come from one run and may change with model versions, API behavior, or dataset changes. BARGAIN A on a sample with your target and delta to see what fraction the proxy can handle.Pass logprobs and top logprobs to ChatOpenAI , then read the scores from response metadata : python import math from langchain openai import ChatOpenAI llm = ChatOpenAI model="gpt-5-nano", temperature=0, logprobs=True, top logprobs=5, response = llm.invoke "Does the text 'zebra' mention an animal? Answer with only True or False." content = response.response metadata "logprobs" "content" first token = content 0 print first token "token" , first token "logprob" e.g. "True" -0.01 print math.exp first token "logprob" probability, e.g. 0.99 Each entry in content is one token with its own logprob. The snippet reads only the first token, which works because the prompt forces a single-word answer. For a multi-token answer, sum all token logprobs instead: total logprob = sum t "logprob" for t in content For classification, prompt for a single word so the response is one token, then use that token's logprob as the confidence score. The top token is the model's answer. For a label it did not pick, look inside top logprobs : candidates = {c "token" : c "logprob" for c in first token "top logprobs" } score for true = candidates.get "True" If a label is absent from top logprobs , its score is unavailable. Do not treat a fallback value as the model's actual score. python def proxy func self, data record: str : response = llm.invoke self.task.format data record first = response.response metadata "logprobs" "content" 0 return first "token" , first "logprob" For binary classification, request enough top logprobs entries to include both labels, normalize the two label probabilities, and return the probability of the selected label: python import math def proxy func self, data record: str : response = llm.invoke self.task.format data record first = response.response metadata "logprobs" "content" 0 candidates = { item "token" : math.exp item "logprob" for item in first "top logprobs" } true prob = candidates.get "True", 0.0 false prob = candidates.get "False", 0.0 total = true prob + false prob if not total: return False, 0.0 true prob /= total false prob /= total output = true prob false prob return output, true prob if output else false prob temperature=0 for more repeatable answers. It does not guarantee identical responses, logprobs are not calibrated probabilities of correctness, and some reasoning models disallow temperature. response metadata has no logprobs , the provider did not return them. logprobs and top logprobs are direct ChatOpenAI arguments; other provider-specific parameters go in extra body .