{"slug": "model-cascade-making-llm-classification-cheaper", "title": "Model Cascade: making LLM classification cheaper", "summary": "A developer introduced Model Cascade, a technique that uses a cheap model's confidence scores to decide when a large, expensive LLM is needed for classification tasks, potentially reducing costs significantly. The approach, based on the BARGAIN paper, calibrates a confidence threshold offline and routes most records to the small model, reserving the oracle for low-confidence cases. The developer also highlighted the follow-up Task Cascades paper, which adds optimizations for further cost savings.", "body_md": "Many LLM workloads are classification tasks. This can get expensive, and I believe it is going to become more and more important, especially with the proliferation of software factories.\n\nSo what is **Model Cascade**? In short, it is a way to make a deterministic system around a cheap model and make it give us the same results as the expensive model.\n\nThe LLM we use gives us the probability of every token in the output, same probability model used to generate the response. We put all the tokens of the response together, and we get the probability of the response.\n\nNow the smart part of the Model Cascade:\n\n```\nflowchart TB\n    subgraph CAL[\"Calibrate once, offline\"]\n        S[\"Sample ~500 records\"] --> O1[\"Label sample with oracle\"]\n        O1 --> T[\"Try every observed confidence <br/> value as a threshold\"]\n        T --> P[\"Pick cheapest threshold that<br/>meets the accuracy target\"]\n        O1 --> G[\"Check that proxy confidence agrees <br/> with oracle labels\"]\n    end\n\n    subgraph ROUTE[\"Route every record, at scale\"]\n        R[\"Record\"] --> PX[\"Proxy: small, cheap model\"]\n        PX --> L[\"Label + confidence score,<br/>from logprob\"]\n        L --> D{\"Confidence above threshold?\"}\n        D -->|\"yes, most records\"| K[\"Keep proxy label\"]\n        D -->|\"no, few records\"| O2[\"Oracle: large, expensive model\"]\n        K --> OUT[\"Final labels\"]\n        O2 --> OUT\n    end\n\n    P -. \"sets threshold\" .-> D\n```\n\nBelow is a summary of the BARGAIN paper I used to learn about this principle. It is more detailed than the first part, so if you want to learn more, read on.\n\nOr read the full paper here: [https://github.com/ucbepic/BARGAIN](https://github.com/ucbepic/BARGAIN)\n\nAcross eight datasets, the BARGAIN paper reports up to 86% more cost reduction than competing methods.\n\nThe follow-up Task Cascades paper adds three optimizations: rewriting prompts into simpler surrogate questions, reading only the most relevant document chunks, and searching over candidate cascades for the cheapest sequence. These cut costs a further 48.5% on average.\n\nUnlike FrugalGPT, BARGAIN gives statistical guarantees. Unlike SUPG, they hold at any sample size, and it uses adaptive sampling and better estimation.\n\n```\npip install bargain\n```\n\nDependencies are numpy, pandas, tqdm, and openai. You can swap providers by defining your own proxy and oracle.\n\nExamples live in [ examples/](https://github.com/ucbepic/BARGAIN/tree/main/examples). Run the Supreme Court one from that directory; it loads\n\n`court_opinion.csv`\n\nby relative path.The Supreme Court numbers come from one run and may change with model versions, API behavior, or dataset changes.\n\n`BARGAIN_A`\n\non a sample with your target and delta to see what fraction the proxy can handle.Pass `logprobs`\n\nand `top_logprobs`\n\nto `ChatOpenAI`\n\n, then read the scores from `response_metadata`\n\n:\n\n``` python\nimport math\nfrom langchain_openai import ChatOpenAI\n\nllm = ChatOpenAI(\n    model=\"gpt-5-nano\",\n    temperature=0,\n    logprobs=True,\n    top_logprobs=5,\n)\n\nresponse = llm.invoke(\n    \"Does the text 'zebra' mention an animal? Answer with only True or False.\"\n)\n\ncontent = response.response_metadata[\"logprobs\"][\"content\"]\nfirst_token = content[0]\nprint(first_token[\"token\"], first_token[\"logprob\"])        # e.g. \"True\" -0.01\nprint(math.exp(first_token[\"logprob\"]))                     # probability, e.g. 0.99\n```\n\nEach entry in `content`\n\nis one token with its own logprob. The snippet reads only the first token, which works because the prompt forces a single-word answer. For a multi-token answer, sum all token logprobs instead:\n\n```\ntotal_logprob = sum(t[\"logprob\"] for t in content)\n```\n\nFor classification, prompt for a single word so the response is one token, then use that token's logprob as the confidence score.\n\nThe top token is the model's answer. For a label it did not pick, look inside `top_logprobs`\n\n:\n\n```\ncandidates = {c[\"token\"]: c[\"logprob\"] for c in first_token[\"top_logprobs\"]}\nscore_for_true = candidates.get(\"True\")\n```\n\nIf a label is absent from `top_logprobs`\n\n, its score is unavailable. Do not treat a fallback value as the model's actual score.\n\n``` python\ndef proxy_func(self, data_record: str):\n    response = llm.invoke(self.task.format(data_record))\n    first = response.response_metadata[\"logprobs\"][\"content\"][0]\n    return first[\"token\"], first[\"logprob\"]\n```\n\nFor binary classification, request enough `top_logprobs`\n\nentries to include both labels, normalize the two label probabilities, and return the probability of the selected label:\n\n``` python\nimport math\n\ndef proxy_func(self, data_record: str):\n    response = llm.invoke(self.task.format(data_record))\n    first = response.response_metadata[\"logprobs\"][\"content\"][0]\n    candidates = {\n        item[\"token\"]: math.exp(item[\"logprob\"])\n        for item in first[\"top_logprobs\"]\n    }\n    true_prob = candidates.get(\"True\", 0.0)\n    false_prob = candidates.get(\"False\", 0.0)\n    total = true_prob + false_prob\n    if not total:\n        return False, 0.0\n    true_prob /= total\n    false_prob /= total\n    output = true_prob > false_prob\n    return output, true_prob if output else false_prob\n```\n\n`temperature=0`\n\nfor more repeatable answers. It does not guarantee identical responses, logprobs are not calibrated probabilities of correctness, and some reasoning models disallow temperature.`response_metadata`\n\nhas no `logprobs`\n\n, the provider did not return them. `logprobs`\n\nand `top_logprobs`\n\nare direct `ChatOpenAI`\n\narguments; other provider-specific parameters go in `extra_body`\n\n.", "url": "https://wpnews.pro/news/model-cascade-making-llm-classification-cheaper", "canonical_source": "https://dev.to/boris9027/model-cascade-making-llm-classification-cheaper-2kii", "published_at": "2026-08-23 20:25:42+00:00", "updated_at": "2026-08-23 20:43:47.812363+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["BARGAIN", "Task Cascades", "FrugalGPT", "SUPG", "OpenAI", "ChatOpenAI"], "alternates": {"html": "https://wpnews.pro/news/model-cascade-making-llm-classification-cheaper", "markdown": "https://wpnews.pro/news/model-cascade-making-llm-classification-cheaper.md", "text": "https://wpnews.pro/news/model-cascade-making-llm-classification-cheaper.txt", "jsonld": "https://wpnews.pro/news/model-cascade-making-llm-classification-cheaper.jsonld"}}