Model Cascade: making LLM classification cheaper

Many LLM workloads are classification tasks. This can get expensive, and I believe it is going to become more and more important, especially with the proliferation of software factories.

So what is Model Cascade? In short, it is a way to make a deterministic system around a cheap model and make it give us the same results as the expensive model.

The LLM we use gives us the probability of every token in the output, same probability model used to generate the response. We put all the tokens of the response together, and we get the probability of the response.

Now the smart part of the Model Cascade:

flowchart TB
    subgraph CAL["Calibrate once, offline"]
        S["Sample ~500 records"] --> O1["Label sample with oracle"]
        O1 --> T["Try every observed confidence <br/> value as a threshold"]
        T --> P["Pick cheapest threshold that<br/>meets the accuracy target"]
        O1 --> G["Check that proxy confidence agrees <br/> with oracle labels"]
    end

    subgraph ROUTE["Route every record, at scale"]
        R["Record"] --> PX["Proxy: small, cheap model"]
        PX --> L["Label + confidence score,<br/>from logprob"]
        L --> D{"Confidence above threshold?"}
        D -->|"yes, most records"| K["Keep proxy label"]
        D -->|"no, few records"| O2["Oracle: large, expensive model"]
        K --> OUT["Final labels"]
        O2 --> OUT
    end

    P -. "sets threshold" .-> D

Below is a summary of the BARGAIN paper I used to learn about this principle. It is more detailed than the first part, so if you want to learn more, read on.

Or read the full paper here: https://github.com/ucbepic/BARGAIN

Across eight datasets, the BARGAIN paper reports up to 86% more cost reduction than competing methods.

The follow-up Task Cascades paper adds three optimizations: rewriting prompts into simpler surrogate questions, reading only the most relevant document chunks, and searching over candidate cascades for the cheapest sequence. These cut costs a further 48.5% on average.

Unlike FrugalGPT, BARGAIN gives statistical guarantees. Unlike SUPG, they hold at any sample size, and it uses adaptive sampling and better estimation.

pip install bargain

Dependencies are numpy, pandas, tqdm, and openai. You can swap providers by defining your own proxy and oracle.

Examples live in examples/. Run the Supreme Court one from that directory; it loads

court_opinion.csv

by relative path.The Supreme Court numbers come from one run and may change with model versions, API behavior, or dataset changes.

BARGAIN_A

on a sample with your target and delta to see what fraction the proxy can handle.Pass logprobs

and top_logprobs

to ChatOpenAI

, then read the scores from response_metadata

:

import math
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-5-nano",
    temperature=0,
    logprobs=True,
    top_logprobs=5,
)

response = llm.invoke(
    "Does the text 'zebra' mention an animal? Answer with only True or False."
)

content = response.response_metadata["logprobs"]["content"]
first_token = content[0]
print(first_token["token"], first_token["logprob"])        # e.g. "True" -0.01
print(math.exp(first_token["logprob"]))                     # probability, e.g. 0.99

Each entry in content

is one token with its own logprob. The snippet reads only the first token, which works because the prompt forces a single-word answer. For a multi-token answer, sum all token logprobs instead:

total_logprob = sum(t["logprob"] for t in content)

For classification, prompt for a single word so the response is one token, then use that token's logprob as the confidence score.

The top token is the model's answer. For a label it did not pick, look inside top_logprobs

:

candidates = {c["token"]: c["logprob"] for c in first_token["top_logprobs"]}
score_for_true = candidates.get("True")

If a label is absent from top_logprobs

, its score is unavailable. Do not treat a fallback value as the model's actual score.

def proxy_func(self, data_record: str):
    response = llm.invoke(self.task.format(data_record))
    first = response.response_metadata["logprobs"]["content"][0]
    return first["token"], first["logprob"]

For binary classification, request enough top_logprobs

entries to include both labels, normalize the two label probabilities, and return the probability of the selected label:

import math

def proxy_func(self, data_record: str):
    response = llm.invoke(self.task.format(data_record))
    first = response.response_metadata["logprobs"]["content"][0]
    candidates = {
        item["token"]: math.exp(item["logprob"])
        for item in first["top_logprobs"]
    }
    true_prob = candidates.get("True", 0.0)
    false_prob = candidates.get("False", 0.0)
    total = true_prob + false_prob
    if not total:
        return False, 0.0
    true_prob /= total
    false_prob /= total
    output = true_prob > false_prob
    return output, true_prob if output else false_prob

temperature=0

for more repeatable answers. It does not guarantee identical responses, logprobs are not calibrated probabilities of correctness, and some reasoning models disallow temperature.response_metadata

has no logprobs

, the provider did not return them. logprobs

and top_logprobs

are direct ChatOpenAI

arguments; other provider-specific parameters go in extra_body

.