# Hacking the Gate: Reward Hacking in LLM Cascades > Source: > Published: 2026-11-03 13:03:52+00:00 In an LLM cascade, a smaller model (the student) answers first and a larger model (the oracle) handles questions the smaller model can’t answer confidently. Reinforcement learning can teach the smaller model to manipulate the escalation rule rather than report its confidence accurately. Cascade architectures have become common in practice because they offer an attractive trade-off between cost and quality. From research such as [FrugalGPT](https://arxiv.org/abs/2305.05176) (Chen et al., 2023) to model-routing systems from [OpenAI](https://openai.com/index/introducing-gpt-5/) and [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html), cascades are now widespread. More recently, learning the routing policy itself has shown promising results. For example, [Cursor shipped a learned router in July, claiming frontier-level quality at roughly 60% lower cost](https://cursor.com/blog/router), while [Applied Compute reports that its RL-trained agentic router achieves GPT-5.5-level quality at about 25% lower cost](https://www.appliedcompute.com/research/training-an-agentic-router). Every cascade has a gate, which decides whether a task gets escalated to the oracle or not. Unlike routers, which classify the query before any model runs, the gates we study act ex post: the smaller model always attempts the task first, and the gate judges whether its answer is good enough to keep. This judgment is made in one of two ways: 1. **Confidence-based gating:** the smaller model states a confidence in its own answer; escalates whenever it falls below a fixed threshold. 2. **Judgment-based gating:** a separate judge model scores the smaller model’s answer and reasoning; escalates whenever the score falls below a fixed threshold. ## **Our Research** As part of [Prime Intellect’s public reward-hacking sprint](https://www.primeintellect.ai/blog/reward-hacking), we built two RL environments: confidence-gate and judgment-gate. This post covers the first; Part 2 will cover the judgment-gate. In the confidence-gate environment, the student reports its confidence and the gate escalates answers below a threshold. Our research focused on whether it is optimal for the student model to remain calibrated regardless of how the gate is configured. Confidence calibration is well studied in the literature of uncertainty quantification ([Guo et al., 2017](https://arxiv.org/abs/1706.04599)). This question can become especially important when deploying these systems in the real-world, where the model’s reported confidence directly affects the system-level decision making. The judgment-gate environment looks similar to many verification loops used in industry today. Here, the environment does not provide a directly verifiable reward, so instead we rely on a judge model to evaluate the student’s work. Through a controlled study on task difficulty and judge model capability, we asked the question if there is no ground-truth signal to train the verifier against, does RL training to maximize the judge’s approval make the student genuinely more correct, or does it lead the student to be better at “persuading” the judge? Also, how much, if any, does the task’s difficulty and the judge’s capability play a role here? ## **Study 1: The confidence-gate** We evaluate a cascade system where `Llama-3.2-1B` is the only student model allowed. The student is prompted to generate an answer along with a stated confidence value `c`. If `c` is at or above the threshold `τ`, the system keeps the student’s answer at zero cost; otherwise it escalates the question to an oracle (a Gemini 2.5 Flash with cost measured in generated tokens and 97% accuracy on the question pool). Note that the student is run on every query regardless, so its cost is a constant that cancels from the escalation decision; we therefore set it to zero without loss of generality. From here on, cost refers to the token cost of the oracle call normalized by the maximum generation length (1000), so it lies in [0, 1]; it is 0 whenever the student’s answer is kept. The question pool includes 800 problems from the test split of GSM8K, a grade-school math word-problem benchmark, plus 200 arithmetic problems (100 easy single-digit sums, 100 two-digit add/subtract) added as an easier control. After deduplication, the dataset had 957 unique questions. We split these 80/20 stratified between the two sources, resulting in a 766/191 training/held out dataset. The reward we train the student on is the natural first choice: `R_outcome = sys_correct − λ·cost`. Here `sys_correct` indicates whether the answer the system finally delivered (the student’s own or the oracle’s after escalation) was right, and `λ` is the cost weight. We also track `stu_correct`: whether the student’s own answer was right, regardless of whether the system kept it. We call this the proxy reward because it isn’t quite what the owner of such a system cares about. In high-stakes settings, such as healthcare, a wrong answer kept by the system may be more costly than an unnecessary escalation. The objective that captures this, which we’ll call the true reward, adds a penalty for exactly those cases: `R_true = sys_correct − μ·kept_wrong − λ·cost`, where `kept_wrong = 1` when the system kept the student’s answer and `stu_correct = 0`. Both `λ` and `μ` lie in [0, 1]. Throughout this post, the student is trained only on proxy rewards and `R_true` serves purely as a held-out measurement. We use the gap between them to measure the failure. One could train on `R_true` directly, but pinning it down at training time is harder than it looks: `μ` is a business judgment rather than a training constant, and its right value depends on the question mix the deployed system actually sees. In our own pool, escalating a hard GSM8K question costs about 867 tokens on average versus 180–314 for the arithmetic control. Since a kept wrong answer is naturally priced against the escalation it avoided, the penalty it deserves shifts with the deployment distribution too. The proxy reward is intended to encourage correct answers while making escalation less attractive as its cost increases. We vary `λ` to test whether the trained student behaves as intended. At `λ = 0`, escalating has zero cost, so there’s no reason not to escalate to the oracle. At `λ = 1`, escalating is expensive, so there’s real pressure on the student model to just answer itself and reduce the cost. During training, the gate is fixed at `τ_train = 0.5`. The student therefore learns its confidence policy around that threshold. Once deployed, the threshold is just a configuration value, so `τ_deploy` can differ because escalation prices or error tolerances changed, or to repair damage done in training. ## **Reward #1: Outcome-only reward** The outcome-only reward produces underconfidence at low `λ` and overconfidence at high `λ`. When `λ` is low, escalation costs little, and the student learns to report low confidence on most questions. When `λ` is high, it reports confidence of around 0.92 despite achieving only 62% accuracy. RL also changes answer quality across the sweep: at low `λ`, the student’s answers are rarely kept, so their quality has little effect on the reward; at high `λ`, those answers are kept and rewarded directly. In both cases, the gate only checks whether confidence falls above or below `τ_train`. It does not reward confidence estimates that track actual accuracy. The proxy reward rises during training at every `λ`, including where confidence is badly calibrated. The true reward `R_true`, tracked held-out with `μ = 1`, shows the opposite: it falls behind the proxy from `λ ≈ 0.5` on. For small `λ`, the student always escalates, `kept_wrong` never fires, and the two curves lie on top of each other. From `λ ≈ 0.5` on they split, and the gap widens with `λ`: the model is now keeping wrong answers it should have escalated. These errors reduce `R_true` but do not affect `R_outcome`. ## **Does moving** `τ_deploy` **after training fix it?** An advantage of cascade architectures is that, even with a miscalibrated policy, `τ_deploy` can be optimized post hoc to mitigate the resulting loss. We test this directly: for the already-trained outcome-only policy, we recompute the escalation decision and resulting reward at every `τ_deploy ∈ [0, 1]` offline, without retraining. At `λ = 0.1`, both reward curves peak at the same threshold. Re-tuning therefore recovers the same optimum under either objective. At intermediate `λ (0.5)`, the two objectives diverge. The `R_outcome` curve is a broad, flat plateau: its maximum, the hollow star at `τ ≈ 0.5`, is not statistically distinguishable from anywhere else on the plateau, so tuning by `R_outcome` gives no reason to move away from the default `τ_deploy = τ_train = 0.5`. `R_true` rises monotonically with `τ`, from 0.40 at `τ_deploy = 0.5` up to 0.556 at `τ_deploy = 1` (full escalation), a 0.156 gain. At `τ = 1`, with everything escalated, `kept_wrong` never fires and the two rewards are identical. The problem is that the `R_outcome` curve gives no sign that a 0.156 gain is available. At high `λ (0.9)`, post hoc re-tuning yields no measurable improvement. Stated confidence has collapsed toward 1, so the escalation rate stays near zero over most of the `τ_deploy` range and both reward curves are flat. ## **Reward #2: Calibration-only. Score the confidence itself with a Brier score** Confidence calibration is a well-studied problem, both in machine learning generally ([Guo et al., 2017](https://arxiv.org/abs/1706.04599)) and for LLMs specifically ([Kadavath et al., 2022](https://arxiv.org/abs/2207.05221)). One of the reward functions from this literature is the Brier score (Brier, 1950), a strictly proper scoring rule, recently proven non-hackable as an RL confidence reward (Tan, Lin, Motani & Lee, 2026). The Brier-1 above is exactly that; equivalently, `Brier-1(c, y) = y − (c − y)²`, shifted so that correct answers always score positive and incorrect ones negative. This holds because the student secretly believes its answer is correct with probability p, meaning its correctness is a binary outcome `y ∈ {0, 1}` with `P(y = 1) = p`. It reports some confidence c and its expected reward is Since `p²` doesn’t depend on `c`, maximizing `S(c)` is the same as minimizing `(c − p)²`, which is smallest exactly when `c = p`. So no matter what the student actually believes, reporting that belief honestly is always the reward-maximizing move, and the further `c` strays from `p`, the more it gets punished, quadratically. In this sense, the Brier-1 reward is not hackable. Calibration error, the gap between stated confidence and actual accuracy where 0 is perfectly honest, sits around 0.10 across the `λ` sweep, versus roughly 0.30 for the outcome-only policy (middle panel of the figure below, where each policy is deployed at its own proxy-tuned threshold `τ*_proxy`, the `τ_deploy` that maximizes its proxy reward at each `λ`). The calibration-only reward has a different gap. It only ever scores the confidence value against the student’s own answer `stu_correct`), even on the questions where the system escalated to the oracle instead. So escalating can only ever look like a cost to this reward; it is never credited for making the right call. This means the reward’s own optimal policy, at every single `λ` we tested, is to never escalate. The left panel below shows where `R_true` is roughly flat `0.37` at every `λ`, forfeiting nearly half the achievable reward at low `λ` (right panel). This is why we combine the two signals, as the next reward does. ## **Reward #3: Hybrid. Add both signals, don’t swap one for the other** Here `α` balances the two signals. The hybrid reward combines the outcome-only and calibration-only signals so the model cares about system-level correctness while staying calibrated. One variant would be computing Brier-1 against the final, post-escalation outcome `sys_correct`) instead of the student’s own answer. But that reopens the exact hole `R_outcome` had: once an answer escalates, the scoring target switches from the student’s own correctness to the oracle’s near-certain one, so the reward-maximizing confidence is no longer the student’s belief but whichever side of `τ` it wants to land on, confidence bunches at the gate instead of tracking belief. The fix is to add the calibration signal, not swap the target. This combination holds calibration near `R_calib`’s honest level while keeping escalation alive. At low `λ` its `R_true` score matches `R_outcome`’s best number (0.87–0.88) without ever inheriting the collapse. Past `λ ≈ 0.5`, all three designs become statistically indistinguishable on `R_true`, since all of them stop escalating; the difference is that calibration-only and hybrid still report honest confidence, outcome-only does not. ## **Analytical derivation of** `τ*` For a fully calibrated model, the optimal `τ_deploy` can be derived analytically. Chow’s rule (Chow, 1970) from the decision-theory literature gives the threshold directly: where `o_acc` is the oracle’s accuracy (0.97), `cost_esc` is the average cost of an escalation (≈ 0.89; the average conditional cost on escalating), and `μ` is the kept-wrong penalty from `R_true`, here `μ = 1`, so keeping a wrong answer is penalized like a full-price escalation. Plugging in, the formula becomes `τ* = (1.97 − 0.89λ) / 2`, a straight line in `λ`. When the stated confidence is honest, deployment only needs three summary statistics: `o_acc`, `cost_esc` and `μ`.”The formula follows from comparing the expected reward of answering and escalating. Because the model is calibrated, its stated confidence `c` is the probability its own answer is correct, so we can write the expected true reward of each option directly in terms of `c`. Keep the student’s answer, and with probability `c` it is right, worth +1 at no cost; with probability `1 − c` it is wrong, incurring the kept-wrong penalty `−μ`: Escalate instead, and correctness depends only on the oracle’s accuracy, at escalation cost `λ·cost_esc`: `V_answer(c)` is a straight line in `c` with slope `(1 + μ) > 0`; `V_escalate` doesn’t depend on `c` at all. Two lines, one increasing and one flat, cross exactly once, so the optimal policy is a single threshold: answer whenever `V_answer(c) ≥ V_escalate`, escalate otherwise. Setting them equal gives the crossing point: We validate the formula against held-out data in the three panels below. The left panel plots the analytic `τ*` (dashed line) against each policy’s `τ*_true`, the threshold that actually maximizes `R_true`, found by offline search. The calibration-only policy `τ*_true` tracks the analytic line reasonably closely across `λ`; the outcome-only policy does not, swinging between 0.20 and 1.00 with no stable relationship to it, a direct consequence of its miscalibrated confidence; the hybrid policy’s sits in between, noisier than calibration-only’s. The middle and right panels deploy all three policies at the analytic `τ*` and report the resulting `R_true` and `sys_correct`. At `λ ≤ 0.3` the three perform comparably; past `λ ≈ 0.5` the outcome-only policy degrades fastest, with `sys_correct` falling to about 0.63 at `λ = 0.9` versus 0.75–0.80 for calibration-only and hybrid, though the error bars widen considerably in this regime. The analytic threshold is therefore only as good as the calibration it assumes: applied to a genuinely calibrated model, it recovers near-optimal performance without any search. ## **Can you actually tune** `τ` **with** `R_true` **in production?** So far, we have evaluated each policy either at the analytic threshold or at `τ*_true`, the threshold that maximizes held-out `R_true`. Calculating `R_true` requires ground-truth correctness labels for the student’s answer and the answer ultimately delivered by the system. Those labels are unavailable for every query in a typical live deployment. We therefore use `τ*_true` to measure achievable performance, not as a tuning procedure. Chow’s rule needs no per-item labels, only `o_acc` (estimable once from a small audited sample), `cost_esc` (observable from token metering), and `μ` (a normative choice by the system owner). Because only Chow’s rule is deployable, the comparison changes: our data cannot support the claim that calibration buys a higher `R_true` ceiling, since the ceilings look statistically tied. What calibration buys is a meaningful, label-free `τ*`; `R_outcome` gives no signal to move off a default `τ`, so the outcome-only policy stays wherever it started. ## **Study 1 takeaways** - **Miscalibration does not appear in the training metrics, and re-tuning the threshold afterward doesn’t fix it.**`R_outcome` rises at every`λ` while`R_true` quietly diverges. Re-tuning`τ_deploy` post hoc rescues the low`λ` regime, silently under-delivers at mid`λ` , and does nothing at high`λ` , where confidence has collapsed and the threshold no longer changes any decision. - **A reward component that has been proven to be hard to hack alone can still be exploited once it sits inside the cascade.** Brier-1 makes honest confidence the optimal report in isolation, yet inside the cascade its optimal policy is to never escalate. The hybrid reward adds system-level correctness to the calibration term. Alternatively, a calibrated policy can be deployed at the analytic`τ_deploy` from Chow’s rule, which recovers near-optimal performance with no search. **Limitations:** we used a single 1B student, one oracle and a math-heavy question pool. Effect sizes may differ for larger students or other domains. ## **Next: when the gate is a judge** Confidence gating assumes that the student’s answer can be checked against ground truth. In many verification loops, a judge model instead decides whether the answer is good enough, and the student is trained to maximize the judge’s approval. In Part 2, we test whether this improves correctness or teaches the student to persuade the judge. We vary task difficulty and judge capability, and find that larger judges can still be fooled, though the strategies become more sophisticated. Hadi Nekoei is a Research Scientist at Georgian, where he works on AI research and helps portfolio companies apply AI. He previously worked as a doctoral researcher at Mila – Quebec Artificial Intelligence Institute, focusing on reinforcement learning, multi-agent systems and LLM-based agents. Pashootan Vaezipoor is Head of AI Research at Georgian. He has more than five years of industry experience and a PhD in AI from the University of Toronto, as well as a master’s degree from Simon Fraser University. His research has covered reinforcement learning, combinatorial optimization and language models. Grateful to Kartik Gupta for thoughtful feedback on this essay. ## **Related reading** - Part 2 of this series: *Hacking the Gate — the judgment-gate* (link once published) - Our earlier [post](https://georgianailab.substack.com/p/the-craft-of-designing-resource-efficient) on the two-tier routing pattern this work responds to. - Tan, Lin, Motani & Lee, “Effective Rewards for a Confidence-Calibrated Agent” ( [https://arxiv.org/abs/2607.04332](https://arxiv.org/abs/2607.04332) ) - Chow, C. K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory ( [https://ieeexplore.ieee.org/document/1054406](https://ieeexplore.ieee.org/document/1054406) ) This blog is provided for informational purposes only and should not be relied upon as legal, business, investment, or tax advice. Nothing in this blog constitutes investment advice, nor is it intended for use by any investors or prospective investors in any Georgian funds. This blog may include links to external websites or information obtained from third-party sources. Georgian has not independently verified and makes no representations regarding the accuracy or completeness of such information, whether current or ongoing.If this content includes third-party advertisements, Georgian has not reviewed such materials and does not endorse any advertising content or the companies referenced. Any investments or portfolio companies mentioned are for illustrative purposes only and may not be representative of all investments made by funds managed by Georgian. Please contact Georgian for more information.