Most work on routing between language models answers one question: which model should handle this request? A classifier scores the request, and if it's an easy one directs it to a cheap model while a hard one goes to a frontier model. For a chatbot answering one-off questions independently, that is the whole characterization of the problem.
For a coding agent it is only half of it. A coding session is a long conversation in which the agent resends everything said so far with every request, and LLM providers cache that history so they need not reprocess it each time. A router that picks the right model for every turn, but ignores the cache, throws it away at each change of model and pays to rebuild it. The second half of the problem is therefore when a running session should change model. That is an online problem, decided turn by turn without knowing the future, between a one-off cost and a stream of small savings, and it has a classic form: the ski-rental problem.
Edgee's Agent Gateway treats routing as both halves. A tier map assigns a model to each level of difficulty, and a switching policy based on ski rental decides when a session actually moves. Self-Learning Routing learns both from an organization's own traffic. The gateway labels every turn as it serves it. Once a week the control plane replays the past 30 days of sessions through a copy of the gateway's algorithm, under every candidate map and several switching settings, and measures each against the schedule that hindsight would have chosen. It recommends the cheapest combination that keeps every turn on a capable model, and applies it when a series of guardrails allow.
Why difficulty is only half the story #
Picking a model once per session is easy, and wasteful: a session that opens with a hard design question often spends the next hour on edits a far cheaper model could make. The savings come from moving mid-task. But every move has a price.
On Anthropic, caching a prefix for five minutes costs 1.25 times the normal input price, and reading it back costs a tenth (prompt caching). A session that changes model loses its warm cache. The new model reads the whole conversation cold and writes it to its own cache. For a long session, that one rewrite can cost more than many turns of the price difference between the two models.
So the two halves interact. A tier map that puts a cheap model on easy turns only saves money if sessions actually move down to it, and stay long enough to recoup the rewrite. Switch on every turn that looks easy and the cache rewrites eat the savings. Never switch and the cheap model is never used. Which map is best therefore depends on the switching policy, and the switching policy that is best depends on the map. Self-Learning Routing learns them together for that reason.
Half one: difficulty tiers #
At the start of each user turn the gateway asks a classifier one question about the latest message:
how much capability does this task need? The answer comes on a four-point scale, from a trivial edit
to the hardest architectural problem, as a probability-weighted level index from 0 to 3. The gateway
normalizes it to a score in [0, 1] and maps the score to a tier:
| Score | Tier | What it looks like |
|---|---|---|
| ≤ 0.2 | low | A typo, a rename, formatting |
| ≤ 0.5 | medium | A small function, a straightforward bug |
| ≤ 0.8 | high | A change across several files, a subtle bug |
| > 0.8 | max | Architecture, security-critical or open-ended work |
The requests inside a turn, as the agent calls tools and reads their output, never decide afresh: they stay on the model the turn began with.
Moving up a tier is a constraint, not a bet. A turn that needs a more capable model gets one, at once and whatever the switching cost: quality is not traded for money. The only filter is the classifier's confidence, so that a single noisy label cannot drag a session upmarket:
| Confidence in the upgrade | Decision |
|---|---|
| ≥ 0.75 | Upgrade now |
| 0.5 to 0.75 | Upgrade once two consecutive turns ask for it |
| < 0.5 | Stay on the current tier |
Upgrades sit outside the cost accounting that follows. Everything else, moving to a cheaper model or to a different model for the same tier, is a cost decision, and that is the second half.
Half two: switching as a ski-rental problem #
Online algorithms
An algorithm is online when it must act on each part of its input as it arrives, without seeing what comes next. Its offline counterpart is handed the whole input in advance. Caching is the classic example: a full cache must evict an entry now, and only later requests reveal which entry was not needed again.
Online algorithms are judged by competitive analysis (Sleator and Tarjan, Amortized efficiency of
list update and paging rules, Communications of the ACM, 1985). An online algorithm is
c-competitive if, on every input, it pays at most c times what the best offline algorithm pays on
that same input. The input may be chosen by an adversary who knows the algorithm, so the ratio is a
promise about the worst case, not the average one.
Routing a coding session is an online problem: the gateway must pick a model for this turn before the developer has written the next one. Half one barely notices, since a turn's tier depends only on that turn. Switching does. Whether a move pays depends on the past, which decides where the warm cache is, and on the future, which decides how long the cheaper model will stay the right one. The gateway knows only the past.
Renting and buying
The simplest online problem with that shape is ski rental. A skier can rent skis for a fee each day
or buy a pair for B, without knowing how many days the season will last. Renting through a long
season costs far more than the skis; buying on the first day wastes the money if the season ends the
next morning. The problem was first analyzed in this form for caches too: multiprocessor caches
deciding whether to keep paying to update a shared block or to drop it (Karlin, Manasse, Rudolph and
Sleator, Competitive snoopy caching, Algorithmica, 1988).
Why break-even is 2-competitive
Let S be what renting would cost over the whole season. The offline optimum knows S, so it pays
min(S, B): it rents through a short season and buys on the first day of a long one. The break-even
rule rents until the rent paid reaches B, then buys. (Rent is treated as continuous, a fair
approximation when a day's rent is small next to the price of skis, as a turn's saving is next to a
cache rewrite.)
- If the season ends before the rent paid reaches
B, the rule has rented throughout, as the optimum did, and paysS. - Otherwise it pays
Bin rent, thenBfor the skis:2B, against the optimum'sB.
Either way it pays at most twice the optimum. Its worst case is the one an adversary would pick: the season ends the moment the skis are bought.
The same adversary shows that no deterministic rule does better. Any such rule amounts to buying once
the rent paid reaches x·B, for some x. End the season just after the purchase: the rule pays
(1 + x)·B and the optimum min(x, 1)·B, a ratio of
1 + max(x, 1/x)
That is also the worst any season can do to the rule, so it is the rule's exact competitive ratio. It
is smallest at x = 1, where it equals 2. Buying early loses to a short season, buying late loses to
a long one, and break-even balances the two regrets. Randomizing the moment of purchase lowers the
bound to e/(e−1) ≈ 1.58 (Karlin, Manasse, McGeoch and Owicki, Competitive randomized algorithms
for nonuniform problems, Algorithmica, 1994). The gateway stays deterministic: the same session
always gets the same decisions, which is what lets the replay re-run them.
Our problem as ski rental
When a cheaper model becomes the right one for a session's turns, the gateway faces the skier's choice:
| Ski rental | A coding session |
|---|---|
| A day | A user turn |
| Renting | Staying on the current model, and paying each turn the difference with what the cheaper model would cost if its cache were warm too |
| Buying | Switching, and paying once for the cheaper model to read the whole conversation cold and cache it |
| The season | The run of turns for which the cheaper model remains the right one |
| The adversary | The developer, whose next request nobody can predict |
The fit is close but not exact, and each difference shaped the design:
- The price of skis rises. A switch rewrites the whole conversation, which usually grows with every turn, so the longer the gateway waits, the dearer buying gets. The rent moves too, with the length of the answers. The gateway re-prices both on every turn. The proof above assumes fixed prices, so in a real session it is a guide rather than a theorem; the replay measures what the rule actually leaves on the table.
- Seasons repeat. A session holds many stretches, not one. When the candidate changes, because the turn's tier or the Strategy's map changed, a new instance begins, and the evidence gathered against the old candidate stops counting.
- The skis can be taken away. A hard turn forces an upgrade whatever it costs, which ends the stretch; coming back down means buying again. A threshold above break-even guards against exactly this.
- Sometimes the skis are free. When the cache has expired or the conversation has just been compacted, the current model must rewrite the conversation anyway, so switching costs nothing extra.
The cost model
For each turn the gateway estimates four token counts: P, the conversation prefix already warm on
the current model p; N, the tokens new this turn; O and R, the expected output and reasoning
tokens, taken from running averages of the session's own past turns. With a model's rates for cache
writes w, cache reads h, output o and reasoning ρ, and c the cheaper candidate:
cold(m) = (P + N)·w(m) + O·o(m) + R·ρ(m) # nothing cached: the whole prefix is written
warm(m) = P·h(m) + N·w(m) + O·o(m) + R·ρ(m) # the prefix is a cache hit
buy B = max(0, cold(c) − warm(p)) # one-off premium of switching this turn
rent r = max(0, warm(p) − warm(c)) # saving forgone per turn by staying
The break-even rule in the gateway
The gateway keeps a running total A of the rent paid against the current candidate, and buys when
it reaches a multiple of the buy price:
A ← A + r on every turn the candidate stays the same
switch when A ≥ λ·B (λ = 1.5 by default)
With λ = 1 this is the rule proved above. If a cold start on the candidate already costs less than
staying warm (B = 0), the switch happens at once.
The guarantee has an honest scope. It holds for one stretch, with a stable candidate, no upgrade and fixed prices, against that pair of models. It is not a bound against the best schedule for the whole session; that comparison is the replay's job, below.
Why λ = 1.5 rather than 1
By the argument above, a threshold λ has a worst-case ratio of 1 + max(λ, 1/λ): 2.5 at
λ = 1.5. The gateway accepts the weaker bound because its seasons can end in a way the textbook's
cannot. Upgrades fire regardless of cost, so a downgrade bought at exactly break-even can be undone
by a hard task on the very next turn, paying a second rewrite for nothing. Waiting until the rent
reaches one and a half times the buy price demands more evidence that the easy stretch is real. Which
threshold does best on average depends on how long an organization's easy stretches tend to last,
which no worst-case argument can tell; the replay measures it.
Forgetting without getting stuck
A router should forgive an isolated blip, so the running total decays. Decay it on every turn,
though, and a subtle bug appears. With a constant rent r and a decay factor d, the recurrence
A ← d·A + r converges to r / (1 − d). If the buy price exceeds that ceiling the total can never
cross the threshold, and the session overpays forever with no way out. The gateway therefore decays
the total (by half per turn) only once the session already sits on its target model. While evidence
is building against a stable candidate, the total only grows, so a long enough run of savings always
triggers the switch. When the candidate itself changes, a new instance of the problem starts and the
total resets to zero.
Free switches, and the order of checks
A switch is free when the cache has expired since the session's last request, or when the conversation has just shrunk below 60% of its previous length, as when Claude Code compacts it. The gateway assumes a cache lives five minutes, the shorter of Anthropic's two lifetimes, which can only make it treat a warm cache as cold early, never the reverse. After any switch, the gateway waits four turns before it will switch again to save money, which bounds how often a session changes model whatever the totals say; upgrades and free switches ignore the wait. In full, each user turn is decided in this order:
- The turn needs a more capable model than the current one: upgrade (subject to the confidence gate).
- The session is already on the tier's target model: decay the total and stay.
- The cache has expired or the conversation was compacted: switch, for free.
- The session switched fewer than four turns ago: stay.
- The candidate is on a lower tier and the Strategy forbids moving down to save money: stay.
- The candidate differs from last turn's: reset the total.
- Add this turn's rent; switch if the total reaches
λ·B.
A session's state is kept for six hours.
A session, replayed
The figure shows 28 turns of one real coding session from Edgee's own traffic, on OpenAI's GPT-6 Astra, run again through this rule, with the gateway's default settings, by the replay described below. Each column is a user turn. The top row is the classifier's label, the middle row the model the session actually used, and the bottom row the tier the gateway would have served.
The first two turns are harder, and the session moves up twice (▲▲): an upgrade is not a cost decision. The next turn is easy, but the gateway is waiting out the four turns that follow any switch. Then the agent compacts the conversation, from about 224,000 tokens to 25,000. The current model must rewrite what remains anyway, so moving down costs nothing extra (▼). Two more upgrades follow (▲▲). Once the wait is over, the session rents for a turn, then buys (▼): the saving forgone has reached one and a half times the cost of the rewrite. The next upgrade (▲) coincides with a second compaction, and this time the session buys as soon as the wait ends (▼). The conversation is short again, so the rewrite is cheap and a single turn's saving covers it: the price of skis rises with the conversation, and here it had just fallen. The last upgrade (▲) follows a 38-minute , and the session is waiting, then renting, when the window closes. The two dashed labels are the confidence gate at work: harder labels the gateway waits to see twice in a row before acting on.
What a Strategy can change
The values above are defaults. A smart-routing Strategy can change six of them, and the replay compares the alternatives, which makes them the second half of what Self-Learning Routing learns.
Four tune the rule itself: the threshold λ; whether a session may move down a tier to save money at
all (if not, it still moves down when its cache goes cold, and it always moves up); how many turns to
wait after a switch; and whether the confidence gate applies, or every upgrade label is trusted at
once.
Two relax assumptions the defaults make out of caution. The cache lifetime can follow what each request asks for (Anthropic's one hour, OpenAI's 24 hours, or an hour when the request names none) rather than a fixed five minutes. And the turns an agent writes for itself, such as a status line, a title or a suggested next prompt, or to carry on, such as a compaction handoff or a background-task notice, can keep the session's model and skip the classifier.
Every default reproduces how the gateway routed before these settings existed, so a Strategy that changes none routes exactly as it always has.
Labeling every turn, live #
The replay needs a label on every user turn: its difficulty and the confidence in it, the kind of work, and who wrote it. An organization switches this on as classification, once an admin has accepted a versioned consent notice.
What is labeled, and by whom
The gateway labels each new turn a user starts, while serving it. It sends the latest user message, cut to 4,000 characters, to the labeler the admin chose: Jev, the classifier live routing already uses, under zero data retention; or OpenAI Decisions, on Edgee's credential rather than the organization's. The labeler answers the same capability question as live routing, plus two more: what kind of work this is (with a finer category when the request is vague), and who wrote it, whether a person, a work order such as a skill or a slash command, or the agent itself.
Some requests are never sent:
- the requests an agent makes between tool calls within a turn, and requests without a session id;
- calls made by subagents, which the gateway spots by the marks agents leave on them: Claude Code's
billing attribution (
cc_is_subagent=true) and Codex'sx-openai-subagentandx-codex-parent-thread-idheaders; - Claude Code's checks on whether a tool call needs permission;
- turns the agent writes for itself, which are recognized by rule and labeled without a call.
When live routing has already scored the turn with the same classifier, the label reuses that score.
What is stored
Only the labels: how hard the turn was, how sure the labeler was, what kind of work it was and who wrote it. They are kept with the request's usage record, and the console uses them to show how each session was classified. The text itself is never stored or logged, and error messages never quote it.
No added latency
The labeling runs alongside the request, not in front of it. The response goes back to the agent as soon as the model has answered; only the usage record, written afterwards, waits for the label. A late label is recorded as missing rather than holding anything up, and when the labeler is busy a turn simply goes unlabeled, so labeling never slows down live routing.
The replay: learning both halves together #
Once a week, for organizations that have also switched on learning, the control plane reads the past 30 days of traffic, up to 2m requests, and replays it through a Go port of the gateway's algorithm: the same tiers, confidence gate, ski-rental accounting, free switches and cooldown.
Sessions and gaps
Requests are grouped into sessions and split by agent (Claude Code, Codex and the rest), since a map that suits one agent's prompts may not suit another's. Spending the replay cannot cover is reported, not extrapolated: traffic from before classification was switched on, sessions without labels, and requests without a session id. The share it does cover is the report's coverage.
Candidate maps
For each tier, the candidates are the models the organization may use whose catalog rating clears a deliberately cautious floor, one notch above the tier's bare minimum, and whose context window holds at least 200,000 tokens:
| Tier | Lowest catalog rating allowed |
|---|---|
| low | Strong |
| medium | Elite |
| high | Frontier |
| max | Frontier |
The ratings come from Edgee's model catalog, which places every model on four levels, from Basic to Frontier, according to its results on public benchmarks: chiefly SWE-bench Verified, which measures how often a model resolves real GitHub issues, and the Artificial Analysis Intelligence Index, which combines a range of evaluations that Artificial Analysis runs independently.
The replay keeps the three cheapest candidates per tier and tries every combination, at most 3⁴ = 81 maps per agent. Each map replays every session under the organization's current switching settings, with every request repriced at catalog prices on the model the replay would have chosen (uncached input, cache writes, cache reads and output counted separately). The recommended map is the cheapest, and never one that would have cost more than the organization actually paid.
The hindsight optimum
Competitive analysis judges an online rule against the offline optimum, which knows the future. A replay knows how every session ended, so it can compute that optimum. For each session and map it finds, by dynamic programming over the requests, the cheapest assignment of the map's models that serves every request at least at the tier the gateway required, counting a cold cache write whenever consecutive requests change model:
best[m] after request i = min over models m' of
best[m'] after request i−1 + cost(request i on m, warm if m' = m, cold otherwise)
restricted to models capable of request i's tier. Because it may change model at any request, even
inside a turn, which the gateway never does, no online policy can beat it. The gap between the cost of the online switching
policy and this optimum is the price of not knowing how long each stretch of a session will last. It
shows how much there is left to gain from switching better, as distinct from choosing better models.
Quality: never below the label
The replay does not guess how a different model would have answered. It has no ground truth for that, and does not pretend otherwise. Quality is enforced structurally instead. Every candidate clears its tier's floor, so a turn served at its labeled tier always lands on a capable model. What the replay measures is the exception: the below-label share, the fraction of routed requests served by a model below their label's tier, as happens while a low-confidence upgrade waits for its second turn. No comparison below trades money for a higher below-label share.
Choosing the switching settings
With the map chosen, the replay compares six switching settings on it: the defaults; a lower
threshold (λ = 0.5) and a higher one (λ = 5); never moving down a tier to save money; no
confidence gate; and a setting built on our own measurements (see below), which follows each
request's cache lifetime, does not wait between switches and keeps the model on turns the agent
writes for itself. The settings in force join the comparison when they match none of these.
The two thresholds put the theory to the test. Their worst-case ratios, 3 and 6, are both worse than the default's 2.5, yet either can win on real traffic, because the worst case is rare. An organization whose easy stretches tend to run long gains from buying early; one whose easy turns come singly, between hard ones, gains from buying late or not at all. Worst-case analysis cannot tell the two apart. A replay of the organization's own sessions can.
A new setting replaces the old one only if it saves at least half a point more of the billed spend without serving more than 0.2 points more requests below their label, which keeps a weekly run from flip-flopping between settings that differ only by noise.
Uncertainty
A replayed saving is an estimate drawn from a few hundred sessions of wildly different sizes, so the report gives a 95% interval, from a bootstrap over sessions (Efron, Bootstrap methods: another look at the jackknife, Annals of Statistics, 1979): resample sessions with replacement, recompute the saving on each resample, and read off the 2.5th and 97.5th percentiles. The seed is fixed, so the same data always yield the same report. Confidence is high when the replayed sessions cover at least 70% of spend and the interval is at most ten points wide, low below 40% coverage, and medium otherwise.
Prior art, and where this differs #
Learned LLM routing has so far focused on the first half. RouteLLM (Ong et al, 2024) learns to choose between a strong and a weak model from human preference data. FrugalGPT (Chen, Zaharia and Zou, 2023) cascades models of increasing cost until an answer is good enough. Hybrid LLM (Ding et al, ICLR 2024) routes between a small and a large model according to a query's predicted difficulty. Each routes queries independently, with no model of what it costs to change model in the middle of a conversation.
On the switching side, the closest prior art is the theory itself. Beyond the deterministic and randomized rules above, learning-augmented algorithms (Purohit, Svitkina and Kumar, Improving online algorithms via ML predictions, NeurIPS 2018) feed a prediction of the season's length into the ski-rental rule, to beat the worst-case bound when the prediction is good while keeping a bound when it is not. Self-Learning Routing does something coarser: it predicts nothing for a single session, but tunes the threshold to the stretches in an organization's past sessions.
Self-Learning Routing adds the second half. It treats the change of model inside a session as an online rent-or-buy decision with an explicit cost model and a worst-case guarantee, evaluates it against the hindsight optimum, and tunes the tier map and the switching policy jointly, because each changes the value of the other. It evaluates itself by replaying the organization's own sessions, in the spirit of replay-based offline evaluation (Li, Chu, Langford and Wang, WSDM 2011), rather than on a public benchmark. Its main assumption should be stated plainly: it reprices the tokens each request actually used, so it does not capture a different model writing longer or shorter answers.
From a recommendation to a running Strategy #
Each weekly run stores its report and records what became of it: kept (nothing would cost less, or the self-learned Strategy already runs the best map), proposed (awaiting an admin), applied, or skipped (a guardrail held an automatic change back). An admin can apply any proposed run from the console. With Apply automatically on, the run updates the self-learned Strategy itself, but only when every guardrail passes:
| Guardrail | Otherwise |
|---|---|
| An admin has already applied a self-learned Strategy and assigned it somewhere | proposed: the first application, and where it runs, stay an admin's choice |
| Nobody has edited the Strategy since it was last applied | proposed: an admin decides whether the new map replaces the edit |
| Confidence is medium or high | skipped |
| The lower end of the saving's 95% interval is above zero | skipped |
| It saves at least three points more than the map in force, on the same sessions | skipped |
| It serves no more than 0.2 points more requests below their label | skipped |
| Every model in the map served at least 30 of the organization's requests in the window | proposed: an admin decides whether to roll out a model the team has not used |
| No model in the map failed more often on the organization's traffic than the models agents asked for | skipped |
Every change records what came before. An admin can undo the latest one from the console: the previous map and switching settings return, and automatic apply turns off, so the next weekly run proposes a change rather than quietly reapplying it.
What we measured #
The defaults are the cautious choice. One measurement of Edgee's own coding-agent traffic shows why the replay also tries a less cautious setting, and why it tests that setting on every organization rather than assuming it: agents trigger many switches themselves. In our replays about half of all switches were set off by text the agent wrote for itself (status lines, titles, suggested prompts, task notices), which says nothing about how hard the work is.
We do not publish a saving from routing alone. Each organization's report shows its own replayed saving, with its interval and coverage, on its own traffic.
Limits, and what comes next #
Two things Self-Learning Routing does not do yet are worth stating plainly, because the next steps start from them:
- Repriced, not re-generated. The replay assumes a different model would have written answers of the same length.
- Training is a separate opt-in. An organization can consent to Edgee keeping the labeled text to train its own routing classifier; nothing is collected yet.
The next steps follow from these limits, and from where the theory stops.
Measuring answers, not only prices. Re-running a sample of turns on the candidate model would show how long its answers really are. The replay's main assumption would become a measurement.
Pricing a switch that gets dearer. The textbook prices skis once; a session's buy price rises with every turn, because a switch rewrites a longer conversation. Waiting therefore costs twice, in rent paid and in a dearer purchase, which argues for moving earlier in a long stretch than a fixed-price rule does. A rule that anticipates the rise, with a guarantee that covers it, is open work. The hindsight optimum already gives the yardstick to judge one.
Predicting how long a stretch will last. Today the threshold is tuned per organization, from the whole history of its sessions. The labels already say what kind of work each turn is and who wrote it, which may predict whether an easy stretch will be long or short. Learning-augmented ski rental shows how to use such a prediction safely: lean on it when it is right, and keep a worst-case bound when it is wrong. The replay can test any such predictor against the hindsight optimum before it touches live traffic.
A classifier trained for coding agents. Turns labeled with an organization's consent could train a routing classifier tuned to coding-agent traffic rather than a general-purpose one. Better labels improve both halves at once: the tier each turn gets, and the stretches the switching rule sees.
Conclusion #
Choosing a model by difficulty tells a router where a turn could go. It does not say whether moving there pays, because the model a coding session is on also holds its cache. That second question is an online one, and ski rental answers it with a simple rule and a worst-case guarantee: keep count of the savings forgone, and move once they match the cost of moving.
What theory cannot say is which models to use, or how long an organization's easy stretches really last. Both depend on the organization's own work. Self-Learning Routing learns them from its own sessions: it replays them under every candidate map and switching setting, measures each against what hindsight would have done, and changes the Strategy only when the evidence is clear. Difficulty picks the model, ski rental decides when to switch, and an organization's own history sets the numbers for both.
Self-Learning Routing belongs to Route, the first of the Agent Gateway's three pillars (Route, Compress, Observe), and its labels serve the third as well: they show a platform team how its work divides by difficulty and kind before it sets a budget or a Strategy.
The only way to know what this would do for your team is to run it on your team's sessions. Routing your coding agents through Edgee takes a CLI install, about five minutes, and no code changes:
curl -fsSL https://edgee.ai/install.sh | bash
edgee launch claude # or codex, opencode, cursor, copilot, codebuddy
On a Team or Enterprise plan, turn on classification in the console. From then on every turn is labeled as it is served, and the replay shows, on your own traffic, which model each tier should use, how often your sessions would switch, and what that would have cost, with its interval and coverage. Nothing changes until you apply it. If you run a fleet of coding agents and want to walk through that report with us, book a call. And if you have questions or ideas about the approach, we would be glad to hear them on Discord.