{"slug": "switching-llms-mid-conversation-is-a-ski-rental-problem", "title": "Switching LLMs mid-conversation is a ski rental problem", "summary": "Edgee's Agent Gateway treats LLM routing for coding agents as two coupled problems — picking a model per turn and deciding when a running session should switch — and models the second as a ski-rental problem, since every model change discards the provider's prompt cache. On Anthropic, caching a prefix for five minutes costs 1.25 times the normal input price and reading it back costs a tenth, so a long session's cache rewrite can exceed many turns of savings between the two models. The gateway uses a tier map assigning models to four difficulty levels (scores ≤0.2 low, ≤0.5 medium, ≤0.8 high, >0.8 max) plus a ski-rental-based switching policy, and its Self-Learning Routing replays the past 30 days of sessions weekly to recommend the cheapest map and switching setting that keeps every turn on a capable model.", "body_md": "# Switching LLMs mid-conversation is a ski-rental problem\n\nMost work on routing between language models answers one question: which model should handle this request? A classifier scores the request, and if it's an easy one directs it to a cheap model while a hard one goes to a frontier model. For a chatbot answering one-off questions independently, that is the whole characterization of the problem.\n\nFor a coding agent it is only half of it. A coding session is a long conversation in which the agent\nresends everything said so far with every request, and LLM providers cache that history so they need\nnot reprocess it each time. A router that picks the right model for every turn, but ignores the\ncache, throws it away at each change of model and pays to rebuild it. The second half of the problem\nis therefore *when* a running session should change model. That is an online problem, decided turn\nby turn without knowing the future, between a one-off cost and a stream of small savings, and it has\na classic form: the ski-rental problem.\n\nEdgee's Agent Gateway treats routing as both halves. A **tier map** assigns a model to each level of\ndifficulty, and a **switching policy** based on ski rental decides when a session actually moves.\nSelf-Learning Routing learns both from an organization's own traffic. The gateway labels every turn\nas it serves it. Once a week the control plane replays the past 30 days of sessions through a copy of\nthe gateway's algorithm, under every candidate map and several switching settings, and measures each\nagainst the schedule that hindsight would have chosen. It recommends the cheapest combination that\nkeeps every turn on a capable model, and applies it when a series of guardrails allow.\n\n## Why difficulty is only half the story\n\nPicking a model once per session is easy, and wasteful: a session that opens with a hard design question often spends the next hour on edits a far cheaper model could make. The savings come from moving mid-task. But every move has a price.\n\nOn Anthropic, caching a prefix for five minutes costs 1.25 times the normal input price, and reading\nit back costs a tenth ([prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)).\nA session that changes model loses its warm cache. The new model reads the whole conversation cold\nand writes it to its own cache. For a long session, that one rewrite can cost more than many turns of\nthe price difference between the two models.\n\nSo the two halves interact. A tier map that puts a cheap model on easy turns only saves money if sessions actually move down to it, and stay long enough to recoup the rewrite. Switch on every turn that looks easy and the cache rewrites eat the savings. Never switch and the cheap model is never used. Which map is best therefore depends on the switching policy, and the switching policy that is best depends on the map. Self-Learning Routing learns them together for that reason.\n\n## Half one: difficulty tiers\n\nAt the start of each user turn the gateway asks a classifier one question about the latest message:\nhow much capability does this task need? The answer comes on a four-point scale, from a trivial edit\nto the hardest architectural problem, as a probability-weighted level index from 0 to 3. The gateway\nnormalizes it to a score in `[0, 1]` and maps the score to a tier:\n\n| Score | Tier | What it looks like | \n|---|---|---|\n| ≤ 0.2 | low | A typo, a rename, formatting | \n| ≤ 0.5 | medium | A small function, a straightforward bug | \n| ≤ 0.8 | high | A change across several files, a subtle bug | \n| > 0.8 | max | Architecture, security-critical or open-ended work | \n\nThe requests inside a turn, as the agent calls tools and reads their output, never decide afresh: they stay on the model the turn began with.\n\nMoving up a tier is a constraint, not a bet. A turn that needs a more capable model gets one, at once and whatever the switching cost: quality is not traded for money. The only filter is the classifier's confidence, so that a single noisy label cannot drag a session upmarket:\n\n| Confidence in the upgrade | Decision | \n|---|---|\n| ≥ 0.75 | Upgrade now | \n| 0.5 to 0.75 | Upgrade once two consecutive turns ask for it | \n| < 0.5 | Stay on the current tier | \n\nUpgrades sit outside the cost accounting that follows. Everything else, moving to a cheaper model or to a different model for the same tier, is a cost decision, and that is the second half.\n\n## Half two: switching as a ski-rental problem\n\n### Online algorithms\n\nAn algorithm is *online* when it must act on each part of its input as it arrives, without seeing\nwhat comes next. Its *offline* counterpart is handed the whole input in advance. Caching is the\nclassic example: a full cache must evict an entry now, and only later requests reveal which entry\nwas not needed again.\n\nOnline algorithms are judged by *competitive analysis* (Sleator and Tarjan, *Amortized efficiency of\nlist update and paging rules*, Communications of the ACM, 1985). An online algorithm is\n*c-competitive* if, on every input, it pays at most `c` times what the best offline algorithm pays on\nthat same input. The input may be chosen by an adversary who knows the algorithm, so the ratio is a\npromise about the worst case, not the average one.\n\nRouting a coding session is an online problem: the gateway must pick a model for this turn before the developer has written the next one. Half one barely notices, since a turn's tier depends only on that turn. Switching does. Whether a move pays depends on the past, which decides where the warm cache is, and on the future, which decides how long the cheaper model will stay the right one. The gateway knows only the past.\n\n### Renting and buying\n\nThe simplest online problem with that shape is ski rental. A skier can rent skis for a fee each day\nor buy a pair for `B`, without knowing how many days the season will last. Renting through a long\nseason costs far more than the skis; buying on the first day wastes the money if the season ends the\nnext morning. The problem was first analyzed in this form for caches too: multiprocessor caches\ndeciding whether to keep paying to update a shared block or to drop it (Karlin, Manasse, Rudolph and\nSleator, *Competitive snoopy caching*, Algorithmica, 1988).\n\n### Why break-even is 2-competitive\n\nLet `S` be what renting would cost over the whole season. The offline optimum knows `S`, so it pays\n`min(S, B)`: it rents through a short season and buys on the first day of a long one. The break-even\nrule rents until the rent paid reaches `B`, then buys. (Rent is treated as continuous, a fair\napproximation when a day's rent is small next to the price of skis, as a turn's saving is next to a\ncache rewrite.)\n\n- If the season ends before the rent paid reaches `B` , the rule has rented throughout, as the optimum\ndid, and pays`S` .\n- Otherwise it pays `B` in rent, then`B` for the skis:`2B` , against the optimum's`B` .\n\nEither way it pays at most twice the optimum. Its worst case is the one an adversary would pick: the season ends the moment the skis are bought.\n\nThe same adversary shows that no deterministic rule does better. Any such rule amounts to buying once\nthe rent paid reaches `x·B`, for some `x`. End the season just after the purchase: the rule pays\n`(1 + x)·B` and the optimum `min(x, 1)·B`, a ratio of\n\n```\n1 + max(x, 1/x)\n```\n\nThat is also the worst any season can do to the rule, so it is the rule's exact competitive ratio. It\nis smallest at `x = 1`, where it equals 2. Buying early loses to a short season, buying late loses to\na long one, and break-even balances the two regrets. Randomizing the moment of purchase lowers the\nbound to `e/(e−1) ≈ 1.58` (Karlin, Manasse, McGeoch and Owicki, *Competitive randomized algorithms\nfor nonuniform problems*, Algorithmica, 1994). The gateway stays deterministic: the same session\nalways gets the same decisions, which is what lets the replay re-run them.\n\n### Our problem as ski rental\n\nWhen a cheaper model becomes the right one for a session's turns, the gateway faces the skier's choice:\n\n| Ski rental | A coding session | \n|---|---|\n| A day | A user turn | \n| Renting | Staying on the current model, and paying each turn the difference with what the cheaper model would cost if its cache were warm too | \n| Buying | Switching, and paying once for the cheaper model to read the whole conversation cold and cache it | \n| The season | The run of turns for which the cheaper model remains the right one | \n| The adversary | The developer, whose next request nobody can predict | \n\nThe fit is close but not exact, and each difference shaped the design:\n\n- **The price of skis rises.** A switch rewrites the whole conversation, which usually grows with\nevery turn, so the longer the gateway waits, the dearer buying gets. The rent moves too, with the\nlength of the answers. The gateway re-prices both on every turn. The proof above assumes fixed\nprices, so in a real session it is a guide rather than a theorem; the replay measures what the\nrule actually leaves on the table.\n- **Seasons repeat.** A session holds many stretches, not one. When the candidate changes, because\nthe turn's tier or the Strategy's map changed, a new instance begins, and the evidence gathered\nagainst the old candidate stops counting.\n- **The skis can be taken away.** A hard turn forces an upgrade whatever it costs, which ends the\nstretch; coming back down means buying again. A threshold above break-even guards against exactly\nthis.\n- **Sometimes the skis are free.** When the cache has expired or the conversation has just been\ncompacted, the current model must rewrite the conversation anyway, so switching costs nothing\nextra.\n\n### The cost model\n\nFor each turn the gateway estimates four token counts: `P`, the conversation prefix already warm on\nthe current model `p`; `N`, the tokens new this turn; `O` and `R`, the expected output and reasoning\ntokens, taken from running averages of the session's own past turns. With a model's rates for cache\nwrites `w`, cache reads `h`, output `o` and reasoning `ρ`, and `c` the cheaper candidate:\n\n```\ncold(m) = (P + N)·w(m) + O·o(m) + R·ρ(m)        # nothing cached: the whole prefix is written\nwarm(m) =  P·h(m) + N·w(m) + O·o(m) + R·ρ(m)    # the prefix is a cache hit\n\nbuy  B = max(0, cold(c) − warm(p))              # one-off premium of switching this turn\nrent r = max(0, warm(p) − warm(c))              # saving forgone per turn by staying\n```\n\n### The break-even rule in the gateway\n\nThe gateway keeps a running total `A` of the rent paid against the current candidate, and buys when\nit reaches a multiple of the buy price:\n\n```\nA ← A + r            on every turn the candidate stays the same\nswitch when A ≥ λ·B  (λ = 1.5 by default)\n```\n\nWith `λ = 1` this is the rule proved above. If a cold start on the candidate already costs less than\nstaying warm (`B = 0`), the switch happens at once.\n\nThe guarantee has an honest scope. It holds for one stretch, with a stable candidate, no upgrade and fixed prices, against that pair of models. It is not a bound against the best schedule for the whole session; that comparison is the replay's job, below.\n\n### Why λ = 1.5 rather than 1\n\nBy the argument above, a threshold `λ` has a worst-case ratio of `1 + max(λ, 1/λ)`: 2.5 at\n`λ = 1.5`. The gateway accepts the weaker bound because its seasons can end in a way the textbook's\ncannot. Upgrades fire regardless of cost, so a downgrade bought at exactly break-even can be undone\nby a hard task on the very next turn, paying a second rewrite for nothing. Waiting until the rent\nreaches one and a half times the buy price demands more evidence that the easy stretch is real. Which\nthreshold does best on average depends on how long an organization's easy stretches tend to last,\nwhich no worst-case argument can tell; the replay measures it.\n\n### Forgetting without getting stuck\n\nA router should forgive an isolated blip, so the running total decays. Decay it on every turn,\nthough, and a subtle bug appears. With a constant rent `r` and a decay factor `d`, the recurrence\n`A ← d·A + r` converges to `r / (1 − d)`. If the buy price exceeds that ceiling the total can never\ncross the threshold, and the session overpays forever with no way out. The gateway therefore decays\nthe total (by half per turn) only once the session already sits on its target model. While evidence\nis building against a stable candidate, the total only grows, so a long enough run of savings always\ntriggers the switch. When the candidate itself changes, a new instance of the problem starts and the\ntotal resets to zero.\n\n### Free switches, and the order of checks\n\nA switch is free when the cache has expired since the session's last request, or when the conversation has just shrunk below 60% of its previous length, as when Claude Code compacts it. The gateway assumes a cache lives five minutes, the shorter of Anthropic's two lifetimes, which can only make it treat a warm cache as cold early, never the reverse. After any switch, the gateway waits four turns before it will switch again to save money, which bounds how often a session changes model whatever the totals say; upgrades and free switches ignore the wait. In full, each user turn is decided in this order:\n\n1. The turn needs a more capable model than the current one: upgrade (subject to the confidence gate).\n2. The session is already on the tier's target model: decay the total and stay.\n3. The cache has expired or the conversation was compacted: switch, for free.\n4. The session switched fewer than four turns ago: stay.\n5. The candidate is on a lower tier and the Strategy forbids moving down to save money: stay.\n6. The candidate differs from last turn's: reset the total.\n7. Add this turn's rent; switch if the total reaches `λ·B` .\n\nA session's state is kept for six hours.\n\n### A session, replayed\n\nThe figure shows 28 turns of one real coding session from Edgee's own traffic, on OpenAI's GPT-6 Astra, run again through this rule, with the gateway's default settings, by the replay described below. Each column is a user turn. The top row is the classifier's label, the middle row the model the session actually used, and the bottom row the tier the gateway would have served.\n\nThe first two turns are harder, and the session moves up twice (▲▲): an upgrade is not a cost decision. The next turn is easy, but the gateway is waiting out the four turns that follow any switch. Then the agent compacts the conversation, from about 224,000 tokens to 25,000. The current model must rewrite what remains anyway, so moving down costs nothing extra (▼). Two more upgrades follow (▲▲). Once the wait is over, the session rents for a turn, then buys (▼): the saving forgone has reached one and a half times the cost of the rewrite. The next upgrade (▲) coincides with a second compaction, and this time the session buys as soon as the wait ends (▼). The conversation is short again, so the rewrite is cheap and a single turn's saving covers it: the price of skis rises with the conversation, and here it had just fallen. The last upgrade (▲) follows a 38-minute pause, and the session is waiting, then renting, when the window closes. The two dashed labels are the confidence gate at work: harder labels the gateway waits to see twice in a row before acting on.\n\n### What a Strategy can change\n\nThe values above are defaults. A smart-routing Strategy can change six of them, and the replay compares the alternatives, which makes them the second half of what Self-Learning Routing learns.\n\nFour tune the rule itself: the threshold `λ`; whether a session may move down a tier to save money at\nall (if not, it still moves down when its cache goes cold, and it always moves up); how many turns to\nwait after a switch; and whether the confidence gate applies, or every upgrade label is trusted at\nonce.\n\nTwo relax assumptions the defaults make out of caution. The cache lifetime can follow what each request asks for (Anthropic's one hour, OpenAI's 24 hours, or an hour when the request names none) rather than a fixed five minutes. And the turns an agent writes for itself, such as a status line, a title or a suggested next prompt, or to carry on, such as a compaction handoff or a background-task notice, can keep the session's model and skip the classifier.\n\nEvery default reproduces how the gateway routed before these settings existed, so a Strategy that changes none routes exactly as it always has.\n\n## Labeling every turn, live\n\nThe replay needs a label on every user turn: its difficulty and the confidence in it, the kind of\nwork, and who wrote it. An organization switches this on as **classification**, once an admin has\naccepted a versioned consent notice.\n\n### What is labeled, and by whom\n\nThe gateway labels each new turn a user starts, while serving it. It sends the latest user message, cut to 4,000 characters, to the labeler the admin chose: Jev, the classifier live routing already uses, under zero data retention; or OpenAI Decisions, on Edgee's credential rather than the organization's. The labeler answers the same capability question as live routing, plus two more: what kind of work this is (with a finer category when the request is vague), and who wrote it, whether a person, a work order such as a skill or a slash command, or the agent itself.\n\nSome requests are never sent:\n\n- the requests an agent makes between tool calls within a turn, and requests without a session id;\n- calls made by subagents, which the gateway spots by the marks agents leave on them: Claude Code's\nbilling attribution (`cc_is_subagent=true` ) and Codex's`x-openai-subagent` and`x-codex-parent-thread-id` headers;\n- Claude Code's checks on whether a tool call needs permission;\n- turns the agent writes for itself, which are recognized by rule and labeled without a call.\n\nWhen live routing has already scored the turn with the same classifier, the label reuses that score.\n\n### What is stored\n\nOnly the labels: how hard the turn was, how sure the labeler was, what kind of work it was and who wrote it. They are kept with the request's usage record, and the console uses them to show how each session was classified. The text itself is never stored or logged, and error messages never quote it.\n\n### No added latency\n\nThe labeling runs alongside the request, not in front of it. The response goes back to the agent as soon as the model has answered; only the usage record, written afterwards, waits for the label. A late label is recorded as missing rather than holding anything up, and when the labeler is busy a turn simply goes unlabeled, so labeling never slows down live routing.\n\n## The replay: learning both halves together\n\nOnce a week, for organizations that have also switched on **learning**, the control plane reads the\npast 30 days of traffic, up to 2m requests, and replays it through a Go port of the gateway's\nalgorithm: the same tiers, confidence gate, ski-rental accounting, free switches and cooldown.\n\n### Sessions and gaps\n\nRequests are grouped into sessions and split by agent (Claude Code, Codex and the rest), since a map\nthat suits one agent's prompts may not suit another's. Spending the replay cannot cover is reported,\nnot extrapolated: traffic from before classification was switched on, sessions without labels, and\nrequests without a session id. The share it does cover is the report's *coverage*.\n\n### Candidate maps\n\nFor each tier, the candidates are the models the organization may use whose catalog rating clears a deliberately cautious floor, one notch above the tier's bare minimum, and whose context window holds at least 200,000 tokens:\n\n| Tier | Lowest catalog rating allowed | \n|---|---|\n| low | Strong | \n| medium | Elite | \n| high | Frontier | \n| max | Frontier | \n\nThe ratings come from Edgee's model catalog, which places every model on four levels, from Basic to\nFrontier, according to its results on public benchmarks: chiefly\n[SWE-bench Verified](https://www.swebench.com/), which measures how often a model resolves real\nGitHub issues, and the [Artificial Analysis Intelligence Index](https://artificialanalysis.ai/),\nwhich combines a range of evaluations that Artificial Analysis runs independently.\n\nThe replay keeps the three cheapest candidates per tier and tries every combination, at most 3⁴ = 81 maps per agent. Each map replays every session under the organization's current switching settings, with every request repriced at catalog prices on the model the replay would have chosen (uncached input, cache writes, cache reads and output counted separately). The recommended map is the cheapest, and never one that would have cost more than the organization actually paid.\n\n### The hindsight optimum\n\nCompetitive analysis judges an online rule against the offline optimum, which knows the future. A replay knows how every session ended, so it can compute that optimum. For each session and map it finds, by dynamic programming over the requests, the cheapest assignment of the map's models that serves every request at least at the tier the gateway required, counting a cold cache write whenever consecutive requests change model:\n\n```\nbest[m] after request i = min over models m' of\n    best[m'] after request i−1 + cost(request i on m, warm if m' = m, cold otherwise)\n```\n\nrestricted to models capable of request `i`'s tier. Because it may change model at any request, even\ninside a turn, which the gateway never does, no online policy can beat it. The gap between the cost of the online switching\npolicy and this optimum is the price of not knowing how long each stretch of a session will last. It\nshows how much there is left to gain from switching better, as distinct from choosing better models.\n\n### Quality: never below the label\n\nThe replay does not guess how a different model would have answered. It has no ground truth for that,\nand does not pretend otherwise. Quality is enforced structurally instead. Every candidate clears its\ntier's floor, so a turn served at its labeled tier always lands on a capable model. What the replay\nmeasures is the exception: the **below-label share**, the fraction of routed requests served by a\nmodel below their label's tier, as happens while a low-confidence upgrade waits for its second turn.\nNo comparison below trades money for a higher below-label share.\n\n### Choosing the switching settings\n\nWith the map chosen, the replay compares six switching settings on it: the defaults; a lower\nthreshold (`λ = 0.5`) and a higher one (`λ = 5`); never moving down a tier to save money; no\nconfidence gate; and a setting built on our own measurements (see below), which follows each\nrequest's cache lifetime, does not wait between switches and keeps the model on turns the agent\nwrites for itself. The settings in force join the comparison when they match none of these.\n\nThe two thresholds put the theory to the test. Their worst-case ratios, 3 and 6, are both worse than the default's 2.5, yet either can win on real traffic, because the worst case is rare. An organization whose easy stretches tend to run long gains from buying early; one whose easy turns come singly, between hard ones, gains from buying late or not at all. Worst-case analysis cannot tell the two apart. A replay of the organization's own sessions can.\n\nA new setting replaces the old one only if it saves at least half a point more of the billed spend without serving more than 0.2 points more requests below their label, which keeps a weekly run from flip-flopping between settings that differ only by noise.\n\n### Uncertainty\n\nA replayed saving is an estimate drawn from a few hundred sessions of wildly different sizes, so the\nreport gives a 95% interval, from a bootstrap over sessions (Efron, *Bootstrap methods: another look\nat the jackknife*, Annals of Statistics, 1979): resample sessions with replacement, recompute the\nsaving on each resample, and read off the 2.5th and 97.5th percentiles. The seed is fixed, so the same\ndata always yield the same report. Confidence is **high** when the replayed sessions cover at least\n70% of spend and the interval is at most ten points wide, **low** below 40% coverage, and **medium**\notherwise.\n\n## Prior art, and where this differs\n\nLearned LLM routing has so far focused on the first half. RouteLLM (Ong et al, 2024) learns to choose between a strong and a weak model from human preference data. FrugalGPT (Chen, Zaharia and Zou, 2023) cascades models of increasing cost until an answer is good enough. Hybrid LLM (Ding et al, ICLR 2024) routes between a small and a large model according to a query's predicted difficulty. Each routes queries independently, with no model of what it costs to change model in the middle of a conversation.\n\nOn the switching side, the closest prior art is the theory itself. Beyond the deterministic and\nrandomized rules above, learning-augmented algorithms (Purohit, Svitkina and Kumar, *Improving online\nalgorithms via ML predictions*, NeurIPS 2018) feed a prediction of the season's length into the\nski-rental rule, to beat the worst-case bound when the prediction is good while keeping a bound when\nit is not. Self-Learning Routing does something coarser: it predicts nothing for a single session,\nbut tunes the threshold to the stretches in an organization's past sessions.\n\nSelf-Learning Routing adds the second half. It treats the change of model inside a session as an online rent-or-buy decision with an explicit cost model and a worst-case guarantee, evaluates it against the hindsight optimum, and tunes the tier map and the switching policy jointly, because each changes the value of the other. It evaluates itself by replaying the organization's own sessions, in the spirit of replay-based offline evaluation (Li, Chu, Langford and Wang, WSDM 2011), rather than on a public benchmark. Its main assumption should be stated plainly: it reprices the tokens each request actually used, so it does not capture a different model writing longer or shorter answers.\n\n## From a recommendation to a running Strategy\n\nEach weekly run stores its report and records what became of it: **kept** (nothing would cost less,\nor the self-learned Strategy already runs the best map), **proposed** (awaiting an admin),\n**applied**, or **skipped** (a guardrail held an automatic change back). An admin can apply any\nproposed run from the console. With *Apply automatically* on, the run updates the self-learned\nStrategy itself, but only when every guardrail passes:\n\n| Guardrail | Otherwise | \n|---|---|\n| An admin has already applied a self-learned Strategy and assigned it somewhere | proposed: the first application, and where it runs, stay an admin's choice | \n| Nobody has edited the Strategy since it was last applied | proposed: an admin decides whether the new map replaces the edit | \n| Confidence is medium or high | skipped | \n| The lower end of the saving's 95% interval is above zero | skipped | \n| It saves at least three points more than the map in force, on the same sessions | skipped | \n| It serves no more than 0.2 points more requests below their label | skipped | \n| Every model in the map served at least 30 of the organization's requests in the window | proposed: an admin decides whether to roll out a model the team has not used | \n| No model in the map failed more often on the organization's traffic than the models agents asked for | skipped | \n\nEvery change records what came before. An admin can undo the latest one from the console: the previous map and switching settings return, and automatic apply turns off, so the next weekly run proposes a change rather than quietly reapplying it.\n\n## What we measured\n\nThe defaults are the cautious choice. One measurement of Edgee's own coding-agent traffic shows why\nthe replay also tries a less cautious setting, and why it tests that setting on every organization\nrather than assuming it: **agents trigger many switches themselves.** In our replays about half of all\nswitches were set off by text the agent wrote for itself (status lines, titles, suggested prompts,\ntask notices), which says nothing about how hard the work is.\n\nWe do not publish a saving from routing alone. Each organization's report shows its own replayed saving, with its interval and coverage, on its own traffic.\n\n## Limits, and what comes next\n\nTwo things Self-Learning Routing does not do yet are worth stating plainly, because the next steps start from them:\n\n- **Repriced, not re-generated.** The replay assumes a different model would have written answers of\nthe same length.\n- **Training is a separate opt-in.** An organization can consent to Edgee keeping the labeled text to\ntrain its own routing classifier; nothing is collected yet.\n\nThe next steps follow from these limits, and from where the theory stops.\n\n**Measuring answers, not only prices.** Re-running a sample of turns on the candidate model would show\nhow long its answers really are. The replay's main assumption would become a measurement.\n\n**Pricing a switch that gets dearer.** The textbook prices skis once; a session's buy price rises with\nevery turn, because a switch rewrites a longer conversation. Waiting therefore costs twice, in rent\npaid and in a dearer purchase, which argues for moving earlier in a long stretch than a fixed-price\nrule does. A rule that anticipates the rise, with a guarantee that covers it, is open work. The\nhindsight optimum already gives the yardstick to judge one.\n\n**Predicting how long a stretch will last.** Today the threshold is tuned per organization, from the\nwhole history of its sessions. The labels already say what kind of work each turn is and who wrote\nit, which may predict whether an easy stretch will be long or short. Learning-augmented ski rental\nshows how to use such a prediction safely: lean on it when it is right, and keep a worst-case bound\nwhen it is wrong. The replay can test any such predictor against the hindsight optimum before it\ntouches live traffic.\n\n**A classifier trained for coding agents.** Turns labeled with an organization's consent could train\na routing classifier tuned to coding-agent traffic rather than a general-purpose one. Better labels\nimprove both halves at once: the tier each turn gets, and the stretches the switching rule sees.\n\n## Conclusion\n\nChoosing a model by difficulty tells a router where a turn could go. It does not say whether moving there pays, because the model a coding session is on also holds its cache. That second question is an online one, and ski rental answers it with a simple rule and a worst-case guarantee: keep count of the savings forgone, and move once they match the cost of moving.\n\nWhat theory cannot say is which models to use, or how long an organization's easy stretches really last. Both depend on the organization's own work. Self-Learning Routing learns them from its own sessions: it replays them under every candidate map and switching setting, measures each against what hindsight would have done, and changes the Strategy only when the evidence is clear. Difficulty picks the model, ski rental decides when to switch, and an organization's own history sets the numbers for both.\n\nSelf-Learning Routing belongs to Route, the first of the Agent Gateway's three pillars (Route, Compress, Observe), and its labels serve the third as well: they show a platform team how its work divides by difficulty and kind before it sets a budget or a Strategy.\n\nThe only way to know what this would do for your team is to run it on your team's sessions. Routing your coding agents through Edgee takes a CLI install, about five minutes, and no code changes:\n\n```\ncurl -fsSL https://edgee.ai/install.sh | bash\nedgee launch claude   # or codex, opencode, cursor, copilot, codebuddy\n```\n\nOn a Team or Enterprise plan, turn on classification in the console. From then on every turn is\nlabeled as it is served, and the replay shows, on your own traffic, which model each tier should use,\nhow often your sessions would switch, and what that would have cost, with its interval and coverage.\nNothing changes until you apply it. If you run a fleet of coding agents and want to walk through that\nreport with us, [book a call](https://www.edgee.ai/contact). And if you have questions or ideas\nabout the approach, we would be glad to hear them on [Discord](https://www.edgee.ai/discord).", "url": "https://wpnews.pro/news/switching-llms-mid-conversation-is-a-ski-rental-problem", "canonical_source": "https://www.edgee.ai/blog/posts/switching-llms-mid-conversation-is-a-ski-rental-problem", "published_at": "2026-11-10 16:47:37+00:00", "updated_at": "2026-11-10 16:59:40.566102+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["Edgee", "Agent Gateway", "Anthropic", "Self-Learning Routing"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/switching-llms-mid-conversation-is-a-ski-rental-problem", "markdown": "https://wpnews.pro/news/switching-llms-mid-conversation-is-a-ski-rental-problem.md", "text": "https://wpnews.pro/news/switching-llms-mid-conversation-is-a-ski-rental-problem.txt", "jsonld": "https://wpnews.pro/news/switching-llms-mid-conversation-is-a-ski-rental-problem.jsonld"}}