Show HN: Throttling AI models under load can backfire and increase demand (SIM)

A Hacker News user (OP) reports that throttling AI models under load can backfire by increasing demand, as users re-ask when they notice degraded responses, and this effect is worse for agents. The OP modeled the problem as a fleet scheduling problem using Queueing Theory and Dynamic Programming, finding that standard throttling is suboptimal and that traffic should be segmented by retry sensitivity. The analysis is backed by a paper on arXiv (2608.23986) and a demo, and the OP invites feedback from inference infrastructure workers.

OP here: TL-DR: When AI providers silently swap weaker models under high demand, this can cause even more demand as users tend to re-ask. This is even worse for agents. Both of these effects cause even higher load on the data centers. I guess all of us have felt when the models "don't feel quite the same", so this could explain part of it. I modeled this as a fleet scheduling problem using Queueing Theory and Dynamic Programming over a finite horizon. The standard practice of throttling once the number of jobs in server exceeds certain threshold is in fact suboptimal. The optimal policy consists in segmenting the part of the traffic that is retry sensitive, from those that are not. For example, an user doing a basic data parsing might still do well under a weaker model, but a power user will certainly feel the degradation and ask more. Demo: Just a toy instance to illustrate the issue. The user can create their own policies and see how they perform against the industry standard and the optimal one. It is roughly 100 lines of Flask + JS frontend. Paper with proofs: https://arxiv.org/abs/2608.23986 https://arxiv.org/abs/2608.23986 For those of you who have worked in inference infra, does this match anything you have seen? Comments URL: https://news.ycombinator.com/item?id=49521092 https://news.ycombinator.com/item?id=49521092 Points: 2 Comments: 0