{"slug": "my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong", "title": "My LLM drift tracker flagged four regressions this week. All four were wrong.", "summary": "A developer's LLM drift tracker flagged four regressions across Gemini, Grok, and Llama models in July, but all were false alarms caused by API failures or measurement granularity. The tracker's reliability metric revealed that rate limits and small test suites produce misleading signals, with one apparent 66-point drop traced to a single 429 error.", "body_md": "I run a public board that probes 16 LLMs on a frozen 35-task suite, once a day, and keeps every score. When a model drops against its previous run, it opens a GitHub issue by itself and writes me a draft post.\n\nBetween 21 and 24 July it did that four times:\n\n```\n23 Jul  Gemini 3.5 Flash   -11.4 pts\n24 Jul  Gemini 3.1 Pro      -2.9 pts\n21 Jul  Grok 4.3            -5.7 pts\n22 Jul  Llama 3.3 70B       -2.9 pts\n```\n\nFour regressions in four days, across three labs. That's a post that writes itself, and it would have been fast, legible, and wrong.\n\nNone of those models got worse. Here's how I know, because the how is the only part worth reading.\n\nEvery point on the board carries a second number next to accuracy: **reliability**, the share of probe calls that actually came back. Look at the two Google alerts with that column showing:\n\n```\ngemini-3.5-flash  22 Jul  acc 1.000  reliability 1.000\n                  23 Jul  acc 0.886  reliability 0.914   <- \"-11.4 pts\"\n\ngemini-3.1-pro    22 Jul  acc 0.914  reliability 0.943\n                  23 Jul  acc 0.886  reliability 0.914   <- \"-2.9 pts\"\n                  24 Jul  acc 0.971  reliability 1.000   <- next clean run\n```\n\nAccuracy and reliability fell together. That's the signature of calls that never returned, not answers that got worse — a failed call has no answer to grade, and an ungraded task scores the same as a wrong one.\n\nI know this signature well because this board already published the lesson. On 20 July, Llama 3.3 70B appeared to fall 66 points overnight:\n\n``` php\napi.groq.com -> 429: Rate limit reached for model `llama-3.3-70b-versatile`\nservice tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30\n```\n\n34 of 35 calls were rate-limited. The model didn't get dumber; a 429 scored as a zero. **A rate limit scoring as a 0% is the single most misleading thing a drift tracker can do, because it looks exactly like the thing the tracker exists to catch.**\n\nGemini 3.1 Pro settles its own case: the next clean run came back at 97.1%, *higher* than before the \"regression.\"\n\nThe remaining two alerts are more interesting, because reliability held at 1.000 the whole time. Those numbers are real:\n\n``` php\ngrok-4.3        0.800 -> 0.743   = -5.7 pts\nllama-3.3-70b   0.800 -> 0.771   = -2.9 pts\n```\n\nThe suite is 35 tasks. One task is `100/35 = 2.86`\n\npoints.\n\nSo -2.9 points is **one question changing its answer.** -5.7 is two. And -11.4, the scariest number in the set, is four.\n\nA 35-task suite cannot resolve anything finer than about three points. Every \"regression\" my board flagged this week was an integer number of questions, which is the tell: I wasn't measuring drift, I was measuring the granularity of my own instrument. Reporting a one-question flip as a model regression is reading noise as signal — and doing it in public, about a named company's model.\n\nThe obvious fix is to make the tracker quieter — only fire above 10 points, say. I don't think that's right. A tracker that only fires on catastrophes misses the drift you actually want to catch, and the -11.4 that turned out to be failed calls is exactly the shape of a real regression. Sensitivity is the feature.\n\nSensitivity is only *safe*, though, if something downstream is willing to say no. So the alert doesn't publish anything. It writes a stub that says, in its own text:\n\nAuto-logged when the scheduled probe flagged a run-over-run regression. Before this becomes a post, check the run log and the Reliability metric — a rate limit or provider outage can look exactly like a regression.\n\nThe automation's job is to notice. Mine is to check. This week that split did real work: four notices, zero posts.\n\nIf you build evals, you already track your models' scores. The metric I'd argue you're missing is **the share of your own alerts that survive checking.**\n\nMine, this week, was zero. That's not a comfortable number to publish, and it's the most useful one I have — it tells me the suite is too small to resolve single-task noise, and that reliability has to sit beside accuracy on every chart or the chart lies.\n\nBoth of those are fixable. Neither would have been visible if I'd shipped the post the tracker wrote for me.\n\nThe hard part of a drift tracker isn't detecting drift. It's not manufacturing it.\n\n**The board:** [egnaro9.github.io/model-drift](https://egnaro9.github.io/model-drift/) — 16 models, 5 metrics, daily, every run kept. The field notes are on the page; this one is \"Four regression alerts, zero regressions.\"\n\n**The code:** [github.com/egnaro9/model-drift](https://github.com/egnaro9/model-drift). No LLM-as-judge anywhere — every task is graded by a fixed deterministic check, so a score change means the model moved, not the test.", "url": "https://wpnews.pro/news/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong", "canonical_source": "https://dev.to/agentdev9/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong-2i6e", "published_at": "2026-07-24 23:55:15+00:00", "updated_at": "2026-07-25 00:01:29.715523+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Gemini", "Grok", "Llama", "Google", "Groq"], "alternates": {"html": "https://wpnews.pro/news/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong", "markdown": "https://wpnews.pro/news/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong.md", "text": "https://wpnews.pro/news/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong.txt", "jsonld": "https://wpnews.pro/news/my-llm-drift-tracker-flagged-four-regressions-this-week-all-four-were-wrong.jsonld"}}