{"slug": "llm-drift-tracking-why-my-alerts-were-all-wrong", "title": "LLM Drift Tracking: Why My Alerts Were All Wrong", "summary": "A public LLM benchmark board triggered four false drift alerts between July 21 and 24, all caused by rate limits or small eval-set noise rather than actual model degradation. Maintainer Egnaro found that Gemini 3.5 Flash's 11.4-point drop was due to 34 of 35 calls being rate-limited, while Grok 4.3 and Llama 3.3 70B shifts of 5.7 and 2.9 points corresponded to just one or two question flips in a 35-task suite. The system now uses a human-in-the-loop stub with a mandatory reliability check, and the author recommends tracking an 'alert survival rate' metric to distinguish real regression from noise.", "body_md": "# LLM Drift Tracking: Why My Alerts Were All Wrong\n\nI maintain a public board that probes 16 LLMs on a frozen suite of 35 tasks daily. The system is designed to be autonomous; if a model's score drops compared to the previous run, it automatically opens a GitHub issue and drafts a post. Between July 21 and 24, the system triggered four times:\n\n```\n23 Jul Gemini 3.5 Flash -11.4 pts\n24 Jul Gemini 3.1 Pro -2.9 pts\n21 Jul Grok 4.3 -5.7 pts\n22 Jul Llama 3.3 70B -2.9 pts\n```\n\nOn the surface, this looks like a systemic collapse of model performance. In reality, none of these models actually got worse.\n\n## The Reliability Gap\n\nThe first failure in the alerting logic was a confusion between intelligence and availability. Every score on my board is paired with a **reliability** metric—the percentage of probe calls that successfully returned a response.\n\nLooking at the logs for the [Gemini](/en/tags/gemini/) 3.5 Flash \"crash\":\n\n```\ngemini-3.5-flash 22 Jul acc 1.000 reliability 1.000\n 23 Jul acc 0.886 reliability 0.914 429: Rate limit reached for model `llama-3.3-70b-versatile`\nservice tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30\n```\n\nThe model didn't lose its capabilities; 34 out of 35 calls were rate-limited. Because the tracker scored a 429 error as 0%, the resulting dip looked like a regression. This is the most dangerous type of false positive in an AI workflow: a provider outage or rate limit masquerading as a cognitive decline. Gemini 3.1 Pro followed the same pattern, bouncing back to 97.1% in the next clean run, higher than its previous baseline.\n\n## Signal vs. Noise in Small Eval Sets\n\nThe other two alerts were more insidious because the reliability remained at 1.000. The numbers were \"real,\" but the interpretation was wrong:\n\n**Grok 4.3:** 0.800 -> 0.743 (-5.7 pts)**Llama 3.3 70B:** 0.800 -> 0.771 (-2.9 pts)\n\nIn a 35-task suite, one single question represents 2.86 points. A drop of -2.9 points is exactly one question changing its answer. A drop of -5.7 is exactly two.\n\nReporting a one-question flip as a \"model regression\" is reading noise as signal. When your eval suite is this small, you cannot distinguish between a genuine drift in LLM reasoning and the inherent stochasticity of the model.\n\n## Building a Robust Drift Detection Workflow\n\nThe temptation is to lower the sensitivity—perhaps only alert if a model drops by 10+ points. However, that ignores the reality that real regressions often start small. The solution isn't to mute the tracker, but to decouple **detection** from **publication**.\n\nMy current deployment uses a \"human-in-the-loop\" stub. The automation logs the event, but the draft contains a mandatory warning:\n\n```\nAuto-logged when the scheduled probe flagged a run-over-run regression. \nBefore this becomes a post, check the run log and the Reliability metric \n— a rate limit or provider outage can look exactly like a regression.\n```\n\n## The \"Alert Survival Rate\" Metric\n\nFor anyone building a practical tutorial for internal evals, I suggest tracking one specific metric: **the share of alerts that survive manual verification.**\n\nThis week, my survival rate was 0%. While that looks bad on a dashboard, it is the most valuable data point I have. It proves that my suite is too small to resolve single-task noise and that accuracy metrics are lies if they aren't displayed alongside reliability data.\n\nThe hardest part of maintaining a deep dive into model performance isn't detecting the drift—it's ensuring you aren't manufacturing it.\n\n`https://egnaro9.github.io/model-drift`\n\n[Next Claude 3.5 Sonnet vs. GPT-4o: Legal Reasoning Benchmarks →](/en/threads/2955/)", "url": "https://wpnews.pro/news/llm-drift-tracking-why-my-alerts-were-all-wrong", "canonical_source": "https://promptcube3.com/en/threads/2975/", "published_at": "2026-07-25 00:13:34+00:00", "updated_at": "2026-07-25 00:36:43.795104+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools"], "entities": ["Gemini 3.5 Flash", "Gemini 3.1 Pro", "Grok 4.3", "Llama 3.3 70B", "Egnaro"], "alternates": {"html": "https://wpnews.pro/news/llm-drift-tracking-why-my-alerts-were-all-wrong", "markdown": "https://wpnews.pro/news/llm-drift-tracking-why-my-alerts-were-all-wrong.md", "text": "https://wpnews.pro/news/llm-drift-tracking-why-my-alerts-were-all-wrong.txt", "jsonld": "https://wpnews.pro/news/llm-drift-tracking-why-my-alerts-were-all-wrong.jsonld"}}