LLM Drift Tracking: Why My Alerts Were All Wrong

A public LLM benchmark board triggered four false drift alerts between July 21 and 24, all caused by rate limits or small eval-set noise rather than actual model degradation. Maintainer Egnaro found that Gemini 3.5 Flash's 11.4-point drop was due to 34 of 35 calls being rate-limited, while Grok 4.3 and Llama 3.3 70B shifts of 5.7 and 2.9 points corresponded to just one or two question flips in a 35-task suite. The system now uses a human-in-the-loop stub with a mandatory reliability check, and the author recommends tracking an 'alert survival rate' metric to distinguish real regression from noise.

LLM Drift Tracking: Why My Alerts Were All Wrong I maintain a public board that probes 16 LLMs on a frozen suite of 35 tasks daily. The system is designed to be autonomous; if a model's score drops compared to the previous run, it automatically opens a GitHub issue and drafts a post. Between July 21 and 24, the system triggered four times: 23 Jul Gemini 3.5 Flash -11.4 pts 24 Jul Gemini 3.1 Pro -2.9 pts 21 Jul Grok 4.3 -5.7 pts 22 Jul Llama 3.3 70B -2.9 pts On the surface, this looks like a systemic collapse of model performance. In reality, none of these models actually got worse. The Reliability Gap The first failure in the alerting logic was a confusion between intelligence and availability. Every score on my board is paired with a reliability metric—the percentage of probe calls that successfully returned a response. Looking at the logs for the Gemini /en/tags/gemini/ 3.5 Flash "crash": gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000 23 Jul acc 0.886 reliability 0.914 429: Rate limit reached for model llama-3.3-70b-versatile service tier on demand ... requests per minute RPM : Limit 30, Used 30 The model didn't lose its capabilities; 34 out of 35 calls were rate-limited. Because the tracker scored a 429 error as 0%, the resulting dip looked like a regression. This is the most dangerous type of false positive in an AI workflow: a provider outage or rate limit masquerading as a cognitive decline. Gemini 3.1 Pro followed the same pattern, bouncing back to 97.1% in the next clean run, higher than its previous baseline. Signal vs. Noise in Small Eval Sets The other two alerts were more insidious because the reliability remained at 1.000. The numbers were "real," but the interpretation was wrong: Grok 4.3: 0.800 - 0.743 -5.7 pts Llama 3.3 70B: 0.800 - 0.771 -2.9 pts In a 35-task suite, one single question represents 2.86 points. A drop of -2.9 points is exactly one question changing its answer. A drop of -5.7 is exactly two. Reporting a one-question flip as a "model regression" is reading noise as signal. When your eval suite is this small, you cannot distinguish between a genuine drift in LLM reasoning and the inherent stochasticity of the model. Building a Robust Drift Detection Workflow The temptation is to lower the sensitivity—perhaps only alert if a model drops by 10+ points. However, that ignores the reality that real regressions often start small. The solution isn't to mute the tracker, but to decouple detection from publication . My current deployment uses a "human-in-the-loop" stub. The automation logs the event, but the draft contains a mandatory warning: Auto-logged when the scheduled probe flagged a run-over-run regression. Before this becomes a post, check the run log and the Reliability metric — a rate limit or provider outage can look exactly like a regression. The "Alert Survival Rate" Metric For anyone building a practical tutorial for internal evals, I suggest tracking one specific metric: the share of alerts that survive manual verification. This week, my survival rate was 0%. While that looks bad on a dashboard, it is the most valuable data point I have. It proves that my suite is too small to resolve single-task noise and that accuracy metrics are lies if they aren't displayed alongside reliability data. The hardest part of maintaining a deep dive into model performance isn't detecting the drift—it's ensuring you aren't manufacturing it. https://egnaro9.github.io/model-drift Next Claude 3.5 Sonnet vs. GPT-4o: Legal Reasoning Benchmarks → /en/threads/2955/