I maintain a public board that probes 16 LLMs on a frozen suite of 35 tasks daily. The system is designed to be autonomous; if a model's score drops compared to the previous run, it automatically opens a GitHub issue and drafts a post. Between July 21 and 24, the system triggered four times:
23 Jul Gemini 3.5 Flash -11.4 pts
24 Jul Gemini 3.1 Pro -2.9 pts
21 Jul Grok 4.3 -5.7 pts
22 Jul Llama 3.3 70B -2.9 pts
On the surface, this looks like a systemic collapse of model performance. In reality, none of these models actually got worse.
The Reliability Gap #
The first failure in the alerting logic was a confusion between intelligence and availability. Every score on my board is paired with a reliability metric—the percentage of probe calls that successfully returned a response.
Looking at the logs for the Gemini 3.5 Flash "crash":
gemini-3.5-flash 22 Jul acc 1.000 reliability 1.000
23 Jul acc 0.886 reliability 0.914 429: Rate limit reached for model `llama-3.3-70b-versatile`
service tier `on_demand` ... requests per minute (RPM): Limit 30, Used 30
The model didn't lose its capabilities; 34 out of 35 calls were rate-limited. Because the tracker scored a 429 error as 0%, the resulting dip looked like a regression. This is the most dangerous type of false positive in an AI workflow: a provider outage or rate limit masquerading as a cognitive decline. Gemini 3.1 Pro followed the same pattern, bouncing back to 97.1% in the next clean run, higher than its previous baseline.
Signal vs. Noise in Small Eval Sets #
The other two alerts were more insidious because the reliability remained at 1.000. The numbers were "real," but the interpretation was wrong:
Grok 4.3: 0.800 -> 0.743 (-5.7 pts)Llama 3.3 70B: 0.800 -> 0.771 (-2.9 pts)
In a 35-task suite, one single question represents 2.86 points. A drop of -2.9 points is exactly one question changing its answer. A drop of -5.7 is exactly two.
Reporting a one-question flip as a "model regression" is reading noise as signal. When your eval suite is this small, you cannot distinguish between a genuine drift in LLM reasoning and the inherent stochasticity of the model.
Building a Robust Drift Detection Workflow #
The temptation is to lower the sensitivity—perhaps only alert if a model drops by 10+ points. However, that ignores the reality that real regressions often start small. The solution isn't to mute the tracker, but to decouple detection from publication.
My current deployment uses a "human-in-the-loop" stub. The automation logs the event, but the draft contains a mandatory warning:
Auto-logged when the scheduled probe flagged a run-over-run regression.
Before this becomes a post, check the run log and the Reliability metric
— a rate limit or provider outage can look exactly like a regression.
The "Alert Survival Rate" Metric #
For anyone building a practical tutorial for internal evals, I suggest tracking one specific metric: the share of alerts that survive manual verification.
This week, my survival rate was 0%. While that looks bad on a dashboard, it is the most valuable data point I have. It proves that my suite is too small to resolve single-task noise and that accuracy metrics are lies if they aren't displayed alongside reliability data.
The hardest part of maintaining a deep dive into model performance isn't detecting the drift—it's ensuring you aren't manufacturing it.
https://egnaro9.github.io/model-drift
Next Claude 3.5 Sonnet vs. GPT-4o: Legal Reasoning Benchmarks →