{"slug": "distinguishing-wrong-from-absent", "title": "Distinguishing wrong from absent", "summary": "A developer building the model-drift evaluation tool describes how its exact-match grader can conflate wrong answers with absent ones, such as rate limits, refusals, or truncations. The tool excludes failures only when a run's reliability falls below 50%, avoiding blanket exclusion that would inflate accuracy on difficult tasks. The methodology is open-sourced on GitHub.", "body_md": "model-drift grades models weekly on a frozen suite with an exact-match grader — no LLM judge, so a score change is real. That design has a sharp edge: a call that returns no valid answer scores identically to a wrong one. A refusal, a max_tokens truncation, a timeout, a parser failure on a quietly-changed schema — all land as zero, indistinguishable from the capability collapse the board exists to catch.\n\nI hit the catchable version: a model appeared to drop from 69% to 3% overnight. The run log showed a 429 on 34 of 35 calls — a rate limit scored as a regression. Rate limits are catchable only because they leave a status code; refusals and truncations don't.\n\nThe board excludes by aggregate reliability: it drops a run's accuracy point only when that run's reliability falls below 50% — a catastrophic-infra floor, never a single failed call. That restraint matters because failures aren't missing-at-random: the long, hard prompts are the ones that hit token caps and timeouts, so a blanket drop-every-failure rule would exclude failures that correlate with difficulty and inflate accuracy on the discriminating tasks. An eval board has to tell a wrong answer from an absent one; miss that and you're scoring the provider's uptime, not the model.\n\nThe design was already exclude-by-class, not blanket. Code: github.com/egnaro9/model-drift\n\nWriting the methodology in public is how it gets read this closely.", "url": "https://wpnews.pro/news/distinguishing-wrong-from-absent", "canonical_source": "https://dev.to/egnaro9/distinguishing-wrong-from-absent-57ep", "published_at": "2026-07-21 19:42:32+00:00", "updated_at": "2026-07-21 20:21:26.800767+00:00", "lang": "en", "topics": ["machine-learning", "mlops", "developer-tools"], "entities": ["model-drift", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/distinguishing-wrong-from-absent", "markdown": "https://wpnews.pro/news/distinguishing-wrong-from-absent.md", "text": "https://wpnews.pro/news/distinguishing-wrong-from-absent.txt", "jsonld": "https://wpnews.pro/news/distinguishing-wrong-from-absent.jsonld"}}