{"slug": "before-scoring-a-model-test-the-evaluator", "title": "Before scoring a model, test the evaluator", "summary": "A developer building IaC Eval Bench, a project for scoring model-generated Kubernetes configurations, found five scoring errors in the evaluator during a review, even though its six reference answers and six targeted negative controls produced the expected results. The evaluator wrongly passed a Deployment selector whose matchExpressions contradicted its matchLabels and pod template, and wrongly rejected a configuration that merely added an extra label. The fix checks every declared selector condition against the submitted pod labels, following Kubernetes' rule that all label-selector conditions must be satisfied.", "body_md": "One version of my evaluator passed a Kubernetes configuration whose selector contradicted its own pod labels. It also rejected a configuration just because an extra label had been added.\n\n**Both results were scoring errors.**\n\nIf you are new to evals, start with [What exactly are AI evals?](https://dev.to/deavgprime/what-exactly-are-ai-evals-775). It explains the tasks, graders, and metrics used in this case study.\n\nThese came up while working on [IaC Eval Bench](https://github.com/deavgprime/iac-eval-bench), a small project for evaluating model-generated Kubernetes configurations. A scoring review found five mistakes in the evaluator, even though its original reference and negative-control examples were producing the expected totals.\n\nThat last part is worth explaining. A passing test suite tells us that the cases we wrote behave as expected. We still have to ask whether we chose enough useful cases.\n\n**On this page**\n\nThe benchmark has six tasks. They ask for configurations such as a Deployment with resource limits, a Service that selects the right pods, or a scheduled job with execution limits.\n\nFor each answer, the evaluator checks:\n\nA schema is a description of the allowed structure. For example, it can tell us that a replica count should be a number. That does not mean it checks every relationship between the fields.\n\nThe evaluator saves the answer and the outcome of its checks so a result can be inspected later.\n\n*Figure 1. The evaluator checks configuration data without deploying it. Task requirements are available when it scores the answer.*\n\nTake a Service, which directs traffic to a set of pods. Kubernetes uses labels to decide which pods the Service selects.\n\nHere is the relevant part of a matching example. These are excerpts, not complete manifests:\n\n```\n# Pod labels in the Deployment template\nlabels:\n  app: api\n\n# Service selector\nselector:\n  app: api\n```\n\nThe Service asks for pods with `app: api`, and the pod template has that label.\n\nNow change only the Service selector:\n\n```\nselector:\n  app: unrelated\n```\n\nBoth versions have the same field structure. The second selector does not match the intended pods. The evaluator needs to check that relationship.\n\nThis is a useful negative control: a constructed answer with a mistake we expect the evaluator to catch. The unchanged answer is the positive control.\n\nThe project has one reference answer and one targeted negative example for each task:\n\n| Constructed examples | Full task passes | Schema and resource checks passed | \n|---|---|---|\n| Six reference answers | 6/6 | 6/6 | \n| Six answers with a targeted mistake | 0/6 | 6/6 | \n\nThose are the expected results. All six negative examples still get through the schema and resource checks. Each fails its intended task requirement.\n\n**Reading these results:** These are constructed examples for testing the evaluator. No model produced them, so the pass counts do not measure model ability.\n\nThe original examples all behaved as expected. Then the review tried changes beyond those six negative controls.\n\nOne change added this condition to a Deployment selector:\n\n```\nselector:\n  matchLabels:\n    app: api\n  matchExpressions:\n    - key: app\n      operator: NotIn\n      values: [api]\n```\n\nRead the two conditions together:\n\n`app` to equal `api`.\nThey cannot both be satisfied. The selector also contradicts the Deployment's own pod template, which has `app: api`.\n\nThe old evaluator passed this configuration because it checked `matchLabels` and ignored `matchExpressions`.\n\nThe fix checks every declared selector condition against the submitted pod labels. This follows Kubernetes' rule that the conditions in a label selector must all be satisfied. [Kubernetes selector documentation](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#label-selectors)\n\nHere is the other side of the problem:\n\n```\nlabels:\n  app: api\n  team: platform\n```\n\nThe required `app: api` label is still present. A selector asking for `app: api` still matches these labels.\n\nThe old label check compared the whole label map with the reference map. The extra `team` entry caused a failure.\n\nThat check was too strict. It now checks for the required label while allowing compatible additions. The task wording also makes those allowed additions explicit, so the prompt and the scoring rule agree.\n\nThis matters for model evaluation. If the evaluator rejects an acceptable answer, the model gets blamed for a scoring mistake. If the evaluator accepts a broken answer, the score looks better than it should.\n\n*Figure 2. The conflicting selector must fail; the compatible extra label must still pass. Both cases need regression coverage.*\n\nThe review recorded five scoring mistakes:\n\n| Change to the reference answer | Old | Corrected | \n|---|---|---|\n| Conflicting Deployment selector | Pass | **Fail** | \n| Compatible extra pod label | Fail | **Pass** | \n| Matching PodDisruptionBudget condition | Fail | **Pass** | \n| Two readiness probe handlers | Pass | **Fail** | \n| UDP workload port with TCP Service | Pass | **Fail** | \n\nA PodDisruptionBudget describes how many pods should remain available during voluntary disruptions. Like the Deployment, it uses a selector to identify the relevant pods.\n\nThe [scoring audit](https://github.com/deavgprime/iac-eval-bench/blob/1dbabecd8cc377cef1082b2652131b6921c4c74b/docs/scoring-audit.md) explains the individual fixes and links the saved before-and-after observations.\n\nThe original controls were useful. They caught the mistakes they were designed to catch. But neither their old nor their corrected totals exposed these five scoring errors. Both versions reported six reference passes and zero negative-control passes.\n\nThe individual cases mattered more than those totals.\n\nThe examples can run locally without a model endpoint or Kubernetes cluster. With Python 3.11 to 3.13 installed:\n\n**1. Get the version used in this article.**\n\n```\ngit clone https://github.com/deavgprime/iac-eval-bench.git\ncd iac-eval-bench\ngit switch --detach 1dbabecd8cc377cef1082b2652131b6921c4c74b\n```\n\n**2. Prepare the Python environment.**\n\n```\npython3 -m venv .venv\nsource .venv/bin/activate\npython -m pip install -r requirements.lock\npython -m pip install --no-deps -e .\n```\n\n**3. Run the tests and constructed examples.**\n\n```\npython -m unittest discover -s tests -v\npython scripts/verify_baselines.py --output runs/article-check\n```\n\nThe revision is pinned so the code stays the same when someone repeats the example. Use a fresh output directory if you run the baseline check again.\n\nThis revision has 51 test methods, including multiple cases within some tests. The baseline check should finish with:\n\n```\nVerified: 6 reference passes; all 6 targeted semantic mutations detected.\n```\n\nThe output folders retain the inputs, individual check results, and reports. Open `runs/article-check/reference/report.html` or `runs/article-check/mutated/report.html` locally to inspect them.\n\nThese checks examine configuration data. They do not submit it to a Kubernetes API server, start a workload, or verify that an application serves traffic.\n\nFor example, a matching Service selector does not prove that the application is listening on the requested port. That needs a different kind of test.\n\nThe suite is also small: six public, synthetic tasks. Its controls and regression tests cover specific requirements and failure cases. They are not evidence that every valid configuration will pass or every invalid one will fail.\n\nFor my next evaluator change, I want to answer three questions before looking at a model score:\n\nFor this project, the third question caught a problem that the original pass/fail examples had missed.", "url": "https://wpnews.pro/news/before-scoring-a-model-test-the-evaluator", "canonical_source": "https://dev.to/deavgprime/before-scoring-a-model-test-the-evaluator-3noi", "published_at": "2026-11-01 19:07:46+00:00", "updated_at": "2026-11-01 19:22:49.678236+00:00", "lang": "en", "topics": ["ai-tools", "mlops", "ai-research"], "entities": ["IaC Eval Bench", "Kubernetes", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/before-scoring-a-model-test-the-evaluator", "markdown": "https://wpnews.pro/news/before-scoring-a-model-test-the-evaluator.md", "text": "https://wpnews.pro/news/before-scoring-a-model-test-the-evaluator.txt", "jsonld": "https://wpnews.pro/news/before-scoring-a-model-test-the-evaluator.jsonld"}}