# Build a Read-Only Eval Slice Before Giving Free Inference Write Authority > Source: > Published: 2026-09-20 10:35:22+00:00 # Build a Read-Only Eval Slice Before Giving Free Inference Write Authority Last Tuesday a teammate wired "draft a reply" onto the ticket page because inference had become free overnight. A support lead clicked that button on a live customer thread, watched the spinner stall, and then saw the draft publish. Was the model actually wrong, or did we skip the layer that should Last Tuesday a teammate wired "draft a reply" onto the ticket page because inference had become free overnight. A support lead clicked that button on a live customer thread, watched the spinner stall, and then saw the draft publish. Was the model actually wrong, or did we skip the layer that should have kept that draft private? The first failure was not generation quality; it was a write path that treated a guess like a commit. I keep arguing the same unpopular position on these prototypes, and I will argue it again here. Free inference is a gift for evaluation loops, but it is not permission to skip handoff tests. If your feature can mutate a ticket, an invoice note, or a user record, the model is not the product. The product is the contract between the button click, the provider, the store, and the identity of the actor. Teams still celebrate a cheap completion the way they once celebrated a green unit test on a mapper. They paste a client call into a route handler, render the string, and call the vertical slice done. Have you noticed how often that slice dies at the first timeout, the first empty body, or the first retry? A contended rehearsal server will show you those failures earlier, which is useful only if you actually look. So I refuse to give a completion write authority until a read-only eval slice can fail in public. The slice uses the same provider protocol the production route will use, minus the database mutation. Think of it like a dress rehearsal with the lighting board live and the curtain still down. You want the lights to flicker now, not during the matinee when customers are already in their seats. When I need a cheap place to run that rehearsal, I point the same protocol at MonkeyCode's free model access and free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. That free path is enough if you want to drive this harness, and I still refuse to treat it as launch capacity. The application pins an envelope, a timeout, and a failure vocabulary that any later provider still has to speak. Here is the working path I actually implement, labeled as a local pattern you can run without my private traffic. Start from the user action, which is draft a reply, and stop at the first layer that can lie. In this feature that layer is usually the HTTP client pretending a hung socket is an empty string. I would rather publish this boring contract than another screenshot of a model sounding confident. # eval_contract.py from __future__ import annotations from dataclasses import dataclass from typing import Protocol @dataclass(frozen=True) class CompletionEnvelope: request_id: str tenant_id: str actor_id: str purpose: str input_sha256: str timeout_ms: int @dataclass(frozen=True) class CompletionResult: request_id: str status: str # ok | empty | timeout | unauthorized | unavailable text: str http_status: int class CompletionProvider(Protocol): def complete(self, envelope: CompletionEnvelope, prompt: str) -> CompletionResult: """Return a typed failure, never a raised surprise, for eval and apply.""" ... Why freeze the envelope instead of letting each caller invent headers like a souvenir shop? Because retries, eval reruns, and later provider swaps need a stable identity for the same user action. If request_id and input_sha256 drift, your read-only eval is just another vibe-coded chat window. Would you accept a payment intent that quietly changed its amount between the authorize call and the capture call? The eval runner loads a fixture, calls the provider, and records status without touching storage. It has no repository handle, which is the entire point of the type boundary. I want a sneaky insert to look ridiculous in review, not clever after an incident. Boring is how you stop a free prototype from becoming an accidental publisher. # eval_runner.py from hashlib import sha256 from uuid import uuid4 from eval_contract import CompletionEnvelope, CompletionProvider, CompletionResult def build_envelope(tenant_id: str, actor_id: str, ticket_body: str) -> CompletionEnvelope: digest = sha256(ticket_body.encode("utf-8")).hexdigest() return CompletionEnvelope( request_id=str(uuid4()), tenant_id=tenant_id, actor_id=actor_id, purpose="ticket_reply_draft", input_sha256=digest, timeout_ms=8000, ) def run_read_only_eval(provider: CompletionProvider, ticket_body: str) -> CompletionResult: envelope = build_envelope("tenant_acme", "actor_support", ticket_body) prompt = f"Draft a support reply. Do not invent policy. Ticket:\n{ticket_body}" result = provider.complete(envelope, prompt) # No session.commit(). No ticket.messages.append(). Eval ends here. return result If a colleague asks why this looks like a lot of ceremony for a text box, ask them where the note goes after a 504. Does the UI keep a ghost paragraph? Does the worker retry and post twice? The interesting work is the failure table you run against this runner before anyone wires INSERT. I also want an HTTP adapter that cannot raise its way past the eval door. Timeouts, transport errors, and empty JSON bodies should become named statuses the apply gate already understands. This adapter is a proposed local seam, not a vendor SDK tour, and it should stay replaceable without rewriting the ticket page. # http_provider.py import httpx from eval_contract import CompletionEnvelope, CompletionResult class HttpCompletionProvider: def __init__(self, base_url: str, token: str, timeout_s: float = 8.0): self.base_url = base_url.rstrip("/") self.token = token self.timeout_s = timeout_s def complete(self, envelope: CompletionEnvelope, prompt: str) -> CompletionResult: try: response = httpx.post( f"{self.base_url}/v1/complete", headers={"Authorization": f"Bearer {self.token}"}, json={ "request_id": envelope.request_id, "purpose": envelope.purpose, "prompt": prompt, }, timeout=self.timeout_s, ) except httpx.TimeoutException: return CompletionResult(envelope.request_id, "timeout", "", 504) except httpx.TransportError: return CompletionResult(envelope.request_id, "unavailable", "", 503) if response.status_code in (401, 403): return CompletionResult( envelope.request_id, "unauthorized", "", response.status_code ) if response.status_code >= 500: return CompletionResult( envelope.request_id, "unavailable", "", response.status_code ) text = "" content_type = response.headers.get("content-type", "") if content_type.startswith("application/json"): payload = response.json() text = str(payload.get("text", "")) if not str(text).strip(): return CompletionResult(envelope.request_id, "empty", text, response.status_code) return CompletionResult(envelope.request_id, "ok", text, response.status_code) Point that class at whatever rehearsal base URL you already have, then keep the UI on the eval route until the tests below are green. Do not let the browser supply purpose, tenant_id, or timeout_ms, because those fields are authorization in slow motion. A stolen token with a writable purpose is not a model problem, and swapping vendors will not fix it. I do not score eloquence in this slice, because pretty prose is the easiest lie in the stack. I score whether each layer tells the truth when the world gets ugly, which is the part demos skip. The tests below are meant to be executed locally; they are not architecture theater. If your only test is that the model sounded pretty, you are measuring the wrong instrument. # test_eval_handoffs.py from eval_contract import CompletionEnvelope, CompletionResult from eval_runner import run_read_only_eval class FakeProvider: def __init__(self, result: CompletionResult): self.result = result self.seen = None def complete(self, envelope: CompletionEnvelope, prompt: str) -> CompletionResult: self.seen = envelope return self.result def test_timeout_does_not_look_like_success(): provider = FakeProvider(CompletionResult("r1", "timeout", "", 504)) result = run_read_only_eval(provider, "Refund arrived twice.") assert result.status == "timeout" assert result.text == "" assert result.http_status == 504 def test_empty_completion_is_a_named_failure(): provider = FakeProvider(CompletionResult("r2", "empty", " ", 200)) result = run_read_only_eval(provider, "Need VAT invoice.") assert result.status == "empty" def test_unauthorized_stops_before_any_apply_path(): provider = FakeProvider(CompletionResult("r3", "unauthorized", "", 401)) result = run_read_only_eval(provider, "Reset mailbox access.") assert result.status == "unauthorized" assert result.http_status == 401 def test_envelope_carries_tenant_and_purpose(): provider = FakeProvider(CompletionResult("r4", "ok", "We can help.", 200)) run_read_only_eval(provider, "Cannot login after SSO change.") assert provider.seen.tenant_id == "tenant_acme" assert provider.seen.purpose == "ticket_reply_draft" assert len(provider.seen.input_sha256) == 64 Run the file with a command you can paste into CI without decorating it for a blog screenshot. python -m pytest test_eval_handoffs.py test_apply_gate.py -q Notice what is missing on purpose. There is no leaderboard, no BLEU footnote, and no chat transcript taped to the pull request. Those artifacts do not tell you whether a 504 became a blank note on a customer timeline. Can a model-card screenshot explain a duplicate send after a retry? I have never seen one that could. Only after the eval slice is green do I add an apply function, and it demands an idempotency key from the envelope. A rehearsal server that retries behind your back is not evil; an apply path that inserts twice is. The apply door stays locked unless status is ok and a reviewer, or a later automated gate, sets approved=True. That second door is the whole opinion: generation is cheap, mutation is not. # apply_draft.py from eval_contract import CompletionEnvelope, CompletionResult class ApplyRejected(Exception): pass def apply_draft(envelope, result, approved, insert_note) -> str: if result.status != "ok" or not result.text.strip(): raise ApplyRejected(f"refusing apply for status={result.status}") if not approved: raise ApplyRejected("eval passed, write authority not granted") if envelope.purpose != "ticket_reply_draft": raise ApplyRejected("purpose mismatch") return insert_note( tenant_id=envelope.tenant_id, actor_id=envelope.actor_id, idempotency_key=envelope.request_id, body=result.text.strip(), ) The insert_note collaborator should hit a unique index on tenant_id plus idempotency_key, not a hopeful comment in the handler. I have watched retry storms from polite clients double-post because the route was probably fine. Was it fine? The second email to the customer answered that question faster than any dashboard. A tiny persistence test belongs here too, because eval-green and apply-broken is how vibe-coded features ship. If you only test the model client, you will launch a very eloquent way to corrupt a thread. # test_apply_gate.py import pytest from apply_draft import ApplyRejected, apply_draft from eval_contract import CompletionEnvelope, CompletionResult def envelope(): return CompletionEnvelope("req-9", "t1", "u1", "ticket_reply_draft", "a" * 64, 8000) def test_ok_but_unapproved_does_not_write(): calls = [] result = CompletionResult("req-9", "ok", "Thanks, we are looking.", 200) with pytest.raises(ApplyRejected): apply_draft( envelope(), result, approved=False, insert_note=lambda **k: calls.append(k), ) assert calls == [] def test_timeout_never_reaches_insert(): calls = [] result = CompletionResult("req-9", "timeout", "", 504) with pytest.raises(ApplyRejected): apply_draft( envelope(), result, approved=True, insert_note=lambda **k: calls.append(k), ) assert calls == [] This pattern will not make a weak draft sound like a senior agent, and it will not turn a rehearsal server into a capacity plan. It also will not replace authentication, tenant isolation, or retention rules you already owe the rest of the app. If you ship healthcare advice, money movement, or anything that must be right the first time, do not use a read-only eval as a substitute for a human in the loop. I would not use this approach on a throwaway weekend toy that never stores state, because the ceremony is heavier than the risk. I also would not let a marketing site call the apply door from the browser with a user-supplied purpose string. The envelope is a server-side object, not a souvenir from the client. Production caveats still apply when the eval backend costs nothing at the register: timeouts bunch up, empty completions arrive with HTTP 200, and copied curl headers rot in silence. Reuse this order before you celebrate a free path. Pin the envelope. Run the read-only eval. Prove timeout, empty, and 401. Lock apply behind approval plus idempotency. Only then aim a real UI at the write door. Which layer handoff is least stable in your stack right now, and what concrete failure state or response code does it return when the draft should never have been stored? ## Key Takeaways - •Last Tuesday a teammate wired "draft a reply" onto the ticket page because inference had become free overnight - •This story was reported by **Dev.to** , covering developments in the**dev** space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: [Read Full Article on Dev.to →](https://dev.to/kongkong1/build-a-read-only-eval-slice-before-giving-free-inference-write-authority-2895)