{"slug": "grade-the-ways-it-breaks", "title": "Grade the Ways It Breaks", "summary": "A developer proposes a four-file take-home contract for evaluating AI engineering candidates, arguing that green test suites and polished agent demos no longer prove a candidate understands system constraints. The proposed zip includes PROMPT.md, RUBRIC.yml, sample_solution/, and FAILURES.md, with the prompt requiring a bounded HTTP quote proxy that exits cleanly when a free model endpoint is missing and never logs secrets. The piece was prepared as part of MonkeyCode's product outreach, with the author noting the method works with any trusted free endpoint.", "body_md": "You unzip the take-home at 11:14 p.m. because the loop closes in the morning. The README is warm. The tests are green. A terminal recording even shows an agent typing like it has a pulse.\n\nThen you hunt for the autopsy. There is no file that names how the thing lies. There is no sample you can rerun on a machine that is not the candidate's laptop. There is a prompt that only finishes if someone already paid for a model.\n\nYou close the lid. You already know the onsite will be a tour of a demo, not a review of a system.\n\nThat scene is the hiring version of a fight the industry is having in public. People are shipping vibes and calling the result engineering. Models are also getting good at the tests we leave in the zip, which means a green bar is no longer evidence that anyone understood the constraint. If you are hiring someone who will design AI-assisted work, the take-home cannot be \"build a cute agent.\" It has to be a spec with teeth.\n\nHere is the contract I want in the zip. Four files, four jobs. `PROMPT.md` is what a stranger can finish without a paid key. `RUBRIC.yml` is how a second reviewer scores the same zip without a hallway chat. `sample_solution/` is a path you can replay. `FAILURES.md` is the autopsy written before the happy path, not after the candidate notices you are watching.\n\nIf any one of those is missing, you are not grading engineering. You are grading presentation.\n\nKeep the story small. You are not hiring someone to invent a platform. You are hiring someone who can bound a model, refuse to log a secret, and leave a receipt a human can read on Monday.\n\nDrop this into `PROMPT.md` and do not decorate it.\n\n```\n# Take-home: bounded quote proxy\n\nBuild a tiny HTTP service that accepts POST /v1/quote\nwith JSON {\"prompt\": \"...\"} and returns JSON:\n\n{\n  \"ok\": true,\n  \"text\": \"...\",\n  \"route\": \"free\",\n  \"bytes_out\": 0,\n  \"truncated\": false\n}\n\nConstraints (all of these are graded):\n\n1. The service must call an LLM only through $FREE_ENDPOINT.\n   If that env var is missing, exit 2 before binding the port.\n2. Never write Authorization, Cookie, or api_key values to\n   stdout, stderr, or the receipt file.\n3. Cap model output at 500 characters. If the model returns\n   more, truncate, set truncated=true, still return HTTP 200.\n4. Write receipts/last.json atomically (write temp, then rename).\n5. Include FAILURES.md that names at least five ways this\n   design lies, including one you could not fix in the timebox.\n\nTimebox: three hours. Do not add a UI. Do not add auth.\nDo not add a second model route.\n```\n\nNotice what the prompt does not do. It does not ask for a framework. It does not ask for streaming. It does not ask the candidate to impress you with tools. It asks for a door that stays shut when the free path is missing.\n\nThat last bit is the point of the current market, not a product pitch. Interviewers have been burned by take-homes that only run on a personal key. The candidate looks fluent. Your replay dies in five minutes. You cannot tell skill from a weekend of paid tokens.\n\nIf you need a replay path that does not start at a checkout form, MonkeyCode's free model access and free server option are one way to point `$FREE_ENDPOINT` at something a second reviewer can actually hit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of this article still works if you use any other free endpoint you already trust. The method is the four files, not the vendor.\n\nA rubric that lives in your head is a vibe. Put the scores in the zip so a skeptical teammate can grade without you in the room.\n\n```\n# RUBRIC.yml\nversion: 1\nmax_points: 20\ntimebox_hours: 3\nweights:\n  boot_without_endpoint: 4   # process exits 2, no port bind\n  secret_hygiene: 4          # no secret material in logs or receipt\n  truncate_and_flag: 3       # 500-char cap, truncated=true\n  atomic_receipt: 3          # temp file + rename, last.json replays\n  failures_doc: 4            # five named lies, one left unfixed\n  prompt_obedience: 2        # no UI, no second route, no extra auth\nfail_closed:\n  - paid_or_mystery_endpoint\n  - missing_FAILURES_md\n  - demo_only_no_sample_solution\nnotes: |\n  Score the sample_solution directory, not a screen recording.\n  If the candidate needs a paid key to finish, score is 0 on\n  boot_without_endpoint even if the demo looks smooth.\n```\n\nRead that file out loud once. If a sentence needs you to explain it, the sentence is still a hallway chat. Tighten it until a stranger can mark the zip in twenty minutes.\n\nLabel this as a worked example, not a production service. It is the smallest Python that makes the rubric angry in the right places when you break it on purpose.\n\n``` bash\n#!/usr/bin/env python3\n\"\"\"Worked example. Not a framework. Replay with FREE_ENDPOINT set.\"\"\"\nfrom __future__ import annotations\n\nimport json\nimport os\nimport sys\nimport tempfile\nfrom http.server import BaseHTTPRequestHandler, ThreadingHTTPServer\nfrom pathlib import Path\nfrom urllib.request import Request, urlopen\n\nCAP = 500\nRECEIPT = Path(\"receipts/last.json\")\nSECRET_KEYS = (\"authorization\", \"cookie\", \"api_key\", \"x-api-key\")\n\ndef die(code: int, msg: str) -> None:\n    sys.stderr.write(msg + \"\\n\")\n    raise SystemExit(code)\n\ndef endpoint() -> str:\n    url = os.environ.get(\"FREE_ENDPOINT\", \"\").strip()\n    if not url:\n        die(2, \"FREE_ENDPOINT missing; refusing to bind\")\n    return url\n\ndef redact(obj):\n    if isinstance(obj, dict):\n        out = {}\n        for k, v in obj.items():\n            if str(k).lower() in SECRET_KEYS:\n                out[k] = \"[redacted]\"\n            else:\n                out[k] = redact(v)\n        return out\n    if isinstance(obj, list):\n        return [redact(x) for x in obj]\n    return obj\n\ndef write_receipt(payload: dict) -> None:\n    RECEIPT.parent.mkdir(parents=True, exist_ok=True)\n    data = json.dumps(redact(payload), indent=2).encode()\n    fd, tmp = tempfile.mkstemp(dir=str(RECEIPT.parent), suffix=\".tmp\")\n    try:\n        os.write(fd, data)\n        os.fsync(fd)\n    finally:\n        os.close(fd)\n    os.replace(tmp, RECEIPT)\n\ndef call_model(prompt: str) -> str:\n    body = json.dumps({\"prompt\": prompt}).encode()\n    req = Request(endpoint(), data=body, method=\"POST\")\n    req.add_header(\"Content-Type\", \"application/json\")\n    with urlopen(req, timeout=30) as resp:\n        raw = json.loads(resp.read().decode() or \"{}\")\n    text = str(raw.get(\"text\") or raw.get(\"content\") or \"\")\n    return text\n\nclass Handler(BaseHTTPRequestHandler):\n    def log_message(self, fmt: str, *args) -> None:\n        # Keep the default logger from echoing headers.\n        sys.stderr.write(\"%s - %s\\n\" % (self.address_string(), fmt % args))\n\n    def do_POST(self) -> None:\n        if self.path != \"/v1/quote\":\n            self.send_error(404)\n            return\n        n = int(self.headers.get(\"Content-Length\") or 0)\n        incoming = json.loads(self.rfile.read(n) or b\"{}\")\n        prompt = str(incoming.get(\"prompt\") or \"\")\n        text = call_model(prompt)\n        truncated = len(text) > CAP\n        if truncated:\n            text = text[:CAP]\n        payload = {\n            \"ok\": True,\n            \"text\": text,\n            \"route\": \"free\",\n            \"bytes_out\": len(text.encode()),\n            \"truncated\": truncated,\n        }\n        write_receipt(payload)\n        blob = json.dumps(payload).encode()\n        self.send_response(200)\n        self.send_header(\"Content-Type\", \"application/json\")\n        self.send_header(\"Content-Length\", str(len(blob)))\n        self.end_headers()\n        self.wfile.write(blob)\n\nif __name__ == \"__main__\":\n    url = endpoint()\n    server = ThreadingHTTPServer((\"127.0.0.1\", 8088), Handler)\n    sys.stderr.write(\"listening on 8088 via %s\\n\" % url.split(\"?\", 1)[0])\n    server.serve_forever()\n```\n\nReplay it like a skeptic, not like a fan. The commands below are the interview. If they only work on the candidate's machine, the zip is a souvenir.\n\n```\nunset FREE_ENDPOINT\npython3 sample_solution/server.py; echo exit:$?\n# expect: exit 2, nothing listening on 8088\n\nexport FREE_ENDPOINT=\"http://127.0.0.1:9/does-not-matter\"\npython3 - <<'PY'\nimport json, os, urllib.request\nprint(\"endpoint configured:\", bool(os.environ.get(\"FREE_ENDPOINT\")))\nPY\n\n# With a real free endpoint in FREE_ENDPOINT:\ncurl -sS localhost:8088/v1/quote \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"prompt\":\"Reply with 800 characters of the letter a.\"}'\ncat receipts/last.json\n```\n\nThe first command is the whole philosophy. Missing env, no bind, non-zero exit. A candidate who \"fixes\" that by silently falling back to a paid key has failed the prompt even if the JSON looks pretty.\n\nAsk them to write `FAILURES.md` before they chase green. People who start with the demo will invent failure modes that their code already avoids. That is fan fiction. You want the lies that are still in the design.\n\nA strong autopsy reads like this, in prose, not as a trophy list.\n\nThe proxy trusts whatever JSON the free endpoint returns. If that server wraps the model text in a different key, `text` becomes empty and `ok` stays true. The receipt then looks healthy while the caller got silence. Truncation is counted in characters, not tokens and not graphemes, so a prompt that returns emoji or CJK will pass the 500 cap and still blow a downstream UI. `urlopen` follows redirects. A compromised endpoint can 302 the prompt at a host that is not free, and the receipt will still say `\"route\": \"free\"` because that string is a label, not a measurement. ThreadingHTTPServer will overlap two writes if you skip the temp-file rename under load; last.json can tear. Logging `self.address_string()` is harmless until someone puts a bearer token in a query string, at which point stderr becomes a secret store. None of those are clever. All of them show up in real take-homes when the model is treated like a clean function.\n\nThe line you should not forgive is the unfixed one. If `FAILURES.md` claims the design is complete, the candidate did not look. Engineering is the leftover risk, named in a file a stranger can read.\n\nYou will see the same collapses. The zip contains a screen recording and no `sample_solution/`. The prompt grew a second route named `premium` because the candidate wanted a better answer. `FAILURES.md` is a paste of generic LLM risks that never mention this HTTP handler. The receipt dumps the full request headers. The server binds even when `FREE_ENDPOINT` is empty, then fails later in a stack trace. Tests assert only that HTTP 200 happened, which is how a model outgrows the suite: it learned the shape of your asserts and stopped touching the constraint.\n\nTreat those as automatic zeros on the matching rubric keys. Do not debate them in the onsite. The onsite is for the one leftover risk they could not fix. Ask them to walk the redirect lie, or the character-versus-token lie, with the receipt file open. If they cannot, they shipped a vibe.\n\nDo not send this packet for a staff role whose job is system design across six services. Three hours and a quote proxy will insult them, and they will be right. Do not send it if nobody on your side will rerun the sample. A rubric you never execute is costume jewelry. Do not send it to candidates you cannot pay for the timebox. A take-home is still labor.\n\nSkip it if your company cannot offer any free replay path. The whole point is that the interviewer and the candidate share a machine story. If the only model you believe in is a paid one, this prompt becomes theater, and you should interview some other way.\n\nThe method also fails closed for people who want a cinematic agent demo. That is the feature. You are not grading whether the model sounded clever. You are grading whether the candidate can name the ways it breaks, bound the blast radius, and leave four files a stranger can grade after midnight.\n\nIf you try the packet, change the prompt's domain so it is not this quote proxy. Keep the four names. Keep the fail-closed boot. Keep the autopsy. That is the work, and it still holds if you strip every product name out of the README.", "url": "https://wpnews.pro/news/grade-the-ways-it-breaks", "canonical_source": "https://dev.to/devio_4040/grade-the-ways-it-breaks-d0a", "published_at": "2026-09-18 07:02:12+00:00", "updated_at": "2026-09-18 07:22:51.256416+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools", "ai-products"], "entities": ["MonkeyCode"], "alternates": {"html": "https://wpnews.pro/news/grade-the-ways-it-breaks", "markdown": "https://wpnews.pro/news/grade-the-ways-it-breaks.md", "text": "https://wpnews.pro/news/grade-the-ways-it-breaks.txt", "jsonld": "https://wpnews.pro/news/grade-the-ways-it-breaks.jsonld"}}