Grade the Ways It Breaks

A developer proposes a four-file take-home contract for evaluating AI engineering candidates, arguing that green test suites and polished agent demos no longer prove a candidate understands system constraints. The proposed zip includes PROMPT.md, RUBRIC.yml, sample_solution/, and FAILURES.md, with the prompt requiring a bounded HTTP quote proxy that exits cleanly when a free model endpoint is missing and never logs secrets. The piece was prepared as part of MonkeyCode's product outreach, with the author noting the method works with any trusted free endpoint.

You unzip the take-home at 11:14 p.m. because the loop closes in the morning. The README is warm. The tests are green. A terminal recording even shows an agent typing like it has a pulse. Then you hunt for the autopsy. There is no file that names how the thing lies. There is no sample you can rerun on a machine that is not the candidate's laptop. There is a prompt that only finishes if someone already paid for a model. You close the lid. You already know the onsite will be a tour of a demo, not a review of a system. That scene is the hiring version of a fight the industry is having in public. People are shipping vibes and calling the result engineering. Models are also getting good at the tests we leave in the zip, which means a green bar is no longer evidence that anyone understood the constraint. If you are hiring someone who will design AI-assisted work, the take-home cannot be "build a cute agent." It has to be a spec with teeth. Here is the contract I want in the zip. Four files, four jobs. PROMPT.md is what a stranger can finish without a paid key. RUBRIC.yml is how a second reviewer scores the same zip without a hallway chat. sample solution/ is a path you can replay. FAILURES.md is the autopsy written before the happy path, not after the candidate notices you are watching. If any one of those is missing, you are not grading engineering. You are grading presentation. Keep the story small. You are not hiring someone to invent a platform. You are hiring someone who can bound a model, refuse to log a secret, and leave a receipt a human can read on Monday. Drop this into PROMPT.md and do not decorate it. Take-home: bounded quote proxy Build a tiny HTTP service that accepts POST /v1/quote with JSON {"prompt": "..."} and returns JSON: { "ok": true, "text": "...", "route": "free", "bytes out": 0, "truncated": false } Constraints all of these are graded : 1. The service must call an LLM only through $FREE ENDPOINT. If that env var is missing, exit 2 before binding the port. 2. Never write Authorization, Cookie, or api key values to stdout, stderr, or the receipt file. 3. Cap model output at 500 characters. If the model returns more, truncate, set truncated=true, still return HTTP 200. 4. Write receipts/last.json atomically write temp, then rename . 5. Include FAILURES.md that names at least five ways this design lies, including one you could not fix in the timebox. Timebox: three hours. Do not add a UI. Do not add auth. Do not add a second model route. Notice what the prompt does not do. It does not ask for a framework. It does not ask for streaming. It does not ask the candidate to impress you with tools. It asks for a door that stays shut when the free path is missing. That last bit is the point of the current market, not a product pitch. Interviewers have been burned by take-homes that only run on a personal key. The candidate looks fluent. Your replay dies in five minutes. You cannot tell skill from a weekend of paid tokens. If you need a replay path that does not start at a checkout form, MonkeyCode's free model access and free server option are one way to point $FREE ENDPOINT at something a second reviewer can actually hit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of this article still works if you use any other free endpoint you already trust. The method is the four files, not the vendor. A rubric that lives in your head is a vibe. Put the scores in the zip so a skeptical teammate can grade without you in the room. RUBRIC.yml version: 1 max points: 20 timebox hours: 3 weights: boot without endpoint: 4 process exits 2, no port bind secret hygiene: 4 no secret material in logs or receipt truncate and flag: 3 500-char cap, truncated=true atomic receipt: 3 temp file + rename, last.json replays failures doc: 4 five named lies, one left unfixed prompt obedience: 2 no UI, no second route, no extra auth fail closed: - paid or mystery endpoint - missing FAILURES md - demo only no sample solution notes: | Score the sample solution directory, not a screen recording. If the candidate needs a paid key to finish, score is 0 on boot without endpoint even if the demo looks smooth. Read that file out loud once. If a sentence needs you to explain it, the sentence is still a hallway chat. Tighten it until a stranger can mark the zip in twenty minutes. Label this as a worked example, not a production service. It is the smallest Python that makes the rubric angry in the right places when you break it on purpose. bash /usr/bin/env python3 """Worked example. Not a framework. Replay with FREE ENDPOINT set.""" from future import annotations import json import os import sys import tempfile from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from urllib.request import Request, urlopen CAP = 500 RECEIPT = Path "receipts/last.json" SECRET KEYS = "authorization", "cookie", "api key", "x-api-key" def die code: int, msg: str - None: sys.stderr.write msg + "\n" raise SystemExit code def endpoint - str: url = os.environ.get "FREE ENDPOINT", "" .strip if not url: die 2, "FREE ENDPOINT missing; refusing to bind" return url def redact obj : if isinstance obj, dict : out = {} for k, v in obj.items : if str k .lower in SECRET KEYS: out k = " redacted " else: out k = redact v return out if isinstance obj, list : return redact x for x in obj return obj def write receipt payload: dict - None: RECEIPT.parent.mkdir parents=True, exist ok=True data = json.dumps redact payload , indent=2 .encode fd, tmp = tempfile.mkstemp dir=str RECEIPT.parent , suffix=".tmp" try: os.write fd, data os.fsync fd finally: os.close fd os.replace tmp, RECEIPT def call model prompt: str - str: body = json.dumps {"prompt": prompt} .encode req = Request endpoint , data=body, method="POST" req.add header "Content-Type", "application/json" with urlopen req, timeout=30 as resp: raw = json.loads resp.read .decode or "{}" text = str raw.get "text" or raw.get "content" or "" return text class Handler BaseHTTPRequestHandler : def log message self, fmt: str, args - None: Keep the default logger from echoing headers. sys.stderr.write "%s - %s\n" % self.address string , fmt % args def do POST self - None: if self.path = "/v1/quote": self.send error 404 return n = int self.headers.get "Content-Length" or 0 incoming = json.loads self.rfile.read n or b"{}" prompt = str incoming.get "prompt" or "" text = call model prompt truncated = len text CAP if truncated: text = text :CAP payload = { "ok": True, "text": text, "route": "free", "bytes out": len text.encode , "truncated": truncated, } write receipt payload blob = json.dumps payload .encode self.send response 200 self.send header "Content-Type", "application/json" self.send header "Content-Length", str len blob self.end headers self.wfile.write blob if name == " main ": url = endpoint server = ThreadingHTTPServer "127.0.0.1", 8088 , Handler sys.stderr.write "listening on 8088 via %s\n" % url.split "?", 1 0 server.serve forever Replay it like a skeptic, not like a fan. The commands below are the interview. If they only work on the candidate's machine, the zip is a souvenir. unset FREE ENDPOINT python3 sample solution/server.py; echo exit:$? expect: exit 2, nothing listening on 8088 export FREE ENDPOINT="http://127.0.0.1:9/does-not-matter" python3 - <<'PY' import json, os, urllib.request print "endpoint configured:", bool os.environ.get "FREE ENDPOINT" PY With a real free endpoint in FREE ENDPOINT: curl -sS localhost:8088/v1/quote \ -H 'Content-Type: application/json' \ -d '{"prompt":"Reply with 800 characters of the letter a."}' cat receipts/last.json The first command is the whole philosophy. Missing env, no bind, non-zero exit. A candidate who "fixes" that by silently falling back to a paid key has failed the prompt even if the JSON looks pretty. Ask them to write FAILURES.md before they chase green. People who start with the demo will invent failure modes that their code already avoids. That is fan fiction. You want the lies that are still in the design. A strong autopsy reads like this, in prose, not as a trophy list. The proxy trusts whatever JSON the free endpoint returns. If that server wraps the model text in a different key, text becomes empty and ok stays true. The receipt then looks healthy while the caller got silence. Truncation is counted in characters, not tokens and not graphemes, so a prompt that returns emoji or CJK will pass the 500 cap and still blow a downstream UI. urlopen follows redirects. A compromised endpoint can 302 the prompt at a host that is not free, and the receipt will still say "route": "free" because that string is a label, not a measurement. ThreadingHTTPServer will overlap two writes if you skip the temp-file rename under load; last.json can tear. Logging self.address string is harmless until someone puts a bearer token in a query string, at which point stderr becomes a secret store. None of those are clever. All of them show up in real take-homes when the model is treated like a clean function. The line you should not forgive is the unfixed one. If FAILURES.md claims the design is complete, the candidate did not look. Engineering is the leftover risk, named in a file a stranger can read. You will see the same collapses. The zip contains a screen recording and no sample solution/ . The prompt grew a second route named premium because the candidate wanted a better answer. FAILURES.md is a paste of generic LLM risks that never mention this HTTP handler. The receipt dumps the full request headers. The server binds even when FREE ENDPOINT is empty, then fails later in a stack trace. Tests assert only that HTTP 200 happened, which is how a model outgrows the suite: it learned the shape of your asserts and stopped touching the constraint. Treat those as automatic zeros on the matching rubric keys. Do not debate them in the onsite. The onsite is for the one leftover risk they could not fix. Ask them to walk the redirect lie, or the character-versus-token lie, with the receipt file open. If they cannot, they shipped a vibe. Do not send this packet for a staff role whose job is system design across six services. Three hours and a quote proxy will insult them, and they will be right. Do not send it if nobody on your side will rerun the sample. A rubric you never execute is costume jewelry. Do not send it to candidates you cannot pay for the timebox. A take-home is still labor. Skip it if your company cannot offer any free replay path. The whole point is that the interviewer and the candidate share a machine story. If the only model you believe in is a paid one, this prompt becomes theater, and you should interview some other way. The method also fails closed for people who want a cinematic agent demo. That is the feature. You are not grading whether the model sounded clever. You are grading whether the candidate can name the ways it breaks, bound the blast radius, and leave four files a stranger can grade after midnight. If you try the packet, change the prompt's domain so it is not this quote proxy. Keep the four names. Keep the fail-closed boot. Keep the autopsy. That is the work, and it still holds if you strip every product name out of the README.