# Grade the Ways It Breaks > Source: > Published: 2026-09-18 07:02:12+00:00 You unzip the take-home at 11:14 p.m. because the loop closes in the morning. The README is warm. The tests are green. A terminal recording even shows an agent typing like it has a pulse. Then you hunt for the autopsy. There is no file that names how the thing lies. There is no sample you can rerun on a machine that is not the candidate's laptop. There is a prompt that only finishes if someone already paid for a model. You close the lid. You already know the onsite will be a tour of a demo, not a review of a system. That scene is the hiring version of a fight the industry is having in public. People are shipping vibes and calling the result engineering. Models are also getting good at the tests we leave in the zip, which means a green bar is no longer evidence that anyone understood the constraint. If you are hiring someone who will design AI-assisted work, the take-home cannot be "build a cute agent." It has to be a spec with teeth. Here is the contract I want in the zip. Four files, four jobs. `PROMPT.md` is what a stranger can finish without a paid key. `RUBRIC.yml` is how a second reviewer scores the same zip without a hallway chat. `sample_solution/` is a path you can replay. `FAILURES.md` is the autopsy written before the happy path, not after the candidate notices you are watching. If any one of those is missing, you are not grading engineering. You are grading presentation. Keep the story small. You are not hiring someone to invent a platform. You are hiring someone who can bound a model, refuse to log a secret, and leave a receipt a human can read on Monday. Drop this into `PROMPT.md` and do not decorate it. ``` # Take-home: bounded quote proxy Build a tiny HTTP service that accepts POST /v1/quote with JSON {"prompt": "..."} and returns JSON: { "ok": true, "text": "...", "route": "free", "bytes_out": 0, "truncated": false } Constraints (all of these are graded): 1. The service must call an LLM only through $FREE_ENDPOINT. If that env var is missing, exit 2 before binding the port. 2. Never write Authorization, Cookie, or api_key values to stdout, stderr, or the receipt file. 3. Cap model output at 500 characters. If the model returns more, truncate, set truncated=true, still return HTTP 200. 4. Write receipts/last.json atomically (write temp, then rename). 5. Include FAILURES.md that names at least five ways this design lies, including one you could not fix in the timebox. Timebox: three hours. Do not add a UI. Do not add auth. Do not add a second model route. ``` Notice what the prompt does not do. It does not ask for a framework. It does not ask for streaming. It does not ask the candidate to impress you with tools. It asks for a door that stays shut when the free path is missing. That last bit is the point of the current market, not a product pitch. Interviewers have been burned by take-homes that only run on a personal key. The candidate looks fluent. Your replay dies in five minutes. You cannot tell skill from a weekend of paid tokens. If you need a replay path that does not start at a checkout form, MonkeyCode's free model access and free server option are one way to point `$FREE_ENDPOINT` at something a second reviewer can actually hit. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The rest of this article still works if you use any other free endpoint you already trust. The method is the four files, not the vendor. A rubric that lives in your head is a vibe. Put the scores in the zip so a skeptical teammate can grade without you in the room. ``` # RUBRIC.yml version: 1 max_points: 20 timebox_hours: 3 weights: boot_without_endpoint: 4 # process exits 2, no port bind secret_hygiene: 4 # no secret material in logs or receipt truncate_and_flag: 3 # 500-char cap, truncated=true atomic_receipt: 3 # temp file + rename, last.json replays failures_doc: 4 # five named lies, one left unfixed prompt_obedience: 2 # no UI, no second route, no extra auth fail_closed: - paid_or_mystery_endpoint - missing_FAILURES_md - demo_only_no_sample_solution notes: | Score the sample_solution directory, not a screen recording. If the candidate needs a paid key to finish, score is 0 on boot_without_endpoint even if the demo looks smooth. ``` Read that file out loud once. If a sentence needs you to explain it, the sentence is still a hallway chat. Tighten it until a stranger can mark the zip in twenty minutes. Label this as a worked example, not a production service. It is the smallest Python that makes the rubric angry in the right places when you break it on purpose. ``` bash #!/usr/bin/env python3 """Worked example. Not a framework. Replay with FREE_ENDPOINT set.""" from __future__ import annotations import json import os import sys import tempfile from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path from urllib.request import Request, urlopen CAP = 500 RECEIPT = Path("receipts/last.json") SECRET_KEYS = ("authorization", "cookie", "api_key", "x-api-key") def die(code: int, msg: str) -> None: sys.stderr.write(msg + "\n") raise SystemExit(code) def endpoint() -> str: url = os.environ.get("FREE_ENDPOINT", "").strip() if not url: die(2, "FREE_ENDPOINT missing; refusing to bind") return url def redact(obj): if isinstance(obj, dict): out = {} for k, v in obj.items(): if str(k).lower() in SECRET_KEYS: out[k] = "[redacted]" else: out[k] = redact(v) return out if isinstance(obj, list): return [redact(x) for x in obj] return obj def write_receipt(payload: dict) -> None: RECEIPT.parent.mkdir(parents=True, exist_ok=True) data = json.dumps(redact(payload), indent=2).encode() fd, tmp = tempfile.mkstemp(dir=str(RECEIPT.parent), suffix=".tmp") try: os.write(fd, data) os.fsync(fd) finally: os.close(fd) os.replace(tmp, RECEIPT) def call_model(prompt: str) -> str: body = json.dumps({"prompt": prompt}).encode() req = Request(endpoint(), data=body, method="POST") req.add_header("Content-Type", "application/json") with urlopen(req, timeout=30) as resp: raw = json.loads(resp.read().decode() or "{}") text = str(raw.get("text") or raw.get("content") or "") return text class Handler(BaseHTTPRequestHandler): def log_message(self, fmt: str, *args) -> None: # Keep the default logger from echoing headers. sys.stderr.write("%s - %s\n" % (self.address_string(), fmt % args)) def do_POST(self) -> None: if self.path != "/v1/quote": self.send_error(404) return n = int(self.headers.get("Content-Length") or 0) incoming = json.loads(self.rfile.read(n) or b"{}") prompt = str(incoming.get("prompt") or "") text = call_model(prompt) truncated = len(text) > CAP if truncated: text = text[:CAP] payload = { "ok": True, "text": text, "route": "free", "bytes_out": len(text.encode()), "truncated": truncated, } write_receipt(payload) blob = json.dumps(payload).encode() self.send_response(200) self.send_header("Content-Type", "application/json") self.send_header("Content-Length", str(len(blob))) self.end_headers() self.wfile.write(blob) if __name__ == "__main__": url = endpoint() server = ThreadingHTTPServer(("127.0.0.1", 8088), Handler) sys.stderr.write("listening on 8088 via %s\n" % url.split("?", 1)[0]) server.serve_forever() ``` Replay it like a skeptic, not like a fan. The commands below are the interview. If they only work on the candidate's machine, the zip is a souvenir. ``` unset FREE_ENDPOINT python3 sample_solution/server.py; echo exit:$? # expect: exit 2, nothing listening on 8088 export FREE_ENDPOINT="http://127.0.0.1:9/does-not-matter" python3 - <<'PY' import json, os, urllib.request print("endpoint configured:", bool(os.environ.get("FREE_ENDPOINT"))) PY # With a real free endpoint in FREE_ENDPOINT: curl -sS localhost:8088/v1/quote \ -H 'Content-Type: application/json' \ -d '{"prompt":"Reply with 800 characters of the letter a."}' cat receipts/last.json ``` The first command is the whole philosophy. Missing env, no bind, non-zero exit. A candidate who "fixes" that by silently falling back to a paid key has failed the prompt even if the JSON looks pretty. Ask them to write `FAILURES.md` before they chase green. People who start with the demo will invent failure modes that their code already avoids. That is fan fiction. You want the lies that are still in the design. A strong autopsy reads like this, in prose, not as a trophy list. The proxy trusts whatever JSON the free endpoint returns. If that server wraps the model text in a different key, `text` becomes empty and `ok` stays true. The receipt then looks healthy while the caller got silence. Truncation is counted in characters, not tokens and not graphemes, so a prompt that returns emoji or CJK will pass the 500 cap and still blow a downstream UI. `urlopen` follows redirects. A compromised endpoint can 302 the prompt at a host that is not free, and the receipt will still say `"route": "free"` because that string is a label, not a measurement. ThreadingHTTPServer will overlap two writes if you skip the temp-file rename under load; last.json can tear. Logging `self.address_string()` is harmless until someone puts a bearer token in a query string, at which point stderr becomes a secret store. None of those are clever. All of them show up in real take-homes when the model is treated like a clean function. The line you should not forgive is the unfixed one. If `FAILURES.md` claims the design is complete, the candidate did not look. Engineering is the leftover risk, named in a file a stranger can read. You will see the same collapses. The zip contains a screen recording and no `sample_solution/`. The prompt grew a second route named `premium` because the candidate wanted a better answer. `FAILURES.md` is a paste of generic LLM risks that never mention this HTTP handler. The receipt dumps the full request headers. The server binds even when `FREE_ENDPOINT` is empty, then fails later in a stack trace. Tests assert only that HTTP 200 happened, which is how a model outgrows the suite: it learned the shape of your asserts and stopped touching the constraint. Treat those as automatic zeros on the matching rubric keys. Do not debate them in the onsite. The onsite is for the one leftover risk they could not fix. Ask them to walk the redirect lie, or the character-versus-token lie, with the receipt file open. If they cannot, they shipped a vibe. Do not send this packet for a staff role whose job is system design across six services. Three hours and a quote proxy will insult them, and they will be right. Do not send it if nobody on your side will rerun the sample. A rubric you never execute is costume jewelry. Do not send it to candidates you cannot pay for the timebox. A take-home is still labor. Skip it if your company cannot offer any free replay path. The whole point is that the interviewer and the candidate share a machine story. If the only model you believe in is a paid one, this prompt becomes theater, and you should interview some other way. The method also fails closed for people who want a cinematic agent demo. That is the feature. You are not grading whether the model sounded clever. You are grading whether the candidate can name the ways it breaks, bound the blast radius, and leave four files a stranger can grade after midnight. If you try the packet, change the prompt's domain so it is not this quote proxy. Keep the four names. Keep the fail-closed boot. Keep the autopsy. That is the work, and it still holds if you strip every product name out of the README.