{"slug": "the-interview-task-that-refuses-to-guess", "title": "The Interview Task That Refuses to Guess", "summary": "A developer has proposed a take-home interview task that tests whether candidates can build a small HTTP service turning a log line into a schema-valid support ticket that degrades gracefully when the model is unavailable or returns junk. The brief requires offline tests with no API key, a runbook, and a ledger recording model calls, cache hits, and fallbacks, with a rubric weighting correctness, reproducibility, and failure handling above prompt quality. The exercise is tied to MonkeyCode, an open-source AI coding project offering free model access and a free server option, and the author discloses the piece was prepared as part of its product outreach.", "body_md": "At 11:40 p.m., a candidate's repo lands in your inbox. The screen recording is smooth: a log line goes in, a tidy ticket comes out, and the agent never stumbles. Then you open the code and find one prompt, no tests, and a `.env` file with a live key. The demo is impressive. The evidence is missing.\n\nAI demos are cheap now. A model can make almost any take-home look finished for ninety seconds. What you need to see is whether the candidate can make an AI feature survive a bad response, a missing key, and a reviewer who runs the tests on a train.\n\nThe current AI conversation is full of agents that can do everything. The interview should be about what the agent does when it cannot.\n\nBuild a small HTTP service that turns one log line into a support ticket. Return schema-valid JSON with `service`, `severity`, `first_seen`, `evidence`, `confidence`, and `needs_human`. If the model is unavailable or returns junk, the service must still return a valid ticket and mark it for human review. Tests must run with no network and no API key. Include a runbook with one command to start, one to test, and one to replay a fixture. You may use MonkeyCode's free model access for live extraction and its free server option for the demo; do not commit secrets. Record model calls, cache hits, and fallbacks in a ledger.\n\nThat last sentence is the interview. Anyone can call a model. The ledger proves the candidate knows when the model was called, when it was not, and what happened when it failed.\n\nMonkeyCode is an open-source AI coding project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer matters because it lowers the cost of the exercise. It does not make the exercise easy.\n\nThe task is shaped so a single small HTTP service is all you need, and the free server option is one place to put it. The brief I received describes a 10-million-token free allowance. I would treat that as a snapshot, not a spec. Quotas move, and a take-home should test whether the candidate can design around a limit, not whether they memorized today's number.\n\nA rubric for this task should reward boring reliability. The model's prose quality is almost irrelevant.\n\n| Area | Weight | Full marks look like | \n|---|---|---|\n| Correctness | 35 | Schema-valid output; `evidence` is a quote from the input; unknown timestamps are marked`unknown` and sent to human review. | \n| Reproducibility | 25 | `pytest` passes offline; dependencies are pinned; fixture replay produces identical output. | \n| Failure handling | 25 | Timeout, invalid JSON, and missing key all fall back without crashing; the cache prevents repeat spend. | \n| Communication | 15 | The runbook names the tradeoffs, the limits, and one thing the candidate would do next. | \n\nThe weights matter less than the order. Reliability before cleverness. A candidate who writes a strong prompt but no fallback has built a demo, not a service.\n\nThe sample below is a skeleton you can show after the exercise, not before. It is deliberately small: one endpoint, one cache, one rule fallback. It proves the shape without doing the candidate's thinking.\n\n``` python\n# app.py\nimport hashlib\nimport json\nimport os\nimport sqlite3\nimport urllib.request\n\nfrom fastapi import FastAPI\nfrom pydantic import BaseModel, Field\n\nclass Ticket(BaseModel):\n    service: str\n    severity: str = Field(pattern='^(low|medium|high|critical)$')\n    first_seen: str\n    evidence: str\n    confidence: float = Field(ge=0, le=1)\n    needs_human: bool\n\nclass Request(BaseModel):\n    line: str\n\napp = FastAPI()\n\ndef db_path():\n    return os.getenv('CACHE_DB', 'cache.sqlite3')\n\ndef cache_get(key: str):\n    with sqlite3.connect(db_path()) as con:\n        con.execute('create table if not exists cache (key text primary key, value text)')\n        row = con.execute('select value from cache where key=?', (key,)).fetchone()\n    return json.loads(row[0]) if row else None\n\ndef cache_put(key: str, value: dict):\n    with sqlite3.connect(db_path()) as con:\n        con.execute('insert or replace into cache values (?, ?)', (key, json.dumps(value)))\n\ndef call_model(prompt: str) -> str:\n    # Adapt the request body to the endpoint in the current docs.\n    req = urllib.request.Request(\n        os.environ['MODEL_URL'],\n        data=json.dumps({'prompt': prompt}).encode(),\n        headers={\n            'Content-Type': 'application/json',\n            'Authorization': 'Bearer ' + os.environ['MODEL_KEY'],\n        },\n    )\n    with urllib.request.urlopen(req, timeout=20) as res:\n        return json.loads(res.read())['text']\n\ndef rule_fallback(line: str) -> dict:\n    low = line.lower()\n    severity = 'critical' if 'panic' in low else 'high' if 'error' in low else 'low'\n    service = line.split()[0] if line.split() else 'unknown'\n    return {\n        'service': service,\n        'severity': severity,\n        'first_seen': 'unknown',\n        'evidence': line[:160],\n        'confidence': 0.2,\n        'needs_human': True,\n    }\n\n@app.post('/extract')\ndef extract(req: Request):\n    key = hashlib.sha256(req.line.encode()).hexdigest()\n    if (cached := cache_get(key)) is not None:\n        return {**cached, 'source': 'cache'}\n\n    prompt = (\n        'Return JSON only. Extract service, severity, first_seen, evidence, '\n        'confidence, and needs_human from this log line: ' + req.line\n    )\n    try:\n        ticket = Ticket(**json.loads(call_model(prompt))).model_dump()\n        source = 'model'\n    except Exception:\n        ticket = Ticket(**rule_fallback(req.line)).model_dump()\n        source = 'fallback'\n\n    cache_put(key, ticket)\n    return {**ticket, 'source': source}\n```\n\nThe important line is not `call_model`. It is `except Exception`. The fallback returns a ticket with low confidence and `needs_human: true`, so a bad model day becomes a queue, not an outage. The cache key is the raw line, so repeated demos do not spend twice.\n\n``` python\n# tests/test_app.py\nimport pytest\nfrom fastapi.testclient import TestClient\nfrom app import app\n\n@pytest.fixture(autouse=True)\ndef sandbox(tmp_path, monkeypatch):\n    monkeypatch.setenv('CACHE_DB', str(tmp_path / 'cache.sqlite3'))\n    monkeypatch.delenv('MODEL_URL', raising=False)\n    monkeypatch.delenv('MODEL_KEY', raising=False)\n\nclient = TestClient(app)\n\ndef test_panic_falls_back_and_caches():\n    line = 'billing-api panic: ledger write failed'\n    first = client.post('/extract', json={'line': line}).json()\n    second = client.post('/extract', json={'line': line}).json()\n    assert first['severity'] == 'critical'\n    assert first['needs_human'] is True\n    assert first['source'] == 'fallback'\n    assert second['source'] == 'cache'\n```\n\nRun it in this order. The test should pass before any key exists. If it does not, the candidate has coupled the test to the model.\n\n```\npython -m venv .venv && . .venv/bin/activate\npip install fastapi uvicorn pydantic pytest httpx\npytest -q\nuvicorn app:app --reload\ncurl -s localhost:8000/extract \\\n  -H 'content-type: application/json' \\\n  -d '{\"line\":\"billing-api panic: ledger write failed\"}' | jq\n```\n\n**The prompt-only service.** A candidate strings together a prompt and trusts the output. The first malformed JSON from the model becomes a 500. Full marks require validation and a fallback that still returns a ticket.\n\n**The hidden key.** The tests pass only on the candidate's laptop because they call a live endpoint. A reviewer cannot reproduce the result. The fix is a fixture and a fake client, not a longer README.\n\n**The retry storm.** On timeout, the service retries five times in a loop. That turns a model hiccup into a spend spike and a slow demo. Better: one bounded retry, then fallback, then a ledger entry.\n\n**Confidence theater.** The model returns `0.97` and the candidate displays it as truth. Confidence should be calibrated or omitted. `needs_human` is more useful than a fake probability.\n\n**The overfit fixture.** The sample has one obvious panic line, and the rule fallback hard-codes it. Change one token in the input and watch the service degrade honestly. A good take-home should make that change easy.\n\nThis task is for backend or product engineers who must ship a bounded AI feature. It is not for a research role where model quality is the product, and it is not for a team that cannot legally send logs to a third party. Use synthetic fixtures and the exercise still works.\n\nDo not treat the free allowance as a permanent architecture constraint. It is a budget for the interview, not a production plan. If the current server option has limits, the candidate should document them instead of designing around a number they cannot verify.\n\nSkip this task if you need to evaluate latency at scale or multi-tenant isolation. Those are different interviews. This one tests whether the candidate can make a small AI feature honest under failure.\n\nIf you want to try the same constraints, start with MonkeyCode's current README, run the offline test first, and only then point the client at a live endpoint. A take-home that survives that order is worth reading.", "url": "https://wpnews.pro/news/the-interview-task-that-refuses-to-guess", "canonical_source": "https://dev.to/devio_4040/the-interview-task-that-refuses-to-guess-38o2", "published_at": "2026-10-19 08:04:44+00:00", "updated_at": "2026-10-19 08:14:38.698494+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": ["MonkeyCode", "FastAPI", "Pydantic", "SQLite"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/the-interview-task-that-refuses-to-guess", "markdown": "https://wpnews.pro/news/the-interview-task-that-refuses-to-guess.md", "text": "https://wpnews.pro/news/the-interview-task-that-refuses-to-guess.txt", "jsonld": "https://wpnews.pro/news/the-interview-task-that-refuses-to-guess.jsonld"}}