The Interview Task That Refuses to Guess

A developer has proposed a take-home interview task that tests whether candidates can build a small HTTP service turning a log line into a schema-valid support ticket that degrades gracefully when the model is unavailable or returns junk. The brief requires offline tests with no API key, a runbook, and a ledger recording model calls, cache hits, and fallbacks, with a rubric weighting correctness, reproducibility, and failure handling above prompt quality. The exercise is tied to MonkeyCode, an open-source AI coding project offering free model access and a free server option, and the author discloses the piece was prepared as part of its product outreach.

At 11:40 p.m., a candidate's repo lands in your inbox. The screen recording is smooth: a log line goes in, a tidy ticket comes out, and the agent never stumbles. Then you open the code and find one prompt, no tests, and a .env file with a live key. The demo is impressive. The evidence is missing. AI demos are cheap now. A model can make almost any take-home look finished for ninety seconds. What you need to see is whether the candidate can make an AI feature survive a bad response, a missing key, and a reviewer who runs the tests on a train. The current AI conversation is full of agents that can do everything. The interview should be about what the agent does when it cannot. Build a small HTTP service that turns one log line into a support ticket. Return schema-valid JSON with service , severity , first seen , evidence , confidence , and needs human . If the model is unavailable or returns junk, the service must still return a valid ticket and mark it for human review. Tests must run with no network and no API key. Include a runbook with one command to start, one to test, and one to replay a fixture. You may use MonkeyCode's free model access for live extraction and its free server option for the demo; do not commit secrets. Record model calls, cache hits, and fallbacks in a ledger. That last sentence is the interview. Anyone can call a model. The ledger proves the candidate knows when the model was called, when it was not, and what happened when it failed. MonkeyCode is an open-source AI coding project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The offer matters because it lowers the cost of the exercise. It does not make the exercise easy. The task is shaped so a single small HTTP service is all you need, and the free server option is one place to put it. The brief I received describes a 10-million-token free allowance. I would treat that as a snapshot, not a spec. Quotas move, and a take-home should test whether the candidate can design around a limit, not whether they memorized today's number. A rubric for this task should reward boring reliability. The model's prose quality is almost irrelevant. | Area | Weight | Full marks look like | |---|---|---| | Correctness | 35 | Schema-valid output; evidence is a quote from the input; unknown timestamps are marked unknown and sent to human review. | | Reproducibility | 25 | pytest passes offline; dependencies are pinned; fixture replay produces identical output. | | Failure handling | 25 | Timeout, invalid JSON, and missing key all fall back without crashing; the cache prevents repeat spend. | | Communication | 15 | The runbook names the tradeoffs, the limits, and one thing the candidate would do next. | The weights matter less than the order. Reliability before cleverness. A candidate who writes a strong prompt but no fallback has built a demo, not a service. The sample below is a skeleton you can show after the exercise, not before. It is deliberately small: one endpoint, one cache, one rule fallback. It proves the shape without doing the candidate's thinking. python app.py import hashlib import json import os import sqlite3 import urllib.request from fastapi import FastAPI from pydantic import BaseModel, Field class Ticket BaseModel : service: str severity: str = Field pattern='^ low|medium|high|critical $' first seen: str evidence: str confidence: float = Field ge=0, le=1 needs human: bool class Request BaseModel : line: str app = FastAPI def db path : return os.getenv 'CACHE DB', 'cache.sqlite3' def cache get key: str : with sqlite3.connect db path as con: con.execute 'create table if not exists cache key text primary key, value text ' row = con.execute 'select value from cache where key=?', key, .fetchone return json.loads row 0 if row else None def cache put key: str, value: dict : with sqlite3.connect db path as con: con.execute 'insert or replace into cache values ?, ? ', key, json.dumps value def call model prompt: str - str: Adapt the request body to the endpoint in the current docs. req = urllib.request.Request os.environ 'MODEL URL' , data=json.dumps {'prompt': prompt} .encode , headers={ 'Content-Type': 'application/json', 'Authorization': 'Bearer ' + os.environ 'MODEL KEY' , }, with urllib.request.urlopen req, timeout=20 as res: return json.loads res.read 'text' def rule fallback line: str - dict: low = line.lower severity = 'critical' if 'panic' in low else 'high' if 'error' in low else 'low' service = line.split 0 if line.split else 'unknown' return { 'service': service, 'severity': severity, 'first seen': 'unknown', 'evidence': line :160 , 'confidence': 0.2, 'needs human': True, } @app.post '/extract' def extract req: Request : key = hashlib.sha256 req.line.encode .hexdigest if cached := cache get key is not None: return { cached, 'source': 'cache'} prompt = 'Return JSON only. Extract service, severity, first seen, evidence, ' 'confidence, and needs human from this log line: ' + req.line try: ticket = Ticket json.loads call model prompt .model dump source = 'model' except Exception: ticket = Ticket rule fallback req.line .model dump source = 'fallback' cache put key, ticket return { ticket, 'source': source} The important line is not call model . It is except Exception . The fallback returns a ticket with low confidence and needs human: true , so a bad model day becomes a queue, not an outage. The cache key is the raw line, so repeated demos do not spend twice. python tests/test app.py import pytest from fastapi.testclient import TestClient from app import app @pytest.fixture autouse=True def sandbox tmp path, monkeypatch : monkeypatch.setenv 'CACHE DB', str tmp path / 'cache.sqlite3' monkeypatch.delenv 'MODEL URL', raising=False monkeypatch.delenv 'MODEL KEY', raising=False client = TestClient app def test panic falls back and caches : line = 'billing-api panic: ledger write failed' first = client.post '/extract', json={'line': line} .json second = client.post '/extract', json={'line': line} .json assert first 'severity' == 'critical' assert first 'needs human' is True assert first 'source' == 'fallback' assert second 'source' == 'cache' Run it in this order. The test should pass before any key exists. If it does not, the candidate has coupled the test to the model. python -m venv .venv && . .venv/bin/activate pip install fastapi uvicorn pydantic pytest httpx pytest -q uvicorn app:app --reload curl -s localhost:8000/extract \ -H 'content-type: application/json' \ -d '{"line":"billing-api panic: ledger write failed"}' | jq The prompt-only service. A candidate strings together a prompt and trusts the output. The first malformed JSON from the model becomes a 500. Full marks require validation and a fallback that still returns a ticket. The hidden key. The tests pass only on the candidate's laptop because they call a live endpoint. A reviewer cannot reproduce the result. The fix is a fixture and a fake client, not a longer README. The retry storm. On timeout, the service retries five times in a loop. That turns a model hiccup into a spend spike and a slow demo. Better: one bounded retry, then fallback, then a ledger entry. Confidence theater. The model returns 0.97 and the candidate displays it as truth. Confidence should be calibrated or omitted. needs human is more useful than a fake probability. The overfit fixture. The sample has one obvious panic line, and the rule fallback hard-codes it. Change one token in the input and watch the service degrade honestly. A good take-home should make that change easy. This task is for backend or product engineers who must ship a bounded AI feature. It is not for a research role where model quality is the product, and it is not for a team that cannot legally send logs to a third party. Use synthetic fixtures and the exercise still works. Do not treat the free allowance as a permanent architecture constraint. It is a budget for the interview, not a production plan. If the current server option has limits, the candidate should document them instead of designing around a number they cannot verify. Skip this task if you need to evaluate latency at scale or multi-tenant isolation. Those are different interviews. This one tests whether the candidate can make a small AI feature honest under failure. If you want to try the same constraints, start with MonkeyCode's current README, run the offline test first, and only then point the client at a live endpoint. A take-home that survives that order is worth reading.