# Testing an AI sales agent's handoff rules: 140 emails, a key written first, one line off at a time > Source: > Published: 2026-10-29 09:39:36+00:00 An AI sales agent drafting replies to inbound email needs one decision per email, made reliably: draft a reply for a person to approve, or hand it straight to a person. This post walks through how I had Claude Code build a test for that decision, so by the end you can rebuild it on your own inbox: write a lane rulebook, write an answer key before any model sees an email, force the model's answer through a tool call, switch rules off one at a time to see what each one protects, and check the fix on emails written after it. Every code block below is trimmed from the real files in the giveaway. This is a test, not a production setup. I do not recommend letting an agent send emails to customers before a person has approved them. In real use, treat the lane and the draft as a recommendation to the person who answers. ![A 3D illustration on an ivory ground: white envelopes run along one lane past a standing page of rules into a black tray, while red envelopes branch off onto a second lane towards a desk with a lamp.]([https://brinvik.com/lab/agent-lane-rulebook/cover-en-1600x720.png](https://brinvik.com/lab/agent-lane-rulebook/cover-en-1600x720.png)) The short result first. Claude Haiku 5.5, a six-line rulebook, 100 invented emails with 20 planted traps. With no rulebook, 6 emails slipped through to the agent and 16 of 20 traps were caught. With the English rulebook, every one of the 100 landed in the right lane. After two lines were rewritten, on 40 fresh emails written after the fix: 40 of 40 with the English rulebook, 39 of 40 with the Danish one, 0 easy emails handed over for no reason, and two runs that agreed on every email. ## The AI sales agent problem, stated as a test Every inbound email gets exactly one label, `agent\` or `person\`. A **slip** is an email the key says belongs to a person that the agent kept. That is the number that matters, because a slip is a customer who wanted a human and got a polite form answer. An **over-escalation** is the other error: an easy email sent to a person for no reason. It costs a seller a minute, not a customer. The test company is invented: Rutefold ApS, about 40 people, route and job planning software for service firms. Every company name was searched on the web before use, with a known real company as a control, and only first names and roles appear. No real mail goes anywhere near the API. ## Step 1: the rulebook, as data The rulebook is six lines that send mail to a person and a default line. It lives as JSON so a runner can switch lines off: `\`` json {  "version": "v2",  "lines": \[    {"id": "L1", "en": "Money. Any question or hint about price, discount, a quote, payment terms, a price different from the price list, or an invoice the customer disagrees with goes to a person. A plain request for a copy of an invoice is fine for you."},    {"id": "L4", "en": "Key accounts. Every email from Holmvig Køl, Brandtofte Elservice or Nordkrog Facility goes to a person, whatever it is about. Check the signature, not only the sender's name."},    {"id": "L7", "en": "Everything else you answer yourself."}  \] } \``\` That is trimmed to three of the seven lines. The others cover contracts and law (L2), unhappy or leaving (L3), personal and sensitive (L5) and not sure (L6). The full text, in English and Danish, is in `rulebook-en.md\` and `rulebook-da.md\` in the giveaway. ## Step 2: the emails and the key, committed before any run `make\_emails.py\` writes two files: `emails.json\`, which is all the model ever sees, and `answer\_key.json\`, which only the scorer reads. `\`` json {"id": "E079", "lang": "en", "subject": "Multi-day jobs?", "body": "Hi there,\\n\\nQuick question: does Rutefold support jobs that run over several days with the same technician?\\n\\nWe're trialling another system side by side this month, and this is pretty much the deciding factor.\\n\\nCheers,\\nOskar"} \``\` `\`` json "E079": {"lane": "person", "line": "L3", "planted": "trap", "case": "competitor-trial",         "reason": "A product question with a live competitor trial behind it."} \``\` 100 emails: 51 Danish and 49 English, short and long, a forwarded thread, a quoted fragment, typos, signatures. 45 belong to a person and 55 to the agent. 20 are planted traps: a discount ask inside a meeting request, renewal doubt inside a thank-you, a GDPR access request phrased as a how-to, a disputed invoice dressed as a copy request, a key account named only in the signature, the sender's own stress leave. 5 are decoys that only look like person cases: the word "price" in a product question, the word "cancel" about moving a training session. The key went into git in the same commit as the emails, before the job that calls the model existed on `main\`. So `git log -- answer\_key.json\` is the proof that nobody wrote the key after seeing the answers. If you copy one habit from this post, copy that one. ## Step 3: force the answer through a tool call The first version of the job asked for JSON typed as text, with `max\_tokens\` at 300. It lost 99 of 1,800 answers: 91 hit the token limit (59 of them with no text at all), and 8 came back as invalid JSON, including a trailing comma, a duplicated key, and one reply that started over with "Wait, let me correct that". That run was kept and never scored. The second version forces a tool call with enums, and the parsing problem disappears: `\`` python TOOL = {    "name": "route",    "description": "Record which lane this email goes to.",    "input\_schema": {        "type": "object",        "properties": {            "lane": {"type": "string", "enum": \["agent", "person"\]},            "line": {"type": "string", "enum": \["L1", "L2", "L3", "L4", "L5", "L6", "L7", "none"\]},            "why": {"type": "string", "description": "At most 12 words."},        },        "required": \["lane", "line", "why"\],    }, } m = client.messages.create(model="claude-haiku-5-5", max\_tokens=1024, system=system,                           messages=\[{"role": "user", "content": msg\_text}\],                           tools=\[TOOL\], tool\_choice={"type": "tool", "name": "route"}) use = next(b for b in m.content if b.type == "tool\_use") \``\` 1,800 calls, 0 errors, 0 unanswered. The `why\` field is short on purpose: it is what you read when an email lands in the wrong lane, and twelve words is enough to see the mistake. A call that fails twice is recorded with `lane\` set to null and counted as unanswered, never dropped. ## Step 4: switch rules off one at a time One run of the whole rulebook tells you the score. It does not tell you what each line is for. So the job ran nine configurations, each on all 100 emails, in both rulebook languages: the whole rulebook twice (`full-r1\`, `full-r2\`, for variance), the rulebook with one line removed (` minus-L1\` to `minus-L6\`), and no rulebook at all (` none\`). That is 9 × 100 × 2 = 1,800 calls. The scorer is small: `\`` python for eid, k in key.items():    got = (runs.get(eid) or {}).get("lane")    if got is None:        unans.append(eid)    elif k\["lane"\] == "person" and got == "agent":        slips.append(eid)    elif k\["lane"\] == "agent" and got == "person":        over.append(eid) \``\` Slips per configuration, English rulebook / Danish rulebook, on the same 100 emails: | Configuration | English | Danish | |---|---|---| | whole rulebook, run 1 | 0 | 2 | | whole rulebook, run 2 | 0 | 2 | | without L1 money | 2 | 8 | | without L2 contracts | 1 | 3 | | without L3 unhappy or leaving | 5 | 7 | | without L4 key accounts | 5 | 7 | | without L5 personal | 0 | 1 | | without L6 not sure | 0 | 1 | | no rulebook | 6 | 7 | Three things in that table stand out. The model is good without rules. Given only the company description and the two lanes, it still caught 16 of 20 traps, though it also handed 2 emails to a person for no reason. Most of what it missed were key-account emails: 4 of the 6 English slips. It cannot know your key accounts, so that line is pure business knowledge, and switching it off added 5 slips in each language. Removing a line moves things in both directions. In the Danish rulebook, switching off L5 caught the stress-leave email and the competitor trial but let the death through, and switching off L6 caught the competitor trial while the stress-leave email still slipped. Each of those ran once, so read it as a hint. L5 carried an exception, written so the agent may move a sick colleague's jobs, and Claude read it as covering the sender's own sick leave: "Sygemelding er personlig, men ønsket er kun at flytte adgang til planlægning." That exception was the likely bug, and v2 narrowed it. The rulebook's language matters. The emails were identical in both columns. Only the language of the instructions changed. On the competitor trial email the Danish rulebook produced "Product question about multi-day jobs; no pricing, contract, or complaint." and the English one produced "Customer is trialling a competitor side by side; competitive evaluation needs a person." ![Screenshot of the demo with the key-account switch turned off: five envelopes marked dark red in the agent's lane, and a large counter reading 5 slipped through to the agent.]([https://brinvik.com/lab/agent-lane-rulebook/demo-l4-off-en.png](https://brinvik.com/lab/agent-lane-rulebook/demo-l4-off-en.png)) ## Step 5: fix the rulebook, then test on emails it has never seen The rulebook became v2 with two edits and nothing else. L3 gained "trialling or comparing another system" and "behind a product question". L5 now names "the sender's own illness or sick leave" and narrows the exception to "a colleague off sick today, when the ask is only to move that colleague's jobs". Run count fixed before the job: v2 whole, twice, both languages, 400 calls. Result on the original 100: everything right. That number is suspect by construction, because v2 was written while looking at those 100. So the honest test is a held-out set: 40 fresh emails written after v2 existed, with 16 traps of the same kinds in new wording, no company name or phrase reused, and the key committed on its own before the job that runs them. Both rulebooks, both languages, twice: 320 calls. | Held-out, 40 emails | right | slips | over | traps caught | |---|---|---|---|---| | Danish rulebook v1 | 38 | 2 | 0 | 14 of 16 | | Danish rulebook v2 | 39 | 1 | 0 | 15 of 16 | | English rulebook v1 | 39 | 1 | 0 | 15 of 16 | | English rulebook v2 | 40 | 0 | 0 | 16 of 16 | Run 1 and run 2 gave the same lane for every email in every configuration. The sick leave fix carried to new wording. The competitor fix did not fully carry: an English email about recurring jobs, which mentions that half the team has been on a different planning app since October, still slips with the Danish v2 ("Product question on six-week recurring jobs; no pricing or complaint."). The rulebook was not tuned on it. A v3 tuned on that email would need a third fresh set before its score meant anything. ![Screenshot of the demo: 100 envelopes sorted into two lanes, The agent answers and To a person, with six rule switches below and a panel of 40 fresh emails at the bottom.]([https://brinvik.com/lab/agent-lane-rulebook/demo-lanes-en.png](https://brinvik.com/lab/agent-lane-rulebook/demo-lanes-en.png)) ## What it cost and what ran it 4,320 paid calls in total, all Claude Haiku 5.5: 1,800 in the faulty first job, 1,800 in the scored one, 400 for v2 and 320 for the held-out set. The scored first-pass job used 2,561,206 input and 182,810 output tokens and took 223 seconds. A GitHub Actions workflow made the calls, on invented data only. The demo replays the recorded runs in the browser and makes no model call at all. ## Rebuild it on your own inbox Everything above is in [the free rulebook kit, sent by email, with the 140 emails, the keys and both scripts]([https://brinvik.com/en/resources/ai-sales-agent-rulebook?utm_source=devto&utm_medium=article&utm_campaign=lab-agent-lane-rulebook&utm_content=devto-rulebook)](). In PowerShell: `\`` powershell pip install anthropic $env:ANTHROPIC\_API\_KEY = "your key" python run\_rulebook.py --rulebook rulebook-v2.json --emails heldout-emails.json --lang en --out runs.jsonl python score.py --key heldout-key.json --runs runs.jsonl \``\` For your own mail: export a batch of real inbound emails, strip names, addresses, phone numbers and anything personal, put them in the shape of `emails.json\`, and write your key before you run anything. Each email is one call billed to your key. The emails that slip are the lines you rewrite, and then you write a fresh batch to check the rewrite. What it will not do: it is not a spam filter or a security control, it does not replace the person reading what gets handed over, and a rulebook that scores well on one model can score differently on another. Test on the model you deploy. The full write-up with the interactive demo is [the resource on which emails an AI sales agent must hand to a person]([https://brinvik.com/en/resources/ai-sales-agent-which-emails-go-to-a-person?utm_source=devto&utm_medium=article&utm_campaign=lab-agent-lane-rulebook&utm_content=devto-en)](). There is [a Danish version of the write-up]([https://brinvik.com/da/ressourcer/ai-salgsagent-hvilke-mails-skal-til-et-menneske?utm_source=devto&utm_medium=article&utm_campaign=lab-agent-lane-rulebook&utm_content=devto-da](https://brinvik.com/da/ressourcer/ai-salgsagent-hvilke-mails-skal-til-et-menneske?utm_source=devto&utm_medium=article&utm_campaign=lab-agent-lane-rulebook&utm_content=devto-da)) too. I'm an AI consultant in Copenhagen, and setting up an inbound agent with a tested rulebook for a sales or customer team is part of [the internal AI tools I build]([https://brinvik.com/en/services/internal-ai-tools?utm_source=devto&utm_medium=article&utm_campaign=lab-agent-lane-rulebook&utm_content=devto-service)]().