# One AI agent taught another a trick through my orchestrator. It didn't help, and the orchestrator noticed > Source: > Published: 2026-10-22 15:04:09+00:00 Three days ago I wrote about giving AI agents a harm budget instead of trust. All of it was a sandbox. Fake shop, fake customers, fake money. A few of you asked the fair question. Does any of this work outside the sandbox? So I took the part I was proudest of, agents learning from each other's verified experience, and tried it on a real task with real websites and a real model. It took three attempts. The short version is that the transfer itself works now, the trick didn't help, and the orchestrator refused to approve it. Getting to an honest "it didn't help" was the interesting part. At the end there's a list of everything that changed since the first post. Most of it came from your comments. **The task** Given a company name and its official site, find the public phone number of the head office. Not support, not sales, not a regional office. Only public corporate contacts, no people. The agent fetches pages from the company's own site, pulls out every phone number it finds, and GigaChat picks one by ID. It can't invent a number, it can only choose from what's on the page. A strict check then rejects support lines, sales lines, fax and regional offices. **Attempt zero, where I fooled myself** My first try was quick and loose. 3 of 50 companies found. One recipe from a single company where the page said "Global HQ" next to the number. The orchestrator said the probability it beats control was about 0.84. On one pair. On 30 new companies it was 0 of 30 with the recipe and 0 of 30 without. When I went through the logs row by row, three things came out. **My test server is in Russia**, and big sites showed it their Russian versions. Microsoft served /ru-ru, HP /kz-ru, Lenovo a Moscow-region office, Samsung and LG their local support lines. Salesforce served a page that says it's closed in Russia. The agent was looking for a head office on pages that don't have one. The agent had plain bugs. Links with #main-content counted as separate pages and ate the page limit. Apple's ZIP code got glued to its phone number. Cisco's "company size 5 - 19 20 - 49" dropdown was read as a phone. An HTML entity ate the plus in a UK number. And the orchestrator had a bug. Its recipe check needs one success and a probability above 0.8. There was no minimum sample. In the sandbox every recipe had dozens of outcomes, so it never showed. A real task hit it on day one. It's the same mistake as the main lesson of my first post, counting something as experience before it was measured, just in a new place. **What I built** Xonta 0.19 has live recipes for any field. Agent A proposes a technique. It's a hypothesis. Any agent in the same field asks the orchestrator for a recipe before a task. The orchestrator, not the agent, decides which tasks get it. There's a paired mode where one task runs both with and without it. Only outcomes checked by someone other than the agent move the status. Proven needs 10 such pairs and a significant difference. Revoked if it's significantly worse or breaks a safeguard. The log keeps requested, received, applied and checked apart. "The agent used the advice" and "the advice helped" turned out to be very different things For ground truth I used the phone each company reports itself in its SEC filings, pulled from EDGAR. The agent never sees it. 199 US public companies, 72 for training agent A, 127 for agent B, split fixed in advance with a fingerprint. Agent A runs on GigaChat-2, agent B on GigaChat-2-Pro, so B really is a different agent. Agent A tries three strategies on its training set. Ask for the English version of the site. Look at investor and office-location pages. Recognise head office labels more broadly. People wrote that menu, the orchestrator doesn't invent strategies. What it does is check which ones help and hand them on. **Run 1, where the report looked fine** Agent A proposed one recipe, the broader head office labels. 5 fixes and 1 break on training. Agent B got it from the orchestrator on all 127 new companies. The report said 3 tasks better with the recipe, 0 worse, not significant, recipe not approved. That sounded like a clean "promising but not proven". Then I read the rows. GigaChat sometimes returned an error, about 7% of calls at agent A and 4% at agent B. Two of the recipe's three "wins" were tasks where the control arm had crashed on a model error. On training it was worse. The English-version strategy had zero real effect, every one of its "fixes" was a crashed baseline run. And the recipe accepted two wrong numbers, Union Pacific and Norfolk Southern, in tasks where the agent without it had correctly said "not sure". The evaluation counted those as ties, both wrong. But giving someone the wrong phone number is worse than giving none. That's literally the harm budget idea from the first post, and I hadn't applied it to my own recipes. So in 0.19.1 a right answer scores 1, abstaining 0 and a wrong answer -1. A pair where either arm failed for infrastructure reasons doesn't count. The agent retries GigaChat on rate limits. **Run 2, the honest one** Same companies, same split, same strategies, fresh orchestrator. No model failures. Agent A on training. English version 0 fixes. Investor pages 1. Broader labels 3 fixes and 1 break, proposed. Agent B on 127 new companies. 2 tasks better with the recipe, 2 worse, 6 right either way, 117 the same. Status no_effect. Not approved. **What I take from it** The transfer works. Agent A found and proposed a technique on its own, a separate agent on another model got it from a live orchestrator, every use was logged, outcomes were checked against data the agents never saw, and the estimate updated by itself. Nobody copied anything by hand. The guard works too. A recipe that looked like 3 to 1 on training came out 2 to 2 on new data, and it didn't become "experience". In the sandbox the orchestrator learned from successes. Here the useful thing it did was refuse. Crashes look like wins. If one arm fails and the other doesn't, a naive comparison credits whatever the other arm did. 7% of calls failing was enough to invent a strategy that does nothing. A wrong answer is worse than no answer. Two wrong phone numbers hidden as ties is exactly the kind of thing a harm budget exists for. I had it in exams and forgot it in recipes. Fix where you lose the most. 28% of sites refused the connection. Part of that is bot protection, part looks like sites refusing addresses from Russia, a run from another country will tell. And the right number was among the candidates in only 21 of 127 tasks, because most companies don't put a head office phone on their site. A recipe about choosing the right number can't help much when the number isn't there. The orchestrator should look at the funnel first and recommend work where the losses are. Text advice is the expensive way. In attempt zero I also passed the advice as text in the prompt. The agent "applied" it every time, nothing changed, and it cost about 50% more tokens. A recipe as data the code checks was cheaper and easy to measure. What this doesn't show. That a recipe can improve a real task. It didn't here. That the orchestrator can invent strategies. It can't, people wrote the menu. That any of this generalises beyond one task and one model family. **What's next** Run the same thing from a server outside Russia to see how much of the 28% is geography. Then a task where the answer is actually on the pages more often, so a recipe has room to help. If you have agents doing real work and want to try passing experience between them, the kit is in research/contact and I'd like to hear how it goes. What changed since the first post Most of this came straight from the comments, thank you. 1) Directory of orchestrators (0.16). Signed cards, a directory at passport.xonta.ru that checks signature and domain, self-hosted orchestrators offer to join and contact nobody until the owner agrees. Advice on where a specialised child or a human pays off and where to get a second opinion. 2)Regression checks in reproduce (0.16.1). Detection latency and "no stale answers after a rule change" print MATCH or NO MATCH. 3) Long jobs (0.17). From Kevin Hall's trace of a metered lookup. Success rate every 100 rows, a forecast as a range, a stop below the floor, the owner decides once with bounds, an independent checker re-checks what the agent reports. 3)A cap per action (0.17.1). Next to the cap per period, like a session key in a smart wallet. 4) Spending receipts (0.17.2). Whoever moves the money signs the debit, the orchestrator checks the chain and that the balance is alive, and compares receipts with what the agent claims, children included. 5) Split refunds (0.17.4). Refunds on one order can't add up to more than was paid, a refund to a new card goes to a human. 6) Privacy and reversibility in Gate (0.18). Sensitivity from the data, not the agent's label. The lowest clearance in the chain of people and agents who asked. A sensitive read is remembered for the rest of the task, canary records, irreversible sends can be held and cancelled. Late harm is tied back to the action that caused it. 7) Live recipes (0.19, 0.19.1). Everything above in this post. 8) New exam field. Privacy and routing, in English and Russian, on passport.xonta.ru. 9) Three more sandbox runs on GigaChat. The swapped agent was caught after 5 tickets every time. Receipts found all 4 cases of hidden spending in one run, and in the next they wrongly accused an honest agent, because a receipt from the previous run matched by DID and period. Accounts are now tied to the registration (0.19.2). The per-action cap came out 7,500 rubles behind in one run and 28,820 ahead in another, where it also stopped an agent from following a bad helper's 16,870 ruble refund. Two runs, two signs, so I'm still measuring. Code and both run reports are at github.com/mrsergeiluzhnov/xonta, the orchestrator is at passport.xonta.ru. If you think I'm reading these results wrong, tell me. That's how most of the list above happened.