My tests all passed. The broker disagreed.

A developer running TopSet, a personal stock-trading system built largely by AI agents, reported that over four months and roughly 440 pull requests, about 1 in 9 hit at least one failed CI build, with 53 failed builds in total — 26 stopped by type checking, 14 by tests and 12 by lint — yet the bugs that got furthest were caught only by a daily post-trade comparison against real brokers Alpaca and Tradier. The most serious was a cash-accounting race in which fill-recording and purchase-planning code held different locks, letting the same cash be counted twice and pushing the bot to overspend until a database rule blocking negative cash stopped it; the first CI run on that pull request failed but the rerun passed. The developer's conclusion is that mocks only repeat the builder's beliefs about the broker, so reconciliation against the live broker is the check that finds errors in those beliefs.

This is a story about the bugs my tests couldn’t find, and the check that found them anyway. TopSet is the trading system I built and run on my own accounts. It picks stocks with machine-learning models, rebalances on a schedule and places real orders through two brokers, Alpaca and Tradier. AI agents wrote most of the code, and my first post about it https://redgeoff.com/posts/ai-agents-test-gate/ described the test gate that decides what merges: thousands of tests, a weekly 100-run soak and a few results I refuse to let change quietly. That gate still matters, but over time I learned that each kind of test answers a different question, and none of them answers the one I care about most: do my records match the broker’s? A mock broker can only repeat what I believe about the real one. The bugs that got furthest were the ones where my belief was wrong, and the thing that caught them was a boring comparison that runs every day after trading. So let’s take a look at what each layer caught, and then at what only the broker could tell me. Here is the short version, with the longer stories below. The first six check my code, either on every pull request or in a weekly run. The last one runs in production, and it is the only one that talks to the real world. CI, short for continuous integration, is the set of checks that runs on every pull request a proposed change before it can merge. Before an agent pushes a change, it runs the same lint and type checks and most of the tests on its own machine, and fixes whatever fails there. Those local failures aren’t recorded anywhere, but I’d guess there were many more of them than in CI. So CI is a second line, and the interesting question is what still got past the first one. Over four months, TopSet had about 440 pull requests, and about 1 in 9 of them still hit at least one failed build in CI. Counting every build, reruns included, 53 failed. Type checking stopped 26 of them, the tests 14 and lint 12, and one never got as far as starting. Those numbers need some care. The steps run in order, so a lint failure hides any test failure behind it. A rerun replaces a failed result, so I counted every attempt; only one failure was hidden that way, and it was a real bug I’ll come back to below. Builds that were canceled because a newer push replaced them aren’t counted as failures. And the local run skips the slow and integration tests, so a test failure in CI is often one that couldn’t have shown up locally. A lint or type failure in CI usually means the local check was skipped or ran against something different. I was curious whether the type checker was catching real bugs or just complaining, so I sampled a dozen type-check failures. One was a real bug: a function returned tuples where its callers expected strings, which would have failed as soon as that code ran. The rest were mostly annotations that didn’t match what pandas and numpy actually return. In other words, type checks mostly catch sloppiness and occasionally catch a real bug, and since agents produce a lot of code quickly, catching the sloppiness is worth it. The integration tests drive a full rebalancing against a mock broker that fills orders at random times, in random pieces, at random prices. That randomness is the point, because the interesting bugs live in the timing. The best example was a race in how the system worked out how much cash it had to spend. It read the cash balance and the cost of orders that had already filled, and subtracted one from the other. The problem was that a fill could be recorded between those two reads, because the code that recorded fills and the code that planned purchases held different locks. When that happened, the same cash was counted as available twice, and the bot tried to spend more than it had. The cash position would have gone negative, but a database rule cash can never be below zero stopped it. The first CI run on that pull request failed. The rerun passed. If I had treated it as a flaky test and moved on, the bug would still be there. A test that fails only sometimes is usually reporting a bug that happens only sometimes. Another came from the scheduled weekly run rather than from a pull request. Some integration runs had slowed to about half an hour. The cause was that the code placing trades retried every error, including ones that could never succeed, like a record that didn’t exist. Each hopeless retry tied up a worker for about 25 minutes. The fix, made with Cursor, retries only the errors that can actually clear up. Running the integration tests over and over found the rarer cases: one lock race that showed up once in 223 runs, and an 11-minute stall in 2 of 144 runs that turned out to be a timeout in how the test shut down its workers. I won’t retell the rest here, as the first post https://redgeoff.com/posts/ai-agents-test-gate/ covers the soak. A snapshot test trains a model on a fixed set of data and compares its picks with a reference file that is checked into the repo. If anything about the training changes, the picks change, and the test fails. After I changed the machine-learning code, the snapshots often didn’t match. Most of the time the right move was to fix the code, because the change had done something I didn’t intend. Sometimes the right move was to update the snapshot, but only when the difference was one I expected and understood. Most of this happened on my own machine before a pull request existed, so there isn’t much of a record of it. There is one good example in the record, though. A company that had been renamed was showing up in the historical stock universe under both its old and new tickers, so on some dates the model held it twice and gave it double the weight. When we fixed the universe, the snapshot test failed, and the difference in the reference files showed exactly why: the old files had held one company under two names. That was a case where updating the snapshot was the right call, because the old snapshot had been wrong all along. The leakage checks get one line here because another post https://redgeoff.com/posts/trading-model-leakage/ tells that story: train on part of the data, continue on the rest, and the picks must match training on all of it from scratch, bit for bit. If they don’t, something is leaking from the future. Every test above checks the code against what I believe. The unit tests encode what I think a function should do, and the mock broker encodes what I think the real brokers do. When my belief is wrong, the code and the test are wrong together, and they agree. What I learned about the real brokers is that they do things you would never think to put in a mock. One reuses an error code for two unrelated errors. Another accepts an order and rejects it later. The code for that second broker had been written against its sandbox, a test environment, and production behaved differently. My tests matched what I believed the brokers did. So the check I rely on most compares my records with the broker’s. After trading each day, the system adds up each bot’s positions in my database and compares them with the positions the broker reports for that account. When it processes corporate actions, like mergers, splits and dividends, it also compares what each bot recorded with the account activity the broker reported. Any difference it can’t explain is an error, and errors send an alert to my phone. Here are some of the times those comparisons found something, followed by one time an audit of my own missed something. When a company is bought for cash, shareholders give up their shares and receive cash. My code booked the cash but left the shares in place, because it assumed the broker always sent a separate stock event that removed them. That’s true when a deal pays partly in stock, but in a pure cash deal there is no stock event. No bot had ever held a company in a pure cash deal before, so the assumption had never been tested, and the first one to touch a bot broke it. The tests passed, because they were built on the same assumption. The reconciliation failed the same day, because the broker’s account activity showed the shares leaving and my bot’s records didn’t. The fix removes the shares when there’s no stock event to do it, and it was checked against all 108 cash mergers in the production data rather than just the one that broke. That same merger exposed a second problem. Alpaca reported the account activity on the same day it changed the position. Tradier changed the position about a day before it reported the activity up to two weeks for some corporate actions . Until the activity arrived, my system hadn’t applied the merger, so the reconciliation saw a mismatch. The quick fix was to tolerate mismatches on a symbol with a corporate action still pending. That turned out to be too generous, because it would also hide an unrelated mismatch on the same symbol, so the tolerance now requires the broker’s quantity to be exactly what the pending action predicts. The follow-up found something worse than a false alarm. While the activity was pending, a Tradier bot could plan trades from the old quantity. So symbols with a pending corporate action are now frozen during a rebalancing: not priced, not traded and not counted. One day after trading, the reconciliation reported 10 mismatches in a live account. On a system that trades real money, that reads as your records being wrong in 10 places. Alarms like this are always scary on the surface. Even with the best safeguards in place, there’s always a chance that something has gone wrong. This time I wasn’t too worried, because the totals in my account were still roughly what I expected, so I had a good feeling it was an accounting problem rather than missing money. In other words, there was a second, cruder reconciliation running in my head. That feeling was right. The broker had renamed a company that day, but my list of stocks came from a source that hadn’t caught up yet, so a bot tried to buy the old ticker. The broker rejected the order, the order got stuck, and the rebalancing never finished. The reconciliation then compared the last finished day in my database with a broker account that already showed the 10 orders that had filled that day. So the alarm’s diagnosis was wrong, but the alarm was still worth having, because a rebalancing was hung and the alarm made sure I looked. The fix makes the stock list follow the broker’s renames and makes sure a rejected ticker can’t hang a rebalancing. Checks like these have blind spots too, and the best example is an audit I had built for exactly the kind of problem it then missed. TopSet has a report that compares what the model planned with what the bots actually got. Once, a missing price made one of its key numbers come out blank. An agent added an audit that warned whenever that number was missing, along with seven unit tests for it. Later, a company that had recently been spun off had a gap of 10 trading days in its prices, and two bots I was comparing both held it. This time the missing price arrived as NaN “not a number” instead of an empty value. In Python, NaN counts as true, so the guard that should have skipped the missing price let it through, and that one value wiped out the whole day’s average for every other stock. The result was written as the text “nan”. The audit was checking for an empty value, so it let “nan” straight through. The report’s main numbers were usable on only 0 or 1 of about 13 days, with no warning, and the audit and its tests all passed. It was found by hand, by going through the per-stock data looking for missing prices. Looking back, I don’t think this kind of failure can be caught up front. You need the unexpected data to show up, then you trace through the calculations, and something like an unexplained outlier tips you off. That’s why iterating on the tests matters so much: every edge case that breaks a pipeline becomes the next test, because an audit only checks what you pointed it at. Reconciliation isn’t free. It raises false alarms, like the renamed ticker and the broker that runs a day behind, and every false alarm is a choice between investigating and loosening a tolerance. Loosen it too far and it stops catching anything, which is why the pending-action tolerance had to be tightened again. Each new kind of corporate action also needs its own rules for what the broker should show, and those rules are code with their own bugs. That being said, I’d make the same trade again. A false alarm costs me some time. A silent mismatch between my books and the broker’s could cost real money, and it would keep growing until something else noticed. The tests decide what merges. The reconciliation decides whether I can leave the system alone, and that’s a different question. A one-person system can’t depend on someone watching it every day. It has to notice when it’s wrong and say so. Here is what I’d tell anyone building something like this: TopSet runs unattended today because of that last layer. When something drifts, the broker disagrees with me, and I hear about it the same day. Thanks for reading. This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.