This is a story about how I spent 21 months building a trading system, wrote down what it would take for it to become a company, and then had two AI models tell me that it hadn’t.
If you’ve read my last three posts, you know TopSet. It’s the system I built, mostly with AI coding agents, to pick a portfolio of stocks with machine-learning models and trade it with real money. The first post was about the automated tests that made agent-written code safe to run, the second was about the ways information from the future leaked into my models, and the third was about noise, and ended with a promise to explain what I decided to do once I knew that my model’s results couldn’t be told apart from luck. This is that post, and you don’t need to have read the others to follow it. There are no charts of returns in it, and that’s on purpose.
I started TopSet because I saw promise in using machine learning to find patterns in the stock market, and my early results looked promising enough that I thought I had something. I also knew that a project like this can absorb years without ever answering the question it was started to answer. So, I gave myself a kill criterion, i.e., a condition, decided ahead of time, under which I would stop: about two years full time to find out whether there was a business here. If there wasn’t, I’d go back to leading engineering teams.
The first live trading bot started on December 2, 2024, and I went full time in January 2025. By September 2026, the system had run continuously on live capital for almost two years, through about 1,600 pull requests (each one a reviewed change to the code), and I was close enough to the deadline that I needed an answer I could trust.
To be honest, I already suspected what the answer would be. Around the beginning of the summer of 2026, I’d started to think that, measured over the long run, my results weren’t statistically strong enough to justify starting a company. But a suspicion is easy to argue yourself out of, especially when you want the other answer, so I wanted a process that would either confirm it or talk me out of it.
Most of the work happened before I asked anything. The obvious way would have been to open a chat and type “here’s my trading system, should I start a company?” I’d have gotten an encouraging answer, because I’d have described the system the way founders describe their systems. Instead, on September 17, 2026, I wrote two documents first, working with a Cursor agent.
The first was an index of everything in the repository: what was built, what was live, and, just as importantly, what had already been tried and abandoned. Every closed experiment was listed with its verdict and a link to where it was decided. The point was that the evaluation would start from the record, not from my pitch.
The second was the evaluation itself, written as a set of gates. Each gate was a question with a written condition for failing it, so the evaluator couldn’t decide what counted as failure after seeing the answer. There were five:
The differentiation gate, for example, failed if the honest pitch collapsed to “levered / concentrated market beta with model complexity attached.” Beta is how much a portfolio moves with the market as a whole. In other words, if the product was really just the stock market with extra risk and a complicated model on top, it failed. That matters because extra risk is cheap to buy: anyone can get more of the market’s ups and downs by adding a little leverage to a low-cost index fund, so a product built on that would be charging for something customers can already get cheaply.
I also wrote down what the evaluator was not allowed to do. The brief at the top says “Do not treat years invested … as evidence of a company.” And a list titled “Anti-patterns (do not do these)” includes “Soften a STOP / JOB SEARCH verdict to protect sunk cost.” Sunk cost is the time and money already spent, which can’t be recovered whatever you decide next. The person most likely to want that verdict softened was me.
There’s a small irony in that brief. Under go-to-market, i.e., how customers would find the product, I’d written that I would attract them by blogging about building AI-powered investment products. Here I am, blogging about it.
About half an hour after those documents were committed, Claude Fable 5.1, running in Claude Code, worked through the evaluation against the index, the issues and pull requests it pointed to, and its own web research on competitors and regulation. The next morning, Cursor’s agent, Composer, did the same evaluation independently.
Both came back with the same verdict: “personal only.” Keep running the system on my own money, and don’t turn it into a company. The first memo spelled it out: don’t found one now, and don’t raise money. It summed up the honest pitch as “concentrated, high-turnover S&P exposure with ML attached, viable only inside an IRA,” which is very close to the failure condition I’d written for the first gate. It also recommended that I start the job search right away, rather than waiting for my two-year mark.
That phrase “viable only inside an IRA” needs some context. My system rebalances roughly once a month, which means it sells stocks it has held for less than a year. In an ordinary brokerage account, those profits are short-term capital gains, which in the US are taxed at the same rate as income, every year, and that tax drag erases a lot of the advantage of trading actively. In a retirement account like an IRA, trades aren’t taxed as they happen, so that drag disappears. So, any product built on my system would have had to be sold into retirement accounts.
I was heartbroken. I was 21 months in, and I felt close to something big. The answer was that launching a financial product on what I had wasn’t justified.
Yes, and that was the first thing I learned once I started pushing back.
On September 20, I had Claude Opus review both memos. It found the same two mistakes in each. The first was that both memos described the strategy as riskier than the version I actually run, calling it levered or high-beta when it moves with the market about one for one.
The second mistake was about those retirement accounts. Both memos said they were small by nature, because the government caps how much you can contribute to an IRA each year. That’s true for new contributions, but it misses rollover IRAs. When people leave a job, they often move their whole 401(k), i.e., their workplace retirement savings, into an IRA, and a rollover isn’t subject to the yearly cap. So, the accounts the product would have been sold into could be much larger than the memos assumed. The correction added to both memos says so plainly.
Neither mistake changed the verdict. But they changed how I read the agreement. Two models that read the same index and the same prompt are not two independent opinions. They share the blind spots of the record they were given, and, as the IRA mistake showed, they can also share a wrong assumption about the world that wasn’t in the record at all. Agreement between them is worth something, just much less than it feels like when you first read it.
So, I did with the memos what I’d started doing with my research: I reviewed them again with a different model, and I kept going until the review stopped finding anything major.
I didn’t accept it. I spent days trying to poke holes in it, and every time I thought of a reason the memos might be wrong, I had it investigated. I didn’t sit down with a list of counterarguments. Each one was a follow-up thought I had while I was digesting the idea that there wasn’t going to be a company.
Each one also got written down, in a section of the memo called “Arguments tried and retired,” with a rule at the top: “A new argument should say which of these it is not.” In other words, before I could spend another afternoon on an idea, I had to check that I hadn’t already lost that argument once.
By October 10, there were 25 entries. Here are a few of them, so you get a feel for it:
A handful were kept in part, because they fixed something the memos had gotten wrong. A few are still open, including two that only a securities lawyer can answer. None of them moved the verdict.
On September 22, I ran an evidence review that went my way, four times in a row. Each of the four was a correction, and each one made the project look better than the memos had said. One of them confirmed that my execution costs, i.e., what I lose between deciding to trade and actually getting filled at the broker, were essentially zero, which is a measurement I’d spent months building the tooling to make.
The pull request I opened that day was drafted to change the verdict from “personal
only” to “90-day discovery,” i.e., go and find out whether real customers would pay.
Then I checked the biggest of the four results on a second, longer set of price
history. The first set covered about 8 years and the second about 24. On the 24 years, the result didn’t hold. It was the second favorable
result in three days to fail on the longer history. The pull request that merged says,
near the top, “This PR was originally drafted as a revision to 90-DAY DISCOVERY on the
eight-year evidence. Checking the 24-year pool reversed it.”
If you haven’t spent time with market data, this pattern is worth knowing about, because it’s the rule rather than the exception. Over a few years, it’s surprisingly easy to find a way of picking stocks that beats the market, partly because there are so many ways to try and some of them will win by chance. Over longer periods, most of those advantages shrink or disappear. One well-known study, by David McLean and Jeffrey Pontiff in the Journal of Finance in 2016, looked at 97 stock-market patterns that academic papers had found to predict returns. After the papers were published, the patterns earned, on average, 58% less than they had in the original research.
It was incredibly disappointing. As an entrepreneur, it’s very exciting when you feel like you’ve discovered something significant, and when it turns out not to be significant, it’s hard to come down from that high. Higher highs and lower lows are the life of an entrepreneur.
Looking back at all 25 arguments, the pattern is simple. Packaging, pricing, partners and the sales channel can all change the business questions. Only time, with real money and results that can’t be revised after the fact, changes the evidence question, and nothing I could build or buy makes that time pass faster.
The reason it takes so long comes down to signal and noise. If a strategy’s average advantage is small compared with how much its results swing from year to year, then a few good years look the same as a few lucky ones. At the size of advantage on offer here, and with how much a concentrated portfolio, one that holds relatively few stocks, swings around the market, telling skill from luck with live money would take decades, not quarters. That’s the same wall the momentum index in my last post hit, even after I reconstructed 24 years of its history.
My system has now run continuously on live capital for almost two years, and on the evidence so far, its results are not statistically distinguishable from cheaper alternatives. In other words, the difference is too small, compared with the noise, to rule out luck. Even the most celebrated stock pickers go through long stretches of trailing the index. A non-professional launching a financial product on that evidence just didn’t hold up.
Somewhere in the middle of the arguing, after the first memo was drafted, I read Burton Malkiel’s A Random Walk Down Wall Street. I’d read a lot of finance books, but never this one. Malkiel’s argument, first published in 1973 and updated through many editions since, is that markets absorb new information so quickly that consistently beating them is extremely hard, so that an investor who buys and holds a low-cost, diversified index fund is likely to do better than most portfolios carefully picked by professionals. I generally knew that broad index funds are near impossible to beat, but hearing it laid out with that much research behind it was very helpful. I felt like I’d spent nearly two years proving the thesis of the book!
Some people will read this as 21 months spent proving a negative. I don’t regret it. I went deep on the principles behind quantitative trading: finding and storing data you can trust, building machine-learning pipelines that are both fast and reliable, and building execution that’s fast and reliable with real money on the line. I learned a ton. That being said, I do wish the outcome had been different.
It’s always better if you can validate an idea up front, so that you don’t spend more time than you need to chasing something that won’t work. The kill criterion and the gates were my attempt at that. But I also don’t think I could have been confident either way without doing the work. Sometimes failing along the way, and the hindsight that comes with it, is the only way to know.
I stopped building toward a company. The system keeps running on my own accounts, and I still do research on the side in between other things. The evidence question stays open, and if it’s ever answered, it will be answered by time rather than by another memo.
Next, I’m going back to leading engineering teams, and I’m taking with me the habits that made this decision possible: writing the gates before looking at the result, holding my own ideas to the same evidence bar I’d hold anyone else’s, and treating a pre-committed stop as something to carry out rather than renegotiate.
Thanks for reading.
This describes software I run on my own accounts. It is not investment advice and I do not offer investment services.