{"slug": "the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9", "title": "The most-starred LLM trading paper claims buy-and-hold lost 5.23%. It actually gained 9.12%.", "summary": "A developer found that the most-starred LLM trading paper TradingAgents (arXiv:2412.20138) reported a 5.23% loss for AAPL buy-and-hold, but actual dividend-adjusted data shows a 9.12% gain. The developer pre-committed a test to a public repo, verifying the baseline error of roughly 14 percentage points. The paper's central claim that 'buy-and-hold lost money, our agent made 26%' is contradicted by the price series.", "body_md": "TradingAgents (arXiv:2412.20138, ~94,700 GitHub stars) is probably the most visible\n\n\"LLM agents trade stocks\" project right now. The headline result: a multi-agent LLM\n\nframework earned a 26.62% cumulative return on AAPL with a Sharpe ratio of 8.21 between\n\n2024-06-19 and 2024-11-19, while buy-and-hold supposedly lost 5.23% over the same\n\nwindow.\n\nThat last number takes one API call to check. So I checked it. With a twist: I wrote\n\ndown the test, the verdict rule, and my predicted outcome, and committed all of it to a\n\npublic repo before touching any data.\n\nFrom commit [ 60e8e2b](https://github.com/trow126/falsify-ledger/commit/60e8e2b),\n\nRecompute AAPL buy-and-hold total return over 2024-06-19 → 2024-11-19 using Yahoo\n\nFinance dividend-adjusted daily closes. KILL if the result is ≥ 0.0% — because then\n\nthe stated −5.23% baseline is off by more than five points and has the wrong sign.\n\nPredicted P(kill): 0.90.\n\n| Series | 2024-06-20 | 2024-11-19 | Buy & hold |\n|---|---|---|---|\n| Adjusted close | 207.88 | 226.83 | +9.12% |\n| Raw close (robustness) | 209.68 | 228.28 | +8.87% |\n\nAAPL went up about nine percent over the paper's own evaluation window. The reported\n\nbaseline is off by roughly 14 percentage points, sign included. Whatever the agent\n\nreturned, the paper's central contrast — \"buy-and-hold lost money, our agent made 26%\"\n\n— is built on a number that the price series contradicts. Verdict: KILL. Full record\n\nand reproduction script are in\n\n[verification/01](https://github.com/trow126/falsify-ledger/blob/main/verification/01/result.md).\n\nA few things I noted but kept out of the verdict: a Sharpe of 8.21 over ~105 trading\n\ndays with a 0.91% max drawdown implies an equity curve that barely ever dips, which in\n\na single-name equity strategy usually points to an evaluation artifact rather than\n\nalpha. The repo's own issue tracker has a look-ahead bias report (Issue #203). Costs\n\naren't modeled.\n\nBecause post-hoc criticism is cheap. Nobody can tell whether a critic ran ten tests and\n\npublished the one that landed. Writing the prediction down first — with a probability\n\nattached — makes the critic auditable too.\n\nI keep these in a public ledger. Five entries so far, each with the verdict rule\n\ncommitted before the data was touched. Predictions: 0.90 / 0.95 / 0.30 / 0.85 / 0.70.\n\nOutcomes: KILL / KILL / SURVIVE / SURVIVE / KILL. The survivors are real: entry 03 was\n\na volatility strategy I expected to pass, and it reproduced almost exactly\n\n([entry](https://github.com/trow126/falsify-ledger/blob/main/entries/03-concretum-vix-strategy.md)).\n\nEntry 04 survived its frozen test on a technicality in my own cost wording, and the\n\nentry says so right at the verdict, along with the reading under which it would have\n\nfailed. Misses cost me Brier score, publicly.\n\nIf you're about to commit money or engineering time to a quantitative claim — yours or\n\na vendor's — and want it tested this way first,\n\n[open an issue](https://github.com/trow126/falsify-ledger/issues). The verdict rule\n\ngets frozen before either of us sees the result.\n\n*This is methodological review of public claims. It is not investment advice, and it is\nnot directed at any individual.*", "url": "https://wpnews.pro/news/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9", "canonical_source": "https://dev.to/trow126/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-523-it-actually-gained-912-1jj6", "published_at": "2026-07-27 20:50:37+00:00", "updated_at": "2026-07-27 21:30:51.323556+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents"], "entities": ["TradingAgents", "arXiv", "AAPL", "Yahoo Finance", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9", "markdown": "https://wpnews.pro/news/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9.md", "text": "https://wpnews.pro/news/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9.txt", "jsonld": "https://wpnews.pro/news/the-most-starred-llm-trading-paper-claims-buy-and-hold-lost-5-23-it-actually-9.jsonld"}}