{"slug": "findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests", "title": "FinDendro Benchmark: LLMs Misreport What They Know in Financial Backtests", "summary": "A new arXiv paper, 2611.06808, titled \"LLMs Don't Know What They Know,\" introduces the FinDendro benchmark, built from 2,226 US facts spanning 1999 through 2026 across twelve monthly FRED macro series, month-end prices for 50 large US stocks, and Federal Reserve target-rate decisions, and reports that common safeguards against hindsight leakage in financial backtests often fail behavioral tests. The authors find GPT-5 answers 76% of controlled facts correctly while claiming knowledge on only 38%, and still answers 61% of items it labels as guesses correctly; Llama-3.3-70B's factual accuracy falls toward chance after mid-2021, before its December 2023 vendor cutoff, while its knowledge claims persist into early 2023. Instructing GPT-5 to predict as of a historical date cuts its \"I know\" claims from 39% to 1% with little change in accuracy, meaning the model sounds like it is forecasting while behaving like it is recalling.", "body_md": "If you backtest an LLM on **historical Fed decisions** or **month-end stock prices**, the model might **recite outcomes** instead of forecasting them. Practitioners try to prevent that with **vendor cutoffs**, **\"answer as of date X\"** prompts, or **\"do you know this?\"** checks.\n\nA new paper, **[LLMs Don't Know What They Know](https://arxiv.org/abs/2611.06808?ref=explainx)** (arXiv **2611.06808**, posted on [Hugging Face Papers](https://huggingface.co/papers/2611.06808?ref=explainx)), introduces **FinDendro** and argues those safeguards often **fail behavioral tests**. The authors measure **what models answer** and **what they claim** on the same dated facts.\n\n## TL;DR\n\n| Safeguard | FinDendro finding (paper) | \n|---|---|\n| Vendor cutoff dates | Llama-3.3-70B accuracy falls to chance by mid-2021 while claims persist into 2023 | \n| Self-reported knowledge | GPT-5 is right on 76% of controlled facts but says it knows only 38% | \n| Historical-date instruction | Cuts GPT-5 \"I know\" claims from 39% to 1% with little accuracy change | \n\n## Why financial LLM backtests need temporal audits\n\nTrading and macro demos on social media often show models \"predicting\" past crises with spooky accuracy. The uncomfortable explanation is **hindsight leakage**: weights may already encode the realized CPI print or the hike the Fed actually delivered.\n\nIndustry benchmarks like **FinBen** and **StockBench** score economic skill, but the FinDendro authors note many suites do not jointly test **knowledge claims** and **factual recall** on identically dated items. Our earlier [Cutoff Bench](https://explainx.ai/blog/cutoff-bench-llm-knowledge-cutoff-reported-vs-remembered-2026) post asked a simpler question — whether models repeat their **card cutoff** — without pairing **know/guess** language on macro facts.\n\nFinDendro closes part of that gap by building **2,226 US facts from 1999 through 2026** across:\n\n- Twelve monthly **FRED** macro series (first-release values)\n- **Month-end prices** for 50 large US stocks (split-adjusted back to quoted history)\n- **Federal Reserve** target-rate decisions\n\nEach fact spawns **four prompts**: a straight value question, recall and predict variants that pair answers with **know versus guess** claims, and a predict variant with an explicit **historical date** instruction.\n\nChance accuracy stays near **0.27** after controls for persistence and dominant policy outcomes, per the abstract.\n\nTimeline marker at a model knowledge cutoff boundary for dated financial probes\n\n## Three failure modes the paper reports\n\n### Self-report understates knowledge\n\nOn controlled facts, **GPT-5** answers **76%** correctly while claiming knowledge on only **38%**, according to the [paper abstract](https://arxiv.org/abs/2611.06808?ref=explainx). Worse, it still answers **61%** of items it labels as guesses correctly — so filtering on \"I do not know\" does not reliably drop leaked outcomes.\n\nThat matches a builder intuition many teams already have: models sound humble while citing exact digits.\n\n### Stated vendor cutoffs miss behavioral ends\n\nThe authors report **Llama-3.3-70B** factual accuracy collapses toward chance after **mid-2021**, before its **December 2023** vendor cutoff, while knowledge claims continue into **early 2023**. If you backtest only after the card date, you may still be inside the model's effective memory window.\n\n### Date instructions re-label answers, not skill\n\nWhen GPT-5 must predict **as of** a historical date, claimed prior knowledge falls from **39% to 1%**, yet accuracy barely moves. The model sounds like it is forecasting while behaving like it is recalling — exactly the rhetoric risk compliance teams worry about in [claimed-versus-verified finance posts](https://explainx.ai/blog/metr-anthropic-ai-rd-acceleration-twice-as-fast-80-percent-ai-redactions-claimed-vs-verified-2026).\n\nBenchmark ceiling chart metaphor for LLM accuracy versus self-reported knowledge claims\n\n## What Hugging Face readers asked\n\nOn the [paper discussion](https://huggingface.co/papers/2611.06808?ref=explainx), commenter **PolyHelperAI** asked whether the historical-date instruction helped at all. The authors' framing in the abstract implies it mainly **suppresses claims**, not errors — worth reproducing on your own evaluator before you trust date-stamped prompts in production.\n\n## How practitioners should respond\n\nFinDendro does not ban historical evaluation. It pushes **behavioral measurement**:\n\n1. **Pair accuracy curves with claim curves** on the same dated items instead of trusting a single cutoff field.\n2. **Hold out recent months prospectively** when you need clean skill estimates — same lesson as[prospective agent benchmarks](https://explainx.ai/blog/streamvibe-proactive-video-agent-google-research-when-to-speak-benchmark-2026) waiting for outcomes.\n3. **Document which FRED release** you score against; FinDendro uses**first-release** values, so revised series would change labels.\n4. **Compare point-in-time fine-tunes** separately; the paper evaluates small PIT families whose accuracy stays near chance on both sides of training cuts — specialized, not a default fix.\n\nIf you fine-tune open models on financial text, treat **leakage tests** as part of release QA beside perplexity and tool accuracy — parallel to how we discuss [verification tax](https://explainx.ai/blog/ai-verification-tax-rework-sage-workday-foxit-surveys-how-to-cut-it-2026) for agent output.\n\n## Task design: why four prompts per fact\n\nFinDendro is built to test **safeguards practitioners already use**:\n\n| Task | What it probes | \n|---|---|\n| Task 1 — value only | Behavioral knowledge without a claim | \n| Task 2 — value plus know/guess | Whether self-report tracks accuracy | \n| Task 3 — dated predict | Whether \"as of\" instructions reduce leakage or only re-label it | \n\nThe paper contrasts FinDendro with **FinBen**, **StockBench**, **ForecastBench**, **HindsightBench**, and **SelfAware** in a comparison table. FinDendro is the row that combines **dated finance**, **cutoff estimation**, **claims**, and **guard evaluation** — a niche overlap, but exactly the gap teams cite when a backtest looks too good.\n\nAuthors use **first-release FRED observations** via ALFRED, **split-adjusted Yahoo prices** as quoted at month-end, and **Federal Reserve target-rate history**. That matters because revised macro series would change labels; if you replicate, mirror their release policy.\n\nSampling is **sparse before 2021** and **dense from 2021 onward** because frontier cutoffs cluster in that window. Of **2,226** facts, **1,340** count as **controlled** after persistence checks — the subset where repeating last month's value is not a winning strategy.\n\n## Numbers worth copying into your evaluator README\n\nFrom the abstract and introduction we fetched:\n\n- Chance accuracy near **0.27** after controls.\n- **GPT-5** :**76%** controlled accuracy versus**38%** claimed knowledge;**61%** correct on items it says it does not know.\n- **Historical-date instruction** : claimed prior knowledge**39% → 1%** for GPT-5 with**little accuracy movement** .\n- **Llama-3.3-70B** : accuracy to chance by**mid-2021** while claims extend into**2023** , ahead of a**December 2023** vendor cutoff cited in the paper.\n\nThose figures are **paper-reported**; treat them like any external benchmark until you reproduce on your harness.\n\n## When FinDendro matters even if you never trade\n\nYou do not need a brokerage account for the lesson to land:\n\n- **Risk teams** backtesting LLM macro summaries for internal memos still leak if the model saw COVID-era CPI in weights.\n- **Agent evals** that score \"forecasting\" on historical quarters inherit the same bias as[OR-License style operations tasks](https://explainx.ai/blog/or-license-benchmark-llm-agents-operations-research-practitioners-claude-code-daq-2026) .\n- **RAG pipelines** that filter documents by date still leave**parametric memory** untouched — FinDendro is a reminder to separate**retrieval cutoff** from**weight cutoff** .\n\nPair FinDendro with **prospective** benchmarks when you need marketing claims: hold out the last **N** months, publish predictions before outcomes, then score — slower, but legible to auditors.\n\n## Implementation ideas for ML platform teams\n\nIf you maintain an internal model garden:\n\n1. **Snapshot FinDendro-style MCQ** on release candidates weekly; track claim-accuracy gap as a regression signal.\n2. **Alert when gap widens** after a silent router change — same discipline as monitoring tool hallucination rate.\n3. **Store vendor cutoff field separately** from**empirical cutoff** you estimate from dated probes; show both in an internal model card.\n4. **Ban \"predict 2020 CPI\" demos** in customer decks unless the demo uses models trained with provable exclusion — marketing risk is reputational, not just statistical.\n\n## Limits and open questions\n\nThe benchmark is **US-only** and **multiple-choice styled** with single letter answers — cheap to run, but not identical to open-ended trading policies. The full pool and construction code are promised at release; until then, replicate tables from the PDF before you cite leaderboard ranks.\n\nThe paper evaluates a handful of frontier and open models; your fine-tune may behave differently. Still, the **mechanism** — claims diverging from accuracy — is the reusable lesson.\n\n**Open questions** we will watch for in follow-on work: multilingual macro facts, intraday prices, options surfaces, and whether **tool-using agents** behave differently when they can fetch ALFRED at runtime (FinDendro prompts forbid browsing in the described setup).\n\n## Glossary alignment for eval teams\n\nIf your organization maintains a model card template, add two fields today: **vendor_stated_cutoff** and **empirical_cutoff_fin_dendro** (or your internal probe name). Require both before a model is marked **approved for historical macro demos**. That single bureaucracy line prevents sales engineers from citing vendor cards alone when FinDendro-style gaps appear in QA.\n\nResearchers evaluating **central bank communication** tools should note FinDendro includes **FOMC decision months** as first-class facts. Stock pickers should treat the fifty large-cap tickers the same way: month-end labels are unforgiving when a model silently remembers earnings-week prices. A model that \"predicts\" past rate moves correctly may still be useless for live policy if the measured boundary shows recall years after the meeting — exactly the scenario monetary economists worry about when LLM summaries enter staff briefing pipelines without audit trails.\n\n## Related reading\n\n## Sources\n\n- [arXiv 2611.06808](https://arxiv.org/abs/2611.06808?ref=explainx) , October 30, 2026 posting.\n- [Hugging Face Papers page](https://huggingface.co/papers/2611.06808?ref=explainx) , community discussion November 2026.\n\n*Benchmark scope and model names are accurate as of the paper text we fetched on November 10, 2026.*", "url": "https://wpnews.pro/news/findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests", "canonical_source": "https://explainx.ai/blog/findendro-llm-temporal-self-knowledge-financial-backtesting-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-11-10 00:00:00+00:00", "updated_at": "2026-11-10 02:50:16.118638+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence", "machine-learning"], "entities": ["FinDendro", "GPT-5", "Llama-3.3-70B", "Federal Reserve", "FRED", "arXiv", "Hugging Face", "FinBen"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests", "markdown": "https://wpnews.pro/news/findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests.md", "text": "https://wpnews.pro/news/findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests.txt", "jsonld": "https://wpnews.pro/news/findendro-benchmark-llms-misreport-what-they-know-in-financial-backtests.jsonld"}}