# FinDendro Benchmark: LLMs Misreport What They Know in Financial Backtests

> Source: <https://explainx.ai/blog/findendro-llm-temporal-self-knowledge-financial-backtesting-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-10 00:00:00+00:00

If you backtest an LLM on **historical Fed decisions** or **month-end stock prices**, the model might **recite outcomes** instead of forecasting them. Practitioners try to prevent that with **vendor cutoffs**, **"answer as of date X"** prompts, or **"do you know this?"** checks.

A new paper, **[LLMs Don't Know What They Know](https://arxiv.org/abs/2611.06808?ref=explainx)** (arXiv **2611.06808**, posted on [Hugging Face Papers](https://huggingface.co/papers/2611.06808?ref=explainx)), introduces **FinDendro** and argues those safeguards often **fail behavioral tests**. The authors measure **what models answer** and **what they claim** on the same dated facts.

## TL;DR

| Safeguard | FinDendro finding (paper) | 
|---|---|
| Vendor cutoff dates | Llama-3.3-70B accuracy falls to chance by mid-2021 while claims persist into 2023 | 
| Self-reported knowledge | GPT-5 is right on 76% of controlled facts but says it knows only 38% | 
| Historical-date instruction | Cuts GPT-5 "I know" claims from 39% to 1% with little accuracy change | 

## Why financial LLM backtests need temporal audits

Trading and macro demos on social media often show models "predicting" past crises with spooky accuracy. The uncomfortable explanation is **hindsight leakage**: weights may already encode the realized CPI print or the hike the Fed actually delivered.

Industry benchmarks like **FinBen** and **StockBench** score economic skill, but the FinDendro authors note many suites do not jointly test **knowledge claims** and **factual recall** on identically dated items. Our earlier [Cutoff Bench](https://explainx.ai/blog/cutoff-bench-llm-knowledge-cutoff-reported-vs-remembered-2026) post asked a simpler question — whether models repeat their **card cutoff** — without pairing **know/guess** language on macro facts.

FinDendro closes part of that gap by building **2,226 US facts from 1999 through 2026** across:

- Twelve monthly **FRED** macro series (first-release values)
- **Month-end prices** for 50 large US stocks (split-adjusted back to quoted history)
- **Federal Reserve** target-rate decisions

Each fact spawns **four prompts**: a straight value question, recall and predict variants that pair answers with **know versus guess** claims, and a predict variant with an explicit **historical date** instruction.

Chance accuracy stays near **0.27** after controls for persistence and dominant policy outcomes, per the abstract.

Timeline marker at a model knowledge cutoff boundary for dated financial probes

## Three failure modes the paper reports

### Self-report understates knowledge

On controlled facts, **GPT-5** answers **76%** correctly while claiming knowledge on only **38%**, according to the [paper abstract](https://arxiv.org/abs/2611.06808?ref=explainx). Worse, it still answers **61%** of items it labels as guesses correctly — so filtering on "I do not know" does not reliably drop leaked outcomes.

That matches a builder intuition many teams already have: models sound humble while citing exact digits.

### Stated vendor cutoffs miss behavioral ends

The authors report **Llama-3.3-70B** factual accuracy collapses toward chance after **mid-2021**, before its **December 2023** vendor cutoff, while knowledge claims continue into **early 2023**. If you backtest only after the card date, you may still be inside the model's effective memory window.

### Date instructions re-label answers, not skill

When GPT-5 must predict **as of** a historical date, claimed prior knowledge falls from **39% to 1%**, yet accuracy barely moves. The model sounds like it is forecasting while behaving like it is recalling — exactly the rhetoric risk compliance teams worry about in [claimed-versus-verified finance posts](https://explainx.ai/blog/metr-anthropic-ai-rd-acceleration-twice-as-fast-80-percent-ai-redactions-claimed-vs-verified-2026).

Benchmark ceiling chart metaphor for LLM accuracy versus self-reported knowledge claims

## What Hugging Face readers asked

On the [paper discussion](https://huggingface.co/papers/2611.06808?ref=explainx), commenter **PolyHelperAI** asked whether the historical-date instruction helped at all. The authors' framing in the abstract implies it mainly **suppresses claims**, not errors — worth reproducing on your own evaluator before you trust date-stamped prompts in production.

## How practitioners should respond

FinDendro does not ban historical evaluation. It pushes **behavioral measurement**:

1. **Pair accuracy curves with claim curves** on the same dated items instead of trusting a single cutoff field.
2. **Hold out recent months prospectively** when you need clean skill estimates — same lesson as[prospective agent benchmarks](https://explainx.ai/blog/streamvibe-proactive-video-agent-google-research-when-to-speak-benchmark-2026) waiting for outcomes.
3. **Document which FRED release** you score against; FinDendro uses**first-release** values, so revised series would change labels.
4. **Compare point-in-time fine-tunes** separately; the paper evaluates small PIT families whose accuracy stays near chance on both sides of training cuts — specialized, not a default fix.

If you fine-tune open models on financial text, treat **leakage tests** as part of release QA beside perplexity and tool accuracy — parallel to how we discuss [verification tax](https://explainx.ai/blog/ai-verification-tax-rework-sage-workday-foxit-surveys-how-to-cut-it-2026) for agent output.

## Task design: why four prompts per fact

FinDendro is built to test **safeguards practitioners already use**:

| Task | What it probes | 
|---|---|
| Task 1 — value only | Behavioral knowledge without a claim | 
| Task 2 — value plus know/guess | Whether self-report tracks accuracy | 
| Task 3 — dated predict | Whether "as of" instructions reduce leakage or only re-label it | 

The paper contrasts FinDendro with **FinBen**, **StockBench**, **ForecastBench**, **HindsightBench**, and **SelfAware** in a comparison table. FinDendro is the row that combines **dated finance**, **cutoff estimation**, **claims**, and **guard evaluation** — a niche overlap, but exactly the gap teams cite when a backtest looks too good.

Authors use **first-release FRED observations** via ALFRED, **split-adjusted Yahoo prices** as quoted at month-end, and **Federal Reserve target-rate history**. That matters because revised macro series would change labels; if you replicate, mirror their release policy.

Sampling is **sparse before 2021** and **dense from 2021 onward** because frontier cutoffs cluster in that window. Of **2,226** facts, **1,340** count as **controlled** after persistence checks — the subset where repeating last month's value is not a winning strategy.

## Numbers worth copying into your evaluator README

From the abstract and introduction we fetched:

- Chance accuracy near **0.27** after controls.
- **GPT-5** :**76%** controlled accuracy versus**38%** claimed knowledge;**61%** correct on items it says it does not know.
- **Historical-date instruction** : claimed prior knowledge**39% → 1%** for GPT-5 with**little accuracy movement** .
- **Llama-3.3-70B** : accuracy to chance by**mid-2021** while claims extend into**2023** , ahead of a**December 2023** vendor cutoff cited in the paper.

Those figures are **paper-reported**; treat them like any external benchmark until you reproduce on your harness.

## When FinDendro matters even if you never trade

You do not need a brokerage account for the lesson to land:

- **Risk teams** backtesting LLM macro summaries for internal memos still leak if the model saw COVID-era CPI in weights.
- **Agent evals** that score "forecasting" on historical quarters inherit the same bias as[OR-License style operations tasks](https://explainx.ai/blog/or-license-benchmark-llm-agents-operations-research-practitioners-claude-code-daq-2026) .
- **RAG pipelines** that filter documents by date still leave**parametric memory** untouched — FinDendro is a reminder to separate**retrieval cutoff** from**weight cutoff** .

Pair FinDendro with **prospective** benchmarks when you need marketing claims: hold out the last **N** months, publish predictions before outcomes, then score — slower, but legible to auditors.

## Implementation ideas for ML platform teams

If you maintain an internal model garden:

1. **Snapshot FinDendro-style MCQ** on release candidates weekly; track claim-accuracy gap as a regression signal.
2. **Alert when gap widens** after a silent router change — same discipline as monitoring tool hallucination rate.
3. **Store vendor cutoff field separately** from**empirical cutoff** you estimate from dated probes; show both in an internal model card.
4. **Ban "predict 2020 CPI" demos** in customer decks unless the demo uses models trained with provable exclusion — marketing risk is reputational, not just statistical.

## Limits and open questions

The benchmark is **US-only** and **multiple-choice styled** with single letter answers — cheap to run, but not identical to open-ended trading policies. The full pool and construction code are promised at release; until then, replicate tables from the PDF before you cite leaderboard ranks.

The paper evaluates a handful of frontier and open models; your fine-tune may behave differently. Still, the **mechanism** — claims diverging from accuracy — is the reusable lesson.

**Open questions** we will watch for in follow-on work: multilingual macro facts, intraday prices, options surfaces, and whether **tool-using agents** behave differently when they can fetch ALFRED at runtime (FinDendro prompts forbid browsing in the described setup).

## Glossary alignment for eval teams

If your organization maintains a model card template, add two fields today: **vendor_stated_cutoff** and **empirical_cutoff_fin_dendro** (or your internal probe name). Require both before a model is marked **approved for historical macro demos**. That single bureaucracy line prevents sales engineers from citing vendor cards alone when FinDendro-style gaps appear in QA.

Researchers evaluating **central bank communication** tools should note FinDendro includes **FOMC decision months** as first-class facts. Stock pickers should treat the fifty large-cap tickers the same way: month-end labels are unforgiving when a model silently remembers earnings-week prices. A model that "predicts" past rate moves correctly may still be useless for live policy if the measured boundary shows recall years after the meeting — exactly the scenario monetary economists worry about when LLM summaries enter staff briefing pipelines without audit trails.

## Related reading

## Sources

- [arXiv 2611.06808](https://arxiv.org/abs/2611.06808?ref=explainx) , October 30, 2026 posting.
- [Hugging Face Papers page](https://huggingface.co/papers/2611.06808?ref=explainx) , community discussion November 2026.

*Benchmark scope and model names are accurate as of the paper text we fetched on November 10, 2026.*