{"slug": "llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for", "title": "LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting", "summary": "A new arXiv paper (2611.06808v1) introduces FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, and finds that frontier LLMs' self-reported knowledge understates what they actually know, undermining common safeguards for financial backtesting. GPT-5 answered 76% of controlled facts correctly but claimed to know only 38%, and answered 61% of the facts it said it did not know; knowledge claims persisted beyond the measured knowledge boundary by up to seven quarters, while declaring a historical date removed most claims of prior knowledge without changing accuracy. The authors conclude that backtests inside a model's training window cannot be certified leak-free by self-report, instruction or a vendor's stated cutoff, and propose behavioral measurement of temporal knowledge boundaries instead.", "body_md": "arXiv:2611.06808v1 Announce Type: new \nAbstract: Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.", "url": "https://wpnews.pro/news/llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for", "canonical_source": "https://www.machinebrief.com/news/llms-dont-know-what-they-know-benchmarking-temporal-self-kno-0qbq", "published_at": "2026-11-05 05:00:00+00:00", "updated_at": "2026-11-05 07:21:22.632590+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "artificial-intelligence", "machine-learning"], "entities": ["FinDendro", "GPT-5", "arXiv", "Federal Reserve"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for", "markdown": "https://wpnews.pro/news/llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for.md", "text": "https://wpnews.pro/news/llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for.txt", "jsonld": "https://wpnews.pro/news/llms-don-t-know-what-they-know-benchmarking-temporal-self-knowledge-for.jsonld"}}