{"slug": "i-stopped-letting-llms-guess-financial-facts", "title": "I stopped letting LLMs guess financial facts", "summary": "A developer built OpenThesis, an Apache-2.0 desktop system for evidence-first, AI-assisted company research, to address the unreliability of LLMs in handling financial facts. The system separates deterministic calculations from probabilistic reasoning, using official filings and validated evidence to ensure traceability and reduce errors.", "body_md": "LLMs can be surprisingly useful for company research. But I kept running into a strange split: parts of the reasoning were useful, while the financial facts underneath them were much harder to trust.\n\nA model could identify an accounting risk in one paragraph, then mix fiscal periods, accounting scopes, or currencies in the next. Missing values might quietly become zeros. A deterministic calculation could be performed probabilistically. A citation could point to a real filing without actually supporting the claim.\n\nThose are different failure modes, and treating all of them as one giant prompting problem did not feel like a reliable architecture.\n\nSo I started building [OpenThesis](https://github.com/zjy1346/OpenThesis), an Apache-2.0 desktop system for evidence-first, AI-assisted company research.\n\nThe project is not a stock picker or a trading bot. The idea is simpler: use ordinary software for work that should be deterministic, and give the LLM a bounded evidence set for the reasoning work where it can actually help.\n\nA common company-research workflow looks roughly like this:\n\n```\ncompany question\n      ↓\n     LLM\n      ↓\n    answer\n```\n\nThat single model call is implicitly responsible for remembering reported values, selecting the right fiscal period, recognizing the accounting scope, finding sources, performing calculations, comparing scenarios, identifying risks, and writing a conclusion.\n\nSome of those tasks are probabilistic by nature. Others are not.\n\nQualitative reasoning, connecting evidence, forming scenarios, and challenging an assumption are reasonable uses of a language model. Remembering an exact reported value, deciding whether a value is missing, and calculating a margin or valuation are poor places to accept probabilistic behavior.\n\nMy design rule became:\n\nDeterministic work should stay deterministic. Use LLMs for reasoning, not as the database and calculator underneath the reasoning.\n\nOpenThesis starts from official filings rather than from model memory or a general web search. The current ingestion paths cover:\n\nThe pipeline is intentionally staged:\n\n```\nofficial filings\n      ↓\nevidence extraction\n      ↓\nvalidated financial facts\n      ↓\ndeterministic finance\n      ↓\nspecialist research agents\n      ↓\nsynthesis\n      ↓\nverification\n      ↓\ntraceable research thesis\n```\n\nDuring a research run, extracted evidence is assigned IDs. Agents receive a bounded evidence set and are expected to attach those IDs to factual claims. Unknown evidence references and unsupported claims can then be flagged instead of being silently accepted because the prose sounds plausible.\n\nThe final synthesis goes through another verification step. If the required evidence coverage is not there, the run can be marked partial rather than being presented as a fully supported report.\n\nThis does not eliminate extraction errors or ambiguous filings. It makes the failure boundary more visible: a reader can inspect what was extracted, what the analysis used, and where support is missing.\n\nThe same separation applies to financial analysis.\n\nOpenThesis builds financial summaries and reverse-DCF calculations in regular code. Missing values are not silently converted to zero. If core coverage is insufficient, the pipeline fails closed before asking an AI model to reason over an invented or incomplete numeric picture.\n\nThat distinction matters more than sophisticated prompting. Consider revenue growth when one period is missing. A fluent model may still produce a percentage because completing patterns is what it does. A deterministic function can return “not available,” preserve the reason, and prevent the result from contaminating downstream analysis.\n\nThe model can still discuss what a reported margin change might imply, compare scenarios, or challenge assumptions. It just does not get to manufacture the inputs or improvise the arithmetic.\n\nOpenThesis has separate roles for financial analysis, business analysis, accounting risk, growth, skepticism, forecasting, synthesis, and verification.\n\nBut “more agents” is not the architectural claim. Eight agents with eight inconsistent versions of the facts would create more confidence without more reliability.\n\nThe useful part is that the roles operate over the same research pack and evidence model. Their boundaries exist because the questions are different:\n\nIn v1.3.0, specialist work is capped at two parallel agents. The goal is not to maximize agent count. It is to make the reasoning roles inspectable and to keep them attached to a shared set of facts.\n\nThe research layer is model-agnostic. The current adapters include hosted providers such as OpenAI, DeepSeek, Gemini, Qwen, Kimi, GLM, and OpenRouter, plus Ollama and custom OpenAI-compatible endpoints.\n\nThat is important for more than cost. It lets the evidence and deterministic-finance layers remain stable while models change. It also makes it possible to run with a local model through Ollama, or to inspect the deterministic pipeline with the offline synthetic demo before configuring any model.\n\nAPI keys are session-only in the current desktop app. They are not written to SQLite, settings, generated reports, or logs. When a hosted model is selected, research context is sent to that provider, so the privacy boundary still depends on the provider you choose.\n\nA generated report is difficult to evaluate if the run configuration disappears.\n\nOpenThesis records the provider and model, parameters, research configuration, evidence and data snapshot, and report language with the run. This is not perfect reproducibility—the behavior of a hosted model can change—but it preserves enough context to explain which data and configuration produced a thesis and to compare runs more honestly.\n\nThe latest formal release is **v1.3.0**. There is a downloadable Windows x64 portable build, and the source combines Python for the research engine, React and TypeScript for the interface, and Rust/Tauri for the desktop shell.\n\nThe offline synthetic demo is the lowest-friction way to inspect the pipeline without an API key or network dependency. A normal research run can ingest filings, create an evidence-backed research pack, run deterministic finance, execute the selected specialist roles, synthesize the result, and verify the final output.\n\n**AI-native, evidence-first company research for long-term investors**\n\nResearch companies—not short-term price movements.\n\nOpenThesis is an open-source desktop research system for individual long-term investors. It turns public filings, deterministic financial analysis, and specialized AI agents into a traceable investment thesis. You choose the model OpenThesis provides the workflow, evidence protocol, financial tools, and reproducibility layer.\n\nImportant\n\nOpenThesis does not connect to brokerage accounts, execute trades, provide short-term signals, or promise investment returns.\n\nNote\n\n**v1.2.0 is the current Windows test release.** In addition to US SEC research\nOpenThesis now researches China A-shares on SSE, SZSE, and BSE, plus Hong Kong\nlistings on HKEX Main Board and GEM. It keeps issuer identity separate from\neach listed security, reads official financial-report PDFs, accepts explicitly\ndated manual market data, and treats banks, insurers, and securities firms as\nFinancials Beta.\n\nOpenThesis is young, and there are real limitations:\n\nThose constraints are part of why I am sharing the project now. I would especially value feedback on the boundary between extraction and evidence, the deterministic checks that should be added next, and filing-analysis failure cases that current verification approaches tend to miss.\n\nOpenThesis is open source under Apache-2.0. If separating evidence from LLM reasoning is useful to you, [take a look at the repository](https://github.com/zjy1346/OpenThesis). Feedback, issues, contributions, and stars are all appreciated.", "url": "https://wpnews.pro/news/i-stopped-letting-llms-guess-financial-facts", "canonical_source": "https://dev.to/zjy1346/i-stopped-letting-llms-guess-financial-facts-2ogl", "published_at": "2026-08-16 12:26:59+00:00", "updated_at": "2026-08-16 12:42:21.910788+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["OpenThesis", "Apache-2.0"], "alternates": {"html": "https://wpnews.pro/news/i-stopped-letting-llms-guess-financial-facts", "markdown": "https://wpnews.pro/news/i-stopped-letting-llms-guess-financial-facts.md", "text": "https://wpnews.pro/news/i-stopped-letting-llms-guess-financial-facts.txt", "jsonld": "https://wpnews.pro/news/i-stopped-letting-llms-guess-financial-facts.jsonld"}}