GlassBallAI: a live, time-locked dataset of Gemini forecasts for studying LLM behaviour

A new dataset called GlassBallAI captures live, time-locked forecasts from Google's Gemini LLMs on stock market direction and sentiment, with over 3,600 rows spanning 90+ days. The dataset reveals a confidence/accuracy inversion where the model's highest-confidence predictions show the lowest accuracy, raising questions about LLM calibration.

The core idea: capture an LLM’s forecasting behavior before the outcome is known, rather than testing on resolved historical data. Each day, Gemini 2.5 Flash, 2.5 Pro, 2.5 Flash Lite, and now 3 Flash Preview generates a 10-trading-day lookahead for price direction, sentiment, and confidence, with full reasoning traces and cited search snippets. Time-locked, so there’s no retrofitting the answer. 90+ days of runs so far Feb 17 – May 19, 2026 for the Flash subset , ~3,655 rows total across all model configs. Dataset card includes full schema docs, a Colab quickstart for hydrating ground truth, and an explicit non-financial-advice / legal framework since this touches market data. One finding I’d like feedback on: accuracy drops to its second-lowest point ~28% exactly in the model’s highest-confidence bin 0.8–0.9 — a confidence/accuracy inversion. Global ECE is 0.217. Sample size in that bin is small, so I’m not overclaiming, but curious if others have seen similar patterns in grounded-LLM calibration work. There’s also a companion site with a dashboard, methodology writeup and results glassballai.com/results https://glassballai.com/results . Note Evaluation: Some tickers have very low run counts due to interrupted tracking or individual tracking runs that are not part of the fixed set of tracked stocks. They are included for full transparency and factor into the global metrics, but their individual ticker-level stats should be ignored due to high variance. Note Custom Tracing Run: the “run your own session” feature is temporarily off while I sort out API costs at scale. Dataset: huggingface.co/datasets/louidev/glassballai http://huggingface.co/datasets/louidev/glassballai CC-BY-NC-4.0 Open to feedback on the dataset structure, or ideas on what additional metadata would make this more useful for calibration/hallucination research.