Member-only story
Yesterday everyone cheered Gemini 3.6 Flash for topping the computer-use leaderboards. Then Jerry Liu, the CEO of LlamaIndex, ran it through a document-understanding benchmark and found the opposite story hiding underneath: on charts, the shiny new Flash model scores 14 points worse than the model it replaced — 45.0 down to 31.0. If you use Gemini Flash to read documents, the “upgrade” is a downgrade.
This is the part of a model launch nobody puts in the blog post. Gemini 3.6 Flash is genuinely better at agentic coding and driving a desktop. But Liu’s ParseBench run shows that the same post-training that bought those agentic gains quietly cost the model its eyesight for documents. Charts collapsed, tables slipped, and the overall document score fell from 69.9 to 66.8. If your pipeline OCRs invoices, parses tables, or reads charts, bumping the version number is exactly the wrong move — and here’s the data, plus how to check it against your own documents before you ship the regression to production.
The benchmark that flips the launch narrative #
ParseBench scores models on real document-parsing tasks — extracting tables, reading charts, preserving layout, staying faithful to the text — rather than…