AI Evals at a Glance: Heatmaps for Stakeholders

Google's inspect_viz library now supports heatmap generation for AI evals, enabling stakeholders to quickly compare model performance across skills. The tool, demonstrated with a script that wraps scores_heatmap(), collapses evaluation data into a 2D matrix where color intensity represents accuracy, making trends legible at a glance. The approach addresses the challenge of communicating eval results to non-technical audiences in high-stakes presentations.

Welcome back to our blog series on running, analyzing and visualizing AI evals. Last time https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii we discussed how to design and run evals for analysis and visualization using inspect eval and inspect view . Many of these methods, such as rearranging the dashboard columns and sorting on them to determine model vs model and skill vs no-skill differences in metrics, can give you a broad overview of the field. While this can inspire further and deeper inquiry, it runs into issues with how predictive it is and how to communicate findings to other people. Imagine the following: Imagine a data science lead needing to present a performance breakdown of four LLM candidates across ten internal tools during a high-stakes, time-crunched live meeting. Reordering and filtering the evals across multiple dimensions is hard to read on a presentation screen and requires doing mental math with an audience: riveting stuff that they definitely won’t fall asleep during of course. What if instead you had a tool that allowed you to easily automate rendering a comparison of the different configurations? What if lining them up by model or by skill showed clear and legible patterns? Well, let me introduce you to heatmaps A heatmap is a 2D visual matrix where color intensity represents numerical magnitude—in our case, quadratic curved accuracy scores. By leveraging the human eye's natural ability to distinguish color gradients, heatmaps instantly reveal performance spectrums at a single glance. This makes trends across dependent and independent variables clear, intuitive, and highly legible. While you can simply read along and learn about using inspect viz to generate heatmaps for inspect AI evals, if you intend to follow along and recreate the charts, please start by replicating the steps in the first entry https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii and come back afterward. Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing. For those playing along at home, in order to perform the following, you will need the inspect viz library installed in your environment alongside inspect ai and pandas . From there, take a look at the heatmap script https://github.com/GoogleCloudPlatform/devrel-demos/blob/main/agents/inspect-agent-skills-eval/inspect viz heatmap.py . While there are some intricacies to the script such as establishing its CLI API the script mainly wraps scores heatmap viz data, orientation="horizontal", ... and marshals our data to assemble a heatmap of it. If you run the script python3 inspect viz heatmap.py logs -o heatmap.html you will get a visual that looks like the following. This heatmap has two axes, one is the model vertical which allows you to visually break down your analysis to just skill inclusions for the same model, the other is the skills horizontal and going down the row allows you to demonstrate how changes to the model affect the evaluation. Stakeholders like investors or your boss can now spot skill gains and model ceilings at a glance without reading line-by-line log values. Some quick takeaways: gemini api skill at 3.6-flash trended higher in score on average across the evaluated skill configurations compared to 3.5-flash-lite . Caption : The inspect viz heatmap projects our full Evaluation Matrix down to a 2D plane by collapsing individual task samples and epochs into aggregate accuracy cells along the Model and Skill axes. While matrix scoreboards provide quick visibility into accuracy differences across models, raw heatmaps suffer from key limitations: To bridge CLI logs with interactive, multidimensional and in-depth reporting, we need a pipeline that exports the raw telemetry metrics into spreadsheets and BI dashboards. In Part 3 , we’ll build a python script to convert output data for processing first in Google Sheets and ultimately Data Studio