Welcome back to our blog series on running, analyzing and visualizing AI evals. Last time we discussed how to design and run evals for analysis and visualization using inspect eval
and inspect view
. Many of these methods, such as rearranging the dashboard columns and sorting on them to determine model vs model and skill vs no-skill differences in metrics, can give you a broad overview of the field. While this can inspire further and deeper inquiry, it runs into issues with how predictive it is and how to communicate findings to other people. Imagine the following:
Imagine a data science lead needing to present a performance breakdown of four LLM candidates across ten internal tools during a high-stakes, time-crunched live meeting. Reordering and filtering the evals across multiple dimensions is hard to read on a presentation screen and requires doing mental math with an audience: riveting stuff that they definitely won’t fall asleep during of course.
What if instead you had a tool that allowed you to easily automate rendering a comparison of the different configurations? What if lining them up by model or by skill showed clear and legible patterns?
Well, let me introduce you to heatmaps! A heatmap is a 2D visual matrix where color intensity represents numerical magnitude—in our case, quadratic curved accuracy scores. By leveraging the human eye's natural ability to distinguish color gradients, heatmaps instantly reveal performance spectrums at a single glance. This makes trends across dependent and independent variables clear, intuitive, and highly legible.
While you can simply read along and learn about using inspect viz
to generate heatmaps for inspect AI evals, if you intend to follow along and recreate the charts, please start by replicating the steps in the first entry and come back afterward.
Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing.
For those playing along at home, in order to perform the following, you will need the inspect_viz
library installed in your environment (alongside inspect_ai
and pandas
).
From there, take a look at the heatmap script. While there are some intricacies to the script (such as establishing its CLI API) the script mainly wraps scores_heatmap(viz_data, orientation="horizontal", ...)
and marshals our data to assemble a heatmap of it.
If you run the script
python3 inspect_viz_heatmap.py logs -o heatmap.html
you will get a visual that looks like the following.
This heatmap has two axes, one is the model (vertical) which allows you to visually break down your analysis to just skill inclusions for the same model, the other is the skills (horizontal) and going down the row allows you to demonstrate how changes to the model affect the evaluation.
Stakeholders (like investors or your boss) can now spot skill gains and model ceilings at a glance without reading line-by-line log values. Some quick takeaways:
gemini_api_skill
at 3.6-flash
trended higher in score on average across the evaluated skill configurations compared to 3.5-flash-lite
.
Caption: The inspect viz
heatmap projects our full Evaluation Matrix down to a 2D plane by collapsing individual task samples and epochs into aggregate accuracy cells along the Model and Skill axes.
While matrix scoreboards provide quick visibility into accuracy differences across models, raw heatmaps suffer from key limitations:
To bridge CLI logs with interactive, multidimensional and in-depth reporting, we need a pipeline that exports the raw telemetry metrics into spreadsheets and BI dashboards.
In Part 3, we’ll build a python script to convert output data for processing first in Google Sheets and ultimately Data Studio!