Step up to the Sheets: AI Eval Export and Illustrating Data

A developer from Google AI detailed a pandas-based pipeline that converts Inspect AI evaluation logs into a CSV optimized for Google Sheets, enabling non-technical stakeholders to create boardroom-ready visualizations. The script pre-computes boolean columns for filtering and exports data while preserving all 32 model-skill-sample-epoch combinations.

Welcome to part three of our series about designing, analyzing and visualizing AI Evals As with the previous entries you are more than welcome to simply read over my exploration or for those of you looking for extra credit, feel free to play along at home. Note: This blog series contains AI-generated diagrams alongside actual screenshots and hand-drawn edits of both. AI also assisted in minor copy editing. If you intend to re implement the walkthrough, all you need to do right now is complete Part 1 Designing AI Evals: Clarity Now and Visualization Next https://dev.to/googleai/designing-ai-evals-clarity-now-and-visualization-next-4eii and then install pandas ; Part 2 AI Evals at a Glance: Heatmaps for Stakeholders https://dev.to/googleai/ai-evals-at-a-glance-heatmaps-for-stakeholders-2mki is illustrative and will give you a deeper understanding going forward both of AI evals and why the following may be useful but it’s more elective than anything else. While engineering teams might evaluate agents using test automations, non-technical stakeholders product managers, business leads, or executive sponsors often use spreadsheets as their lingua franca and as such it would make sense to speak to them in a way they are almost certain to understand and which enables you all to collaboratively make visualizations to further get out the word. To make this a reality, today we’ll use a pandas pipeline that transforms raw Inspect AI evaluation logs into a rich CSV optimized for Google Sheets, allowing you or anyone on your team with a little spreadsheet knowledge to make boardroom-ready visuals. Drawing again from previous blog entries, the evals we are working on have three dimensions: model x skill x sample and epoch that cross to form the following matrix: In Part 2 https://dev.to/googleai/ai-evals-at-a-glance-heatmaps-for-stakeholders-2mki we created a simplified heatmap reflecting a planar slice of this space as reduced over the sample dimension. While we will ultimately do something similar in a future blog post using Data Studio, for the time being we actually want to keep all 32 combinations separate and pull more of their underlying stats from the inspect eval files and derive further information using pandas . This information will be exported as a flat csv file we can then import and view in Google Sheets. tocsv.py This script is used to parse the eval logs using Inspect AI ’s python pandas interface. The script is relatively self documenting, and you can read up more on pandas https://pandas.pydata.org/docs/user guide/10min.html and pandas’ csv IO https://pandas.pydata.org/docs/user guide/io.html if you are unfamiliar. While most of the script deals with mapping scalar metrics the like of which we’ve already discussed, there are two that are purely logical/boolean. They represent specific statistical cases that we will allow end users to partition the data set over for analysis later. These boolean conditions are derived and since we want to pare away excess information and therefore not supply the full eval files to Google Sheets, tocsv.py does the heavy lifting upstream during log ingestion. It pre-computes two key boolean columns to simplify downstream dashboard filtering: completed is baseline or has activated skill TRUE and TRUE when the skill tool was actually “activated” which in inspect means uptaken from FALSE , where skills were available but never activated .In Google Sheets or Data Studio non-technical stakeholders can simply filter/toggle these pre-computed boolean keys to isolate clean, active performance or otherwise instantly. In order to execute the pipeline I called: python3 tocsv.py The output file data mega export.csv is a standard comma-separated values file where each cell is surrounded by commas. You can preview the raw and unformatted contents in a standard text editor. While most of this is normal spreadsheet data such as strings and integers, you will also see fairly complicated Google Sheets formulas resulting from interpolating eval metrics into formula templates. The above formula for instance will be transformed into a bar graph normalized against the range of values in the same column as its source data and tiered into colors by tertile. How do you take this text and get it to show its stuff? By opening sheets.new https://sheets.new of course, and importing data mega export.csv . When presented with the import dialog check Convert text to numbers, dates, and formulas . On clicking Import data Google Sheets will instantly display the standard cells and render active, color-coded visual sparklines directly inside data cells What conclusions can you draw from what you see? One obvious one in this example is that the correlation between score and latency how long an agent takes to answer a question may not be very strict. But what if you wanted to get something deeper? Well, since this is a spreadsheet, obviously you can make pivot tables, and custom functions or sort on multiple keys… you know, the whole accountant’s playbook. Let’s say though that you want custom views into the data rather than to permanently mutate it. How would you go about doing that? Google Sheets, unlike inspect view and inspect viz heatmaps, allows you to dynamically sort AND filter on more than one key. Not only that but you can sort on ANY key, so I elected to do so with the previously unavailable score and latency fields. As shown in the image above, use Data Create filter view which presents you with a new set of controls at the top of the main columns. Note: Filter views do not mutate or change the underlying contents of a sheet; Instead they are optionally named and shareable configurations for restricting what rows are displayed and in what order. If, as in the picture, you click the cost column Filter by condition , select the Less than option and fill the comparison with 0.01, you can narrow the displayed rows to only those with costs less than a cent. This can be used for instance to only look over the samples that were “less costly” to try to analyze why. Filter views despite their name also allow you to sort the display of the data. Because the sort operations are cumulative, the “primary” filter needs to be applied last. Thus, after sorting the filter view first by latency and then by score both ‘Z to A’ aka decreasing gives us the following: Optionally you can name and save this filter view to toggle it on and off as well as share it with other collaborators. To learn more about Filter views, you can read Sort & filter your data https://support.google.com/docs/answer/3540681?hl=en&co=GENIE.Platform%3DDesktop . This ability to sort and filter the display live without changing the underlying information, makes presenting the data and analyzing it during discussions with teammates significantly easier and faster. Now, in spite of this tremendous flexibility, sheets have some drawbacks as visualization tools: So, how can we take this to the next level and circumvent those issues? What about an environment, which while it still offers you the ability to custom script anything you need, comes with a lot of WYSIWYG controllable visualizations and graphs, enabling you, without code, to bring data to an even more vibrant life? Better yet, what if that environment comes with built in visualizations for 3 dimensions or more and has broad compatibility with multiple data source types? In Part 4 , we’ll connect this Google Sheet directly to Data Studio to build a codeless executive summary dashboard and be able to quickly and intuitively display our findings