Evaluating AI Agents as Products

Andrew Marble of willows.ai introduced three tasks to evaluate AI coding agents as products, measuring efficiency, collaboration, and taste in interactive sessions, and tested five agents including three smaller local models and two near-frontier models. Marble argues that traditional benchmarks no longer capture real-world utility, as most agents can solve tasks with human-in-the-loop, and the key differentiators are product-like qualities such as user experience and judgment.

Coding benchmarks evaluate AI agents on well specified tasks. Open ended qualitative checks show which can make the nicest demo but leave enough flexibility that they don’t measure the rigidity of real work. Neither capture human-in-the-loop performance nor do they speak to efficiency and productivity. To explore how we can better measure how “useful” AI agents are, I introduce three tasks to generate data labeling interfaces in an ideally short interactive coding session. The specified interface functionality combines mandatory requirements with open ended choices, allowing us to evaluate efficiency, collaboration, and taste, providing a proxy for how well the agents perform as tools in real coding situations. Here I introduce the tasks and evaluate five agents – three smaller local models and two near frontier. Andrew Marble marble.onl andrew@willows.ai Aug 15, 2026 When we started doing ML, benchmarks scores pretty closely captured the performance of a model. If a classifier had a certain precision and recall, as long as the data distribution didn’t change this told you what you needed to know. I think AI’s origins in simpler ML models, and the current narrow gap between academic advances and commercial offerings has kept an outsized emphasis on benchmarks. Benchmarks can be most useful when a we want to compare models and b we are in a period of rapid model advancement. And in the brief history of LLMs, model performance was probably a suitable proxy for product performance. Anything you build on GPT-5.6 is going to be better than if you’d built it in GPT-4. A known issue of course if that benchmarks saturate and all models can basically “ace” old benchmarks making them useless for comparison. Less discussed is that the entire concept of model benchmarking may be saturating in many applications, and the remaining alpha is in more product-like qualities around user experience and the elusive “taste”. The Artificial Intelligence Coding Agent Index1 is a composite index of three public benchmark datasets DeepSWE, Terminal-Bench V2, SWE-Atlas-QnA . The first two of these have agents do software engineering tasks with well defined expected behavior. The last one has the agent answer questions about a codebase. I would contend that such evaluations no longer align with how effective a given tool say Cursor or Codex or Claude Code is at improving developer outcomes saving time, shipping more, etc . Well specified tasks test for a different set of behaviors than more open-ended real-world problems where taste and judgment are necessary2. Furthermore, we’re at a point where most agents can basically solve any task when a human is in the loop. Finding tasks they can’t solve becomes increasingly about tricking them or identifying edge cases instead of testing which is more useful in the real world. The relevant question is no longer “can it do this” like it was in the early days, but how efficiently does it do it with a human in the loop , does it exercise good judgement that leads to a quality product, does it provide helpful input as a co-worker, etc. On the other hand, there is a danger in open-ended testing without clear goals. Recently Andrej Karpathy suggested prompting for video generation3, and there are analogous coding focused tasks like one-shotting video games. These make cool demos but are too open ended to realistically test utility. AI’s perennial problem is that it’s great for demos but real world considerations take the wind out of its sails. A Netlify blog post also compared how well models could single-shot web design tasks4. This has a similar problem, it doesn’t test against any subjective use criteria, and it only measures single-shot performance, not interaction, so doesn’t obviously predict how useful a tool will be when used with a human in the loop. To explore how useful different coding agents consisting of a model and harness are, I wanted a set of tasks that was specific enough to have constraints and necessary success criteria, while also involving sufficient flexibility to let models demonstrate their taste. Crucially I also wanted real useability criteria. I didn’t want to just build something and say it passed the tests and looked well designed, I wanted there to be the equivalent of a user that could decide if the end result met their needs. The goal is not to fully simulate real use, but to find proxy tasks that give a sense of how efficient and effective the tools are in real situations by simulating meaningful interaction and success criteria. Based on these criteria I experimented with tasks requiring agents to build software in support of dataset labeling or annotation. This is a task I use AI for frequently in real life. It involves clear functional criteria – a user is presented with a set of data points and needs to make some determination s about each add a label , so the software must support that. It involves taste and judgement in terms of how the material is presented, the labeling flow, how progress is saved, etc. And its ultimate success rests in how effectively the user is able to use the software to perform the labeling. Tasks General criteria: some simple tasks requiring human judgement that could be completed rapidly. Sufficient volume to get a feel for how efficiently a user can run through many samples. Meaningful upside in how the information could be laid out to support the user in their task. Class labeling: I extracted 75 sentences each from Wikipedia articles about a the Dot-Com bubble and b the 2008 financial crisis. The goal is to build an interface to help the user label which article the sentence came from assuming they can tell by reading it . Starting prompt: In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates Ranking: For 50 unicode characters ♠, ♥, ☀, ☃, etc I asked an LLM Qwen 3.6 35B A3B to generate five attempts at a .svg line drawing. The goal is to build an interface that lets the user rank the attempts for each character. Starting prompt: in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. Code editing and annotation: I generated 100 python code snippets consisting of functions with various syntax errors introduced. The goal is to build an interface that lets the user correct the code, and annotate / validate the location of the error, given the error message from the python interpreter. Starting prompt: dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. Think of this as a test for the human to use python’s error message to confirm what is wrong and make it right. Test Setup: The models tested so far are Z.ai GLM 5.2, Qwen 3.6 35B A3B 8-bit quant , Laguna Poolside 2.1 4-bit quant , Meta Muse Glimmer 8-bit quant , and Claude Opus 5. Claude was used with Claude Code and the rest were used with stock Pi another coding harness . I used whatever the default thinking was for each and this isn’t meant to be a detailed comparison between thinking modes. The models were selected based on ones I wanted to use for one reason or another. Claude because it’s generally good and I’ve used it for a long time, GLM as a modern open weights model, and the three smaller ones as examples I can run locally with different tradeoffs between active and total parameters. Test Procedure: For each of the three tasks I used GLM to build a reference implementation in an interactive session beginning with the prompts mentioned earlier. I used each of these sessions to decide on an appropriate end-state that captured the base functionality I wanted to see in the application. I approached the task as I would in a real development setting, iteratively running the application, identifying issues or refinements, and them prompting to address them. For the subsequent agents I replayed, to the extent possible, the sessions, beginning with the same starting prompt, and using the same follow-ups when applicable, but adjusting as necessary for different issues that arose. Evaluation: We can look at the number of tokens used and the number of user turns required as measures of efficiency and native taste. Fewer turns indicates the model arrives at what I consider a good result faster, despite the underspecification of the initial prompt. The more interesting part of the evaluation comes from actually experiencing the interaction and the output and is mostly manual and qualitative. This may be unappealing in comparison with automated benchmarks, or feel like a cop-out. The goal is to replicate “product testing” however, which in other domains, say running shoes or cars, ultimately boils down to using something and seeing how well it works for you. One quantitative measure that this method supports but was not attempted is actually timing how efficiently and accurately the downstream data labeling task can be completed using the different tools. This, combined with the development time and manual steering required, could give a firmer quantitative picture of how effective the different agents are. In the qualitative evaluation we look for what stylistic choices were made to support the tasks, how much intuition or common sense the agent had about the task, and the apparent ease of use of the developed solution. We also consider the overall experience and impression using the agent. In running the evals, they felt very representative of the kind of interactions I have with coding agents, including the debugging side quests and need for steering. Quantitatively, we can also measure how many turns were taken and how many tokens were used, both proxies for efficiency. Apps Below are screenshots of the three applications generated with each model Z.ai GLM 5.2 / Pi Class Labeling Ranking Editing On the class labeling task, GLM needed to be told to add a sidebar, took too long thinking and reached the default token limit twice, and initially messed up the syntax highlighting, breaking the layout whenever it highlighted a word. It particularly struggled when asked to come up with a way of highlighting words and waffled on trying to make a word list. It also made in my view a poor design choice in picking red and green for the two class colors, given the good/bad connotation of these two colors. For the ranking task, GLM made near black-on-black interface to draw the pictures at first which made it impossible to see. It also displayed a bunch of metadata that had no relevance to the task. On the editing task, GLM had some initial scroll and layout issues, and displayed extraneous information – the instructions specify that this is a test using just the python error message but it displayed other info anyway. Overall GLM generated easy to use applications and made good stylistic choices. For example, it up front made the sentence large and easy to read for the class labeling task. For the ranking task, there was a but if busyness in the interface and the dark mode made it hard to read, though I retrospectively noticed it added a light / dark toggle. The editing interface requires scrolling to see the whole main panel which makes it slower to use. Qwen 3.6 35B A3B / Pi Class Labeling Ranking Editing Qwen didn’t make a great use of space in the class labeling app, the sentence to be reviewed was smaller than it needed to be. Similar to GLM, Qwen didn’t initially add a scroll bar, and messed it up on its first pass. It also had trouble with the layout when applying syntax highlighting. On the ranking task, Qwen didn’t fully finish according to what I wanted but completed something that worked. It ran into errors that needed multiple human-in-the-loop debugging rounds, first with loading the pictures and then dragging and dropping. It also initially displayed the pictures in black-on-black. For the editing task, Qwen initially shared information it was not supposed to with the user, required steering on the error highlighting, had an error in the button states that needed manual correction Laguna Poolside 2.1 / Pi Class Labeling Ranking Editing For the class labeling task, Laguna initially built a terminal application. It did however respond to the prompt to create a web-app but adding a sidebar automatically. It didn’t display the full sentences initially which is poor judgement in an application specifically for reviewing sentences. Laguna also picked red for one of the class colors. For the ranking task, Laguna made only minor judgement errors such as displaying the images in a stack that needed scrolling at first. Laguna experienced a number of problems on the annotation task: scroll and layout issues, timeout due to excessive thinking, errors highlighting error location and with persistence, and an issue with loading an external library pyodide that required some back and forth. Meta Muse Glimmer / Pi Class Labeling Ranking Editing Muse also generated a terminal app to start with for the class labeling task and didn’t initially add a side scroll bar. It initially didn’t show how each sample had been labeled and only that it had been. It performed well on the syntax highlighting in that it pursued a quick, simple solution. GLM, Qwen, and Laguna each took an outsized amount of thought to come up with highlighting schemes. For the ranking task, Muse took additional back and forth to resolve some minor bugs that surfaced about display and dragging. On the annotation task, Muse didn’t initially add a sidebar, some loading issues, left in error information it shouldn’t have, and popped up an alert between evals which slows the user down. Claude Opus 5 / Claude Code Class Labeling Ranking Editing On the class labeling task, Claude also didn’t add a scroll bar at first, otherwise it correctly built the app with minimal steering. Claude mostly single-shotted the ranking task, the only issue was that it focused on speed only clicking the pictures in order to rank but didn’t include the fallback cases of moving them around from the outset. For the annotation task, Claude added extra information that should have been hidden from the user, and lacked judgement about sidebar labels, labelling by error type when they were substantially all the same error type. Efficiency The tables below compare token counts and turns for the different tasks. Note that Claude reports its token use slightly differently and most models don’t differentiate between output and reasoning tokens. Muse consistently uses a smaller number of tokens than the others, while taking about an average number of turns. Claude is always the fewest turns, largely because it includes lots of internal testing and debugging within turns. While it doesn’t appear overly token-heavy, the usage here apparently cost ~$50. Both Qwen and Laguna had instances where the interaction took an outsized number of turns due to some bug that required iteration. Class Labeling Agent Input Output Reasoning Cache read Cache creation Total Turns GLM 5.2 / Pi 89,139 127,572 106,284 1,403,456 — 1,620,167 6 Qwen 3.6 35B A3B / Pi 107,722 33,518 0 2,204,257 — 2,345,497 9 162,784 34,994 0 2,535,413 — 2,733,191 Laguna 2.1 / Pi 90,829 109,040 0 4,253,740 — 4,453,609 5 Muse Glimmer / Pi 16,405 19,201 0 663,177 — 698,783 5 Opus 5.0 / Claude Code 281 155,820 — 11,022,272 165,744 — 4 Ranking Agent Input Output Reasoning Cache read Cache creation Total Turns GLM 5.2 / Pi 116,016 84,130 48,363 2,562,304 — 2,762,450 4 Qwen 3.6 35B A3B / Pi 141,296 75,554 0 5,533,304 — 5,750,154 14 Laguna 2.1 / Pi 70,904 88,254 0 6,716,033 — 6,875,191 5 Muse Glimmer / Pi 25,637 34,620 0 4,809,671 — 4,869,928 10 Opus 5.0 / Claude Code 199 166,573 — 7,528,675 697,690 — 2 Editing Agent Input Output Reasoning Cache read Cache creation Total Turns GLM 5.2 / Pi 97,851 89,789 53,014 4,403,648 — 4,591,288 6 Qwen 3.6 35B A3B / Pi 108,892 38,328 0 4,520,312 — 4,667,532 9 Laguna 2.1 / Pi 130,250 291,206 0 7,919,440 — 8,340,896 18 Muse Glimmer / Pi 34,558 28,456 0 5,200,109 — 5,263,123 8 Opus 5.0 / Claude Code 284 190,226 — 13,309,141 626,046 — 3 Overall Impressions Claude + Claude code is clearly the “smartest” and most capable of the models, which should be no surprise and shows in its ability to nearly single shot many of the tasks, with only minor tweaks necessary, and its generally good judgement. What I like the least about it is what I call “co-working” – it does way too much useless stuff, particularly extensive testing, without asking. This makes any iteration that does occur take way longer, and of course uses more tokens. Some people might appreciate the rigor, I found it overkill for these tasks and that it made it hard to collaborate with. I would have preferred a few extra rounds of back and forth than sitting and waiting for it to run yet another test of something. It’s important to note that Claude Code itself probably plays a big role here, I did not try Claude with Pi and this may significantly reduce the testing it does. Note added in draft: While I didn't use any AI to write this, proofread it, etc., I did get Claude Code to apply some formatting to the HTML version. Ironically it again spent an outsized amount of time running various tests, even when it had clearly read what I wrote above, I felt at some level like it was rubbing it in. GLM , for all intents and purposes is as capable as Claude for the tasks tested here. Claude doesn’t share it’s reasoning tokens so we can’t compare it, but GLM felt very verbose, like it waffled a lot before coming to decisions. In one case it timed out because it just went on thinking instead of ever concluding. My sense is also that it’s judgement might be subtly worse than Claude’s – it needed a little more steering. Overall it was easy to work with in Pi and I appreciated not having to sit through any excessive validation. The end result didn’t seem any worse for it. I will lump Qwen and Laguna together. These are capable small models Laguna is bigger but had to be quantized to fit on my machine , but the gap with the frontier models shows on a task like this. Both took consistently more steering to get the apps right, and lacked some of the judgement the bigger models could exercise. Ultimately the interfaces they generated were simpler looking and had subtly lower production values, however for an application like this it doesn’t really matter. I found the autonomy of the models good – they did their thing but involved me as necessary didn’t go off in any side-quests of their own cough Claude and supported me well in debugging. They both also think too much and get stuck in long or infinite thinking loops sometimes, failing to make a decision. Muse came out as I was running the tests and I decided to include it to see how it differed from the other two small models. It’s particularly unique in how it thinks. It’s extremely terse and to the point, has a completely different style no “Wait, actually…” like all the other ones, and generally is very action oriented. This is a dense model so is much slower running on my machine, but the overall interactions ended up faster than with Qwen or Laguna because of how direct it was. I had expected speed to be the biggest barrier to using it. In terms of performance, it was as good as or better than the other two small models in terms of judgment and the steering it required. A summary of overall impressions is below. It's subjective but based on my experience building the different apps and the observations above. Agent Taste Co-working Intelligence Verbosity GLM 5.2 / Pi ◕ ● ● ◕ Qwen 3.6 35B A3B / Pi ◑ ◕ ◑ ◑ Laguna 2.1 / Pi ◑ ◕ ◑ ◔ Muse Glimmer / Pi ◕ ● ◕ ● Opus 5.0 / Claude Code ● ◑ ● ◕ Conclusions Having run these evaluations, I feel personally like I have a good handle on the model’s relative strengths and weaknesses for interactive coding, particularly how relatively productive one can be using the different tools. Historically people have preferred purely quantitative benchmarks when comparing LLMs, including coding agents. I think this will change as we start to care more about the product vs just raw AI ability. Especially now that all of the mainstream models are above a certain bar and can essentially do any task with the right coaching. We’re going to need new “product review” evals, and rely on trusted reviewers, the same way we would with any other product. The examples I’ve shown here are good proxies for real tasks because they combine mandatory functionality with some taste and intangible criteria, and the expectation of a human in the loop. They can be extended, if we wanted to measure real end-to-end performance, by measuring how well as user can use the AI coded apps labeling speed and accuracy which can be an appealing property for measuring the value of AI coding agents. Appendix Below are the user turns for the different sessions. While I didn’t include the agent turns in the interest of length, the user turns give a sense of the interaction. Class Labeling Z.ai GLM 5.2 / Pi user In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates user Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them? user Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform since we don't have a train test split that would be cheating , let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. Before doing that, please change the coloring away from red/green which to me implies good / bad to something more neutral but still with good contrast user You errored due to max token limit, keep going please user OK there have been two passes where you reached to max token limit overthinking the heuristics to use. I'm just looking for a simple set of words or terms to highlight, it doesn't have to be exhaustive or 100% correct, it should just be a net help to the user. Err on the side of simplicity and come up with some basic highlighting rules without overthinking user The highlighted words are in their own divs or something that don't flow with the rest of the sentence and the text wraps within them, so instead of just appearing as highlighted words they completely change the layout of the text and make it unreadable Qwen 3.6 35B A3B / Pi user In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates user Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them? user It seems like I can't actually navigate with the side bar, when I click on a sentence it doesn't go there, it just advances to the next one as I grade them user Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform since we don't have a train test split that would be cheating , let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. user The highlighted words are in their own divs or something that don't flow with the rest of the sentence and the text wraps within them, so instead of just appearing as highlighted words they completely change the layout of the text and make it unreadable user There's still something wrong. The text wraps separately in different sections. It's like there is only ever one row that gets divided into columns, the highlighted words go in a column, and then the interstitial non-highlighted text goes in adjacent columns and wraps when its too long to fit the column which is always user Great that worked. Now is there a way to export the data as a json once it's labeled? user Can you make it so I can download it at any intermediate stage and not just when done? user It looks like I'm stuck on the done screen now when I load the app and not able to label any Laguna Poolside 2.1 / Pi user In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates user OK that's a good start - I want to be able to move back and forth more easily though and not just step through. Can we make this into a web app to give some more flexibility? user OK looking good. I noticed it doens't display the full sentences on the web app, they get truncated after a certain length. Can you fix that please? user Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform since we don't have a train test split that would be cheating , let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. Before doing that, please change the coloring away from using red which to me implies bad to something more neutral but still with good contrast user Keep going Meta Muse Glimmer / Pi user In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates user OK that's a good start - I want to be able to move back and forth more easily though and not just step through. Can we make this into a web app to give some more flexibility? user Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them? Also I'd like to be able to see which way a sample was classified when I go back to it, right now it looks like they all just get highlighted in yellow to indicate they have been labeled but I don't see the actual label anywhere user Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform since we don't have a train test split that would be cheating , let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. user The operation stopped for some reason, I may have hit escape by accident Claude Opus 5 / Claude Code user In this directory is a file that contains random sentences from wikipedia articles about either the 2008 financial crisis or the dotcom bubble. I want to make an application that displays this data for a person to review and manually label as one or the other. The data is labeled but ignore those labels. Our goal is to allow the user to review and decide as quickly as possible the article from which each sentence originates user Can you add a panel on the left that contains a scrollable list of all the sentences so we can rapidly move between them? user Great. Can you add some heuristic syntax highlighting to make it easier for the user to tell at a glance some of the features that might predict which article the sentence came from? Don't bother doing any testing of how well they perform since we don't have a train test split that would be cheating , let's just experiment with some basic highlighting. For example anything that's a year would be telling, words related to finance or mortgages vs words related to technology would be telling. user Is there a way to export the current progress back to a json? I didn't see anything Ranking Z.ai GLM 5.2 / Pi user in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. user I can barely see the shapes, it looks like they are drawn in a thin black line on a black and dark blue checkered background, can you make them thicker and add contrast please user The metadata about each drawing isn't really important to assessing quality, and right now the UI is a bit clumsy. I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order? user Looks good, one last minor thing, can you make clicking on an image in the top row default to moving it to the next available rank, so if the user wanted to they could just click them all in rank order best to worst and it would complete the task without needing to drag. They should still be able to move them around as before if they want Qwen 3.6 35B A3B / Pi user in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. user It runs but all I ever see is "loading" - is it supposed to take a really long time? user The spinner spins forever, there is an uncaught syntax error, invalid assignment left-hand side one line 591 of rank.html according to the console: card.querySelector '.rank-badge' ?.textContent = rankings symSlug ? 'Ranked' : 'Not ranked'; user It works but there is a SVG load error 404 for every picture, maybe we have the urls wrong? user Still seeing e.g. SVG load error for output/zap-U26A1/attempt 0.svg Error: HTTP 404 loadSVG http://localhost:9876/rank.html:374 user No more 404s but I still don't see the images and am getting this error SVG parse error for output/fleur de lis-U269C/attempt 2.svg TypeError: can't access property "maxWidth", svgEl.style is undefined user I can barely see the shapes, it looks like they are drawn in a thin dark gray line on a black background, can you make them thicker and add contrast please user OK great it works. Can you clean it up by making it so we see only one set of images at a time, and have a navigation panel on the side that lets us move through the different symbols user Right now the UI is a bit clumsy. I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order? user Looks good but the drop part of drag and drop is not working, I can drag the cards over but they don't actually stick on the bottom user It still doesn't work. Could I be doing something wrong? Is there a way to add some debugging so we can see what state it is in an confirm it knows when it's being dragged, when it's over a slot, when it's released? They disappear when plaed but don't actually show up in the bottom squares. And they can't be dragged back to the top. I want it to be possible to move them around or back up after the've been placed, and to see where they are placed Laguna Poolside 2.1 / Pi user in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. user The overall look and feel is good but right now the images are stacked vertically, making it very difficult to sort them as they go off the bottom of the page. Can you make then side-by-side? user I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order? user Looks good, one last minor thing, can you make clicking on an image in the top row default to moving it to the next available rank, so if the user wanted to they could just click them all in rank order best to worst and it would complete the task without needing to drag. They should still be able to move them around as before if they want user One small thing, is there currently a way to export the current state as a json? Can you add one? Meta Muse Glimmer / Pi user in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. user I want to be able to sort them as fast as possible, can we modify it so the initial images are displayed and can be dragged into a row of slots below that correspond to their rank order? user Ok looks good but a bit of cleanup: right now the squares can only be dragged by the bottom border, can we make it so we can drag them by any part? Also, the presentation looks good on the top part, we see the pictures nicely and all five fit across the page. On the bottom row, the sorting slots are bigger and wrap so only three fit on a row, and when the image is dragged into them it becomes smaller and distorted. user Looks like pi froze on the test user There's something wrong with the server, it doesn't respond, I checked it manually as well user Looks good, one last minor thing, can you make clicking on an image in the top row default to moving it to the next available rank, so if the user wanted to they could just click them all in rank order best to worst and it would complete the task without needing to drag. They should still be able to move them around as before if they want user That worked. Now can you make it so it saves the state automatically without needing to press Save Ranking ? Right now if I sort them and click next it saves, but if I move manually to another on the nav bar it does not user Great. Now I noticed a few of the svgs don't draw and an error message is shown in the pane. For example crescent moon attempt 5 and umbrella attempt 1. The message says This XML file does not appear to have any style information associated with it . I have rendered these elsewhere and they worked fine user Now it just shows the name, e.g. Attempt 5 in the square, it doesn't render the image user OK that's fixed. Now can you make it so the user can bring the images back to the starting row once they've all been sorted. If I start sorting, I can still bring some back to the original row, but once they've all been moved to the ranking row, the original row disappears and I can only drag them within the ranked row. I'd prefer if the layout stays consistent even when they've all been brought to the bottom row Claude Opus 5 / Claude Code user in ./output are 50 folders each corresponding to a symbol and containing five attempts by an LLM at drawing that symbol as a svg. Can you make me a ranking interface that lets me view the five for each symbol and rank them from best to worst. Don't overfit anything to the data, it should work for other shapes in this format as well. user OK looks mostly good. I'd like to be able to drag e.g. one to fifth place immediately, or more generally move them between any two spots, either from the unranked to ranked row or within either row, just to make it easier to play with the rankings. I like the click approach, that would normally be the way to speedrun it but it should also be flexible. Editing Z.ai GLM 5.2 / Pi user dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right. user There are some scroll issues, the main panel appears blank at the top, and when I scroll down I see "loading" and then scrolling much further down I see the actual annotation interface. We should just see the annotation interface in the main panel without having to scroll, once it has loaded user OK, that fixed it. However, the sidebar can only scroll when the whole page scrolls. Please make it scroll separately so the annotation workflow remains on screen in the main panel but the user can scroll the sidebar. I think this was an underlying reason for why previous problem. user OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code user OK, two things I see: What is the user supposed to do with "annotation status: pending?" can't they just modify and save, right now it could be modified but then they need to separately change the status, that feels confuising. Second, what purpose does "run code" serve? It can only run the original code, for which we already have the error message. This could check the message if it was in quesiton, but the messages are not in question, we assume they are accurate user That's fixed. What is the meaning of the tags in the top pane? They seem only to say "IndentantionError:syntax error" or "SyntaxError:syntax error" which isn't super helpful. I guess it's just listing the python error class, but since they are basically all SyntaxError except IndentationError which python treats separately for some reason they are not at all informative Qwen 3.6 35B A3B / Pi user dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right. user I need port 8765 for something else, can you run it on 8764 please? user Can you add a sidebar for navigation that lets the user scroll through the samples? user OK great. Now let's hide the error category from the user. The user should only base their annotation off of the python error message and the code user Looks good. Can you make it automatically save any code changes when edits are made? Right now if I fix the code then click to another page without confirming, the changes are lost. user One last thing, can you make it possible to highlight a set of columns within a row where the error occurs? Right we can only pick the row for the annotation user Can you make it so highlighting automaticall adds the annotation, like if I highlight cols 5-7 it will populate the fields with those values? user That works. Now can you add a way to verify that there are no longer syntax errors in the corrected code? user Small issue, the button starts yellow and says "check syntax" but once clicked it doesn't go back to this state, for example it says "syntax error" and stays that way even if I go to another sample Laguna Poolside 2.1 / Pi user dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right. user Can you please change it so the dataset is preloaded for now user OK - seems to load now. Can you add a navigation pane on the left to scroll through and move to any of the items? user There is an issue with the alignment, the sidebar is on top of the main panel. The side navigation bar should scroll separately and the main annotation panel should stay in a fixed spot occupying most of the screen user Looks like you ran out of space before deciding on any action. Please concisely take some action to fix the alignment issue user There seems to be a bug where the main panel doesn't scroll now, so I can't see all the fields. The rest looks good. user A weird error now, the main panel scrolls independently but it's all in a thin horizontal area towards the top of the screen so only about two lines of text are visible at a time and most of the screen is blank user Now it seems like neither the sidebar nor the main pane scrolls? The content and size look right user You went in a seemingly infinite thinking loop and failed to reach a decision. Please complete the request with minimal thinking user OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code user Better take them out of the sidebar too please. Can you just keep the numbers and add the name of the function as an identifier user Did you restart the server, I still see the old interface even when I refresh user Works, thanks. One last thing, when I click verify fix I get Failed to load Pyodide: V.default.parse is not a function user This is on a mac, what's the right way to refresh? cmd-shift-r still results in the same error user OK now it gets past that error, says it's loading pyodide, then errors with Verification failed: can't access property "startsWith", result is undefined user Works . Now I notice I can only select the line of the error. Is it possible to let the user highlight the columns as well as the row where the error occurs? user That worked. But edits and annotations don't persist when you go to a different item, either by clicking next or by clicking in the nav panel. If I make and edit and then click to another item, then click back, it's gone, same for when I change the annotation. user The fix works for the code changes but not the annotation Meta Muse Glimmer / Pi user dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right. user Great. Can you add a side navigation bar where the user can scroll through the samples easily? user It seems like it's not loading the samples, I see the layout including a sidebar but I don't see any of the content and the interface doesn't work. Also, when I kill the server with ctrl-c it doesn't free up the port which is annoying user OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code user OK - a bit of cleanup. When I press tab in the corrected code block, it takes be to the next field the save review button . I'd expect it to add a tab in the editor window user Great. Now with respect to the annotation. Can you make it so the user can click and highlight the location? user OK that works. can you also update the titles shown in the side navigation bar so it shows the function name along with the number as a way to make it easier to navigate user Can you take off the alert than pops up when saving and just automatically save any updates. This will make is faster for a person to work through them Claude Opus 5 / Claude Code user dataset.json contains a list of 100 python code snippets functions that have errors introduced. Considering only the code and the "error display" that shows the error python would generate, write an app that a human reviewer could use to confirm the error by adding an annotation of the location and updating the error message if necessary, and edit the code to make it correct. This of this as a test for the human to use pythons error message to confirm what is wrong and make it right. user OK great. Now let's hide the error category, as well as the error description at the top from the user. The user should only base their annotation off of the python error message and the code user I think the verdict is OK for now. Can you remove the SyntaxError/TabError/IndentationError tags as well. I understand the intent but they are basically all syntax errors, which is the point of the exercise, python just has more resolution re errors related to whitespace for some reason, but it's not super helpful to have about 90/100 just labeled syntax error. I think we can just keep the numbering, maybe add the function name as an identifier to keep them more memorable