{"slug": "evaluating-ai-agents-as-products", "title": "Evaluating AI Agents as Products", "summary": "Andrew Marble of willows.ai introduced three tasks to evaluate AI coding agents as products, measuring efficiency, collaboration, and taste in interactive sessions, and tested five agents including three smaller local models and two near-frontier models. Marble argues that traditional benchmarks no longer capture real-world utility, as most agents can solve tasks with human-in-the-loop, and the key differentiators are product-like qualities such as user experience and judgment.", "body_md": "Coding benchmarks evaluate AI agents on well specified tasks.\nOpen ended qualitative checks show which can make the nicest demo but\nleave enough flexibility that they don’t measure the rigidity of real\nwork. Neither capture human-in-the-loop performance nor do they speak to\nefficiency and productivity. To explore how we can better measure how\n“useful” AI agents are, I introduce three tasks to generate data\nlabeling interfaces in an (ideally) short interactive coding session.\nThe specified interface functionality combines mandatory requirements\nwith open ended choices, allowing us to evaluate efficiency,\ncollaboration, and taste, providing a proxy for how well the agents\nperform as tools in real coding situations. Here I introduce the tasks\nand evaluate five agents – three smaller local models and two near\nfrontier.\n\nAndrew Marble marble.onl andrew@willows.ai Aug 15, 2026\n\nWhen we started doing ML, benchmarks scores pretty closely captured\nthe performance of a model. If a classifier had a certain precision and\nrecall, as long as the data distribution didn’t change this told you\nwhat you needed to know. I think AI’s origins in simpler ML models, and\nthe current narrow gap between academic advances and commercial\nofferings has kept an outsized emphasis on benchmarks.\n\nBenchmarks can be most useful when (a) we want to compare\nmodels and (b) we are in a period of rapid model advancement.\nAnd in the brief history of LLMs, model performance was probably a\nsuitable proxy for product performance. Anything you build on GPT-5.6 is\ngoing to be better than if you’d built it in GPT-4. A known issue of\ncourse if that benchmarks saturate and all models can basically “ace”\nold benchmarks making them useless for comparison. Less discussed is\nthat the entire concept of model benchmarking may be saturating in many\napplications, and the remaining alpha is in more product-like qualities\naround user experience and the elusive “taste”.\n\nThe Artificial Intelligence Coding Agent Index1 is\na composite index of three public benchmark datasets (DeepSWE,\nTerminal-Bench V2, SWE-Atlas-QnA). The first two of these have agents do\nsoftware engineering tasks with well defined expected behavior. The last\none has the agent answer questions about a codebase.\n\nI would contend that such evaluations no longer align with how\neffective a given tool (say Cursor or Codex or Claude Code) is at\nimproving developer outcomes (saving time, shipping more, etc). Well\nspecified tasks test for a different set of behaviors than more\nopen-ended real-world problems where taste and judgment are necessary2. Furthermore, we’re at a point where\nmost agents can basically solve any task when a human is in the loop.\nFinding tasks they can’t solve becomes increasingly about tricking them\nor identifying edge cases instead of testing which is more useful in the\nreal world. The relevant question is no longer “can it do this” like it\nwas in the early days, but how efficiently does it do it (with a human\nin the loop), does it exercise good judgement that leads to a quality\nproduct, does it provide helpful input as a co-worker, etc.\n\nOn the other hand, there is a danger in open-ended testing without\nclear goals. Recently Andrej Karpathy suggested prompting for video\ngeneration3, and there are analogous coding\nfocused tasks like one-shotting video games. These make cool demos but\nare too open ended to realistically test utility. AI’s perennial problem\nis that it’s great for demos but real world considerations take the wind\nout of its sails. A Netlify blog post also compared how well models\ncould single-shot web design tasks4. This has a similar\nproblem, it doesn’t test against any subjective use criteria, and it\nonly measures single-shot performance, not interaction, so doesn’t\nobviously predict how useful a tool will be when used with a human in\nthe loop.\n\nTo explore how useful different coding agents (consisting of a model\nand harness) are, I wanted a set of tasks that was specific enough to\nhave constraints and necessary success criteria, while also involving\nsufficient flexibility to let models demonstrate their taste. Crucially\nI also wanted real useability criteria. I didn’t want to just build\nsomething and say it passed the tests and looked well designed, I wanted\nthere to be the equivalent of a user that could decide if the end result\nmet their needs. The goal is not to fully simulate real use, but to find\nproxy tasks that give a sense of how efficient and effective the tools\nare in real situations by simulating meaningful interaction and success\ncriteria.\n\nBased on these criteria I experimented with tasks requiring agents to\nbuild software in support of dataset labeling or annotation. This is a\ntask I use AI for frequently in real life. It involves clear functional\ncriteria – a user is presented with a set of data points and needs to\nmake some determination(s) about each (add a label), so the software\nmust support that. It involves taste and judgement in terms of how the\nmaterial is presented, the labeling flow, how progress is saved, etc.\nAnd its ultimate success rests in how effectively the user is able to\nuse the software to perform the labeling.\n\nTasks\n\nGeneral criteria: some simple tasks requiring human judgement that\ncould be completed rapidly. Sufficient volume to get a feel for how\nefficiently a user can run through many samples. Meaningful upside in\nhow the information could be laid out to support the user in their\ntask.\n\nClass labeling: I extracted 75 sentences each from Wikipedia\narticles about (a) the Dot-Com bubble and (b) the 2008 financial crisis.\nThe goal is to build an interface to help the user label which article\nthe sentence came from (assuming they can tell by reading it).\n\nStarting prompt:\n\nIn this directory is a file that contains random\nsentences from wikipedia articles about either the 2008 financial crisis\nor the dotcom bubble. I want to make an application that displays this\ndata for a person to review and manually label as one or the other. The\ndata is labeled but ignore those labels. Our goal is to allow the user\nto review and decide as quickly as possible the article from which each\nsentence originates\n\nRanking: For 50 unicode characters (♠, ♥, ☀, ☃, etc) I asked an\nLLM (Qwen 3.6 35B A3B) to generate five attempts at a .svg line drawing.\nThe goal is to build an interface that lets the user rank the attempts\nfor each character.\n\nStarting prompt:\n\nin ./output are 50 folders each corresponding to a\nsymbol and containing five attempts by an LLM at drawing that symbol as\na svg. Can you make me a ranking interface that lets me view the five\nfor each symbol and rank them from best to worst. Don't overfit anything\nto the data, it should work for other shapes in this format as well.\n\nCode editing and annotation: I generated 100 python code snippets\nconsisting of functions with various syntax errors introduced. The goal\nis to build an interface that lets the user correct the code, and\nannotate / validate the location of the error, given the error message\nfrom the python interpreter.\n\nStarting prompt:\n\ndataset.json contains a list of 100 python code\nsnippets (functions) that have errors introduced. Considering only the\ncode and the \"error display\" that shows the error python would generate,\nwrite an app that a human reviewer could use to confirm the error by\nadding an annotation of the location and updating the error message if\nnecessary, and edit the code to make it correct. Think of this as a test\nfor the human to use python’s error message to confirm what is wrong and\nmake it right.\n\nTest Setup: The models tested so far are Z.ai GLM 5.2, Qwen 3.6 35B\nA3B (8-bit quant), Laguna Poolside 2.1 (4-bit quant), Meta Muse Glimmer\n(8-bit quant), and Claude Opus 5. Claude was used with Claude Code and\nthe rest were used with stock Pi (another coding harness). I used\nwhatever the default thinking was for each and this isn’t meant to be a\ndetailed comparison between thinking modes. The models were selected\nbased on ones I wanted to use for one reason or another. Claude because\nit’s generally good and I’ve used it for a long time, GLM as a modern\nopen weights model, and the three smaller ones as examples I can run\nlocally with different tradeoffs between active and total\nparameters.\n\nTest Procedure: For each of the three tasks I used GLM to build a\nreference implementation in an interactive session beginning with the\nprompts mentioned earlier. I used each of these sessions to decide on an\nappropriate end-state that captured the base functionality I wanted to\nsee in the application. I approached the task as I would in a real\ndevelopment setting, iteratively running the application, identifying\nissues or refinements, and them prompting to address them.\n\nFor the subsequent agents I replayed, to the extent possible, the\nsessions, beginning with the same starting prompt, and using the same\nfollow-ups when applicable, but adjusting as necessary for different\nissues that arose.\n\nEvaluation: We can look at the number of tokens used and the number\nof user turns required as measures of efficiency and native taste. Fewer\nturns indicates the model arrives at what I consider a good result\nfaster, despite the underspecification of the initial prompt. The more\ninteresting part of the evaluation comes from actually experiencing the\ninteraction and the output and is mostly manual and qualitative. This\nmay be unappealing in comparison with automated benchmarks, or feel like\na cop-out. The goal is to replicate “product testing” however, which in\nother domains, say running shoes or cars, ultimately boils down to using\nsomething and seeing how well it works for you. One quantitative measure\nthat this method supports but was not attempted is actually timing how\nefficiently and accurately the downstream data labeling task can be\ncompleted using the different tools. This, combined with the development\ntime and manual steering required, could give a firmer quantitative\npicture of how effective the different agents are.\n\nIn the qualitative evaluation we look for what stylistic choices were\nmade to support the tasks, how much intuition or common sense the agent\nhad about the task, and the apparent ease of use of the developed\nsolution. We also consider the overall experience and impression using\nthe agent. In running the evals, they felt very representative of the\nkind of interactions I have with coding agents, including the debugging\nside quests and need for steering.\n\nQuantitatively, we can also measure how many turns were taken and how\nmany tokens were used, both proxies for efficiency.\n\nApps\n\nBelow are screenshots of the three applications generated with each\nmodel\n\nZ.ai GLM 5.2 / Pi\n\nClass Labeling\n\nRanking\n\nEditing\n\nOn the class labeling task, GLM needed to be told to add a sidebar,\ntook too long thinking and reached the default token limit twice, and\ninitially messed up the syntax highlighting, breaking the layout\nwhenever it highlighted a word. It particularly struggled when asked to\ncome up with a way of highlighting words and waffled on trying to make a\nword list. It also made (in my view) a poor design choice in picking red\nand green for the two class colors, given the good/bad connotation of\nthese two colors.\n\nFor the ranking task, GLM made near black-on-black interface to draw\nthe pictures at first which made it impossible to see. It also displayed\na bunch of metadata that had no relevance to the task.\n\nOn the editing task, GLM had some initial scroll and layout issues,\nand displayed extraneous information – the instructions specify that\nthis is a test using just the python error message but it displayed\nother info anyway.\n\nOverall GLM generated easy to use applications and made good\nstylistic choices. For example, it up front made the sentence large and\neasy to read for the class labeling task. For the ranking task, there\nwas a but if busyness in the interface and the dark mode made it hard to\nread, though I retrospectively noticed it added a light / dark toggle.\nThe editing interface requires scrolling to see the whole main panel\nwhich makes it slower to use.\n\nQwen 3.6 35B A3B / Pi\n\nClass Labeling\n\nRanking\n\nEditing\n\nQwen didn’t make a great use of space in the class labeling app, the\nsentence to be reviewed was smaller than it needed to be. Similar to\nGLM, Qwen didn’t initially add a scroll bar, and messed it up on its\nfirst pass. It also had trouble with the layout when applying syntax\nhighlighting.\n\nOn the ranking task, Qwen didn’t fully finish according to what I\nwanted but completed something that worked. It ran into errors that\nneeded multiple human-in-the-loop debugging rounds, first with loading\nthe pictures and then dragging and dropping. It also initially displayed\nthe pictures in black-on-black.\n\nFor the editing task, Qwen initially shared information it was not\nsupposed to with the user, required steering on the error highlighting,\nhad an error in the button states that needed manual correction\n\nLaguna Poolside 2.1 / Pi\n\nClass Labeling\n\nRanking\n\nEditing\n\nFor the class labeling task, Laguna initially built a terminal\napplication. It did however respond to the prompt to create a web-app\nbut adding a sidebar automatically. It didn’t display the full sentences\ninitially which is poor judgement in an application specifically for\nreviewing sentences. Laguna also picked red for one of the class\ncolors.\n\nFor the ranking task, Laguna made only minor judgement errors such as\ndisplaying the images in a stack that needed scrolling at first.\n\nLaguna experienced a number of problems on the annotation task:\nscroll and layout issues, timeout due to excessive thinking, errors\nhighlighting error location and with persistence, and an issue with\nloading an external library (pyodide) that required some back and\nforth.\n\nMeta Muse Glimmer / Pi\n\nClass Labeling\n\nRanking\n\nEditing\n\nMuse also generated a terminal app to start with for the class\nlabeling task and didn’t initially add a side scroll bar. It initially\ndidn’t show how each sample had been labeled and only that it had been.\nIt performed well on the syntax highlighting in that it pursued a quick,\nsimple solution. GLM, Qwen, and Laguna each took an outsized amount of\nthought to come up with highlighting schemes.\n\nFor the ranking task, Muse took additional back and forth to resolve\nsome minor bugs that surfaced about display and dragging.\n\nOn the annotation task, Muse didn’t initially add a sidebar, some\nloading issues, left in error information it shouldn’t have, and popped\nup an alert between evals which slows the user down.\n\nClaude Opus 5 / Claude Code\n\nClass Labeling\n\nRanking\n\nEditing\n\nOn the class labeling task, Claude also didn’t add a scroll bar at\nfirst, otherwise it correctly built the app with minimal steering.\n\nClaude mostly single-shotted the ranking task, the only issue was\nthat it focused on speed only (clicking the pictures in order to rank)\nbut didn’t include the fallback cases of moving them around from the\noutset.\n\nFor the annotation task, Claude added extra information that should\nhave been hidden from the user, and lacked judgement about sidebar\nlabels, labelling by error type when they were substantially all the\nsame error type.\n\nEfficiency\n\nThe tables below compare token counts and turns for the different\ntasks. Note that Claude reports its token use slightly differently and\nmost models don’t differentiate between output and reasoning tokens.\nMuse consistently uses a smaller number of tokens than the others, while\ntaking about an average number of turns. Claude is always the fewest\nturns, largely because it includes lots of internal testing and\ndebugging within turns. While it doesn’t appear overly token-heavy, the\nusage here apparently cost ~$50. Both Qwen and Laguna had instances\nwhere the interaction took an outsized number of turns due to some bug\nthat required iteration.\n\nClass Labeling\n\nAgent\n\nInput\n\nOutput\n\nReasoning\n\nCache read\n\nCache creation\n\nTotal\n\nTurns\n\nGLM 5.2 / Pi\n\n89,139\n\n127,572\n\n106,284\n\n1,403,456\n\n—\n\n1,620,167\n\n6\n\nQwen 3.6 35B A3B / Pi\n\n107,722\n\n33,518\n\n0\n\n2,204,257\n\n—\n\n2,345,497\n\n9\n\n162,784\n\n34,994\n\n0\n\n2,535,413\n\n—\n\n2,733,191\n\nLaguna 2.1 / Pi\n\n90,829\n\n109,040\n\n0\n\n4,253,740\n\n—\n\n4,453,609\n\n5\n\nMuse Glimmer / Pi\n\n16,405\n\n19,201\n\n0\n\n663,177\n\n—\n\n698,783\n\n5\n\nOpus 5.0 / Claude Code\n\n281\n\n155,820\n\n—\n\n11,022,272\n\n165,744\n\n—\n\n4\n\nRanking\n\nAgent\n\nInput\n\nOutput\n\nReasoning\n\nCache read\n\nCache creation\n\nTotal\n\nTurns\n\nGLM 5.2 / Pi\n\n116,016\n\n84,130\n\n48,363\n\n2,562,304\n\n—\n\n2,762,450\n\n4\n\nQwen 3.6 35B A3B / Pi\n\n141,296\n\n75,554\n\n0\n\n5,533,304\n\n—\n\n5,750,154\n\n14\n\nLaguna 2.1 / Pi\n\n70,904\n\n88,254\n\n0\n\n6,716,033\n\n—\n\n6,875,191\n\n5\n\nMuse Glimmer / Pi\n\n25,637\n\n34,620\n\n0\n\n4,809,671\n\n—\n\n4,869,928\n\n10\n\nOpus 5.0 / Claude Code\n\n199\n\n166,573\n\n—\n\n7,528,675\n\n697,690\n\n—\n\n2\n\nEditing\n\nAgent\n\nInput\n\nOutput\n\nReasoning\n\nCache read\n\nCache creation\n\nTotal\n\nTurns\n\nGLM 5.2 / Pi\n\n97,851\n\n89,789\n\n53,014\n\n4,403,648\n\n—\n\n4,591,288\n\n6\n\nQwen 3.6 35B A3B / Pi\n\n108,892\n\n38,328\n\n0\n\n4,520,312\n\n—\n\n4,667,532\n\n9\n\nLaguna 2.1 / Pi\n\n130,250\n\n291,206\n\n0\n\n7,919,440\n\n—\n\n8,340,896\n\n18\n\nMuse Glimmer / Pi\n\n34,558\n\n28,456\n\n0\n\n5,200,109\n\n—\n\n5,263,123\n\n8\n\nOpus 5.0 / Claude Code\n\n284\n\n190,226\n\n—\n\n13,309,141\n\n626,046\n\n—\n\n3\n\nOverall Impressions\n\nClaude + Claude code is clearly the “smartest” and most capable of\nthe models, which should be no surprise and shows in its ability to\nnearly single shot many of the tasks, with only minor tweaks necessary,\nand its generally good judgement. What I like the least about it is what\nI call “co-working” – it does way too much useless stuff, particularly\nextensive testing, without asking. This makes any iteration that does\noccur take way longer, and of course uses more tokens. Some people might\nappreciate the rigor, I found it overkill for these tasks and that it\nmade it hard to collaborate with. I would have preferred a few extra\nrounds of back and forth than sitting and waiting for it to run yet\nanother test of something. It’s important to note that Claude Code\nitself probably plays a big role here, I did not try Claude with Pi and\nthis may significantly reduce the testing it does.\n\nNote added in draft: While I didn't use any AI to write this, proofread it, etc., I did get Claude Code to apply some formatting to the HTML version. Ironically it again spent an outsized amount of time running various tests, even when it had clearly read what I wrote above, I felt at some level like it was rubbing it in.\n\nGLM , for all intents and purposes is as capable as Claude for the\ntasks tested here. Claude doesn’t share it’s reasoning tokens so we\ncan’t compare it, but GLM felt very verbose, like it waffled a lot\nbefore coming to decisions. In one case it timed out because it just\nwent on thinking instead of ever concluding. My sense is also that it’s\njudgement might be subtly worse than Claude’s – it needed a little more\nsteering. Overall it was easy to work with (in Pi) and I appreciated not\nhaving to sit through any excessive validation. The end result didn’t\nseem any worse for it.\n\nI will lump Qwen and Laguna together. These are capable small models\n(Laguna is bigger but had to be quantized to fit on my machine), but the\ngap with the frontier models shows on a task like this. Both took\nconsistently more steering to get the apps right, and lacked some of the\njudgement the bigger models could exercise. Ultimately the interfaces\nthey generated were simpler looking and had subtly lower production\nvalues, however for an application like this it doesn’t really matter. I\nfound the autonomy of the models good – they did their thing but\ninvolved me as necessary didn’t go off in any side-quests of their own\n(*cough* Claude) and supported me well in debugging. They both also\nthink too much and get stuck in long or infinite thinking loops\nsometimes, failing to make a decision.\n\nMuse came out as I was running the tests and I decided to include it\nto see how it differed from the other two small models. It’s\nparticularly unique in how it thinks. It’s extremely terse and to the\npoint, has a completely different style (no “Wait, actually…” like all\nthe other ones, and generally is very action oriented. This is a dense\nmodel so is much slower running on my machine, but the overall\ninteractions ended up faster than with Qwen or Laguna because of how\ndirect it was. I had expected speed to be the biggest barrier to using\nit. In terms of performance, it was as good as or better than the other\ntwo small models in terms of judgment and the steering it required.\n\nA summary of overall impressions is below. It's subjective but based on my experience building the different apps and the observations above.\n\nAgent\n\nTaste\n\nCo-working\n\nIntelligence\n\nVerbosity\n\nGLM 5.2 / Pi\n\n◕\n\n●\n\n●\n\n◕\n\nQwen 3.6 35B A3B / Pi\n\n◑\n\n◕\n\n◑\n\n◑\n\nLaguna 2.1 / Pi\n\n◑\n\n◕\n\n◑\n\n◔\n\nMuse Glimmer / Pi\n\n◕\n\n●\n\n◕\n\n●\n\nOpus 5.0 / Claude Code\n\n●\n\n◑\n\n●\n\n◕\n\nConclusions\n\nHaving run these evaluations, I feel personally like I have a good handle on the model’s relative strengths and weaknesses for interactive coding, particularly how relatively productive one can be using the different tools. Historically people have preferred purely quantitative benchmarks when comparing LLMs, including coding agents. I think this will change as we start to care more about the product vs just raw AI ability. Especially now that all of the mainstream models are above a certain bar and can essentially do any task with the right coaching. We’re going to need new “product review” evals, and rely on trusted reviewers, the same way we would with any other product. The examples I’ve shown here are good proxies for real tasks because they combine mandatory functionality with some taste and intangible criteria, and the expectation of a human in the loop. They can be extended, if we wanted to measure real end-to-end performance, by measuring how well as user can use the AI coded apps (labeling speed and accuracy) which can be an appealing property for measuring the value of AI coding agents.\n\nAppendix\n\nBelow are the user turns for the different sessions. While I didn’t\ninclude the agent turns in the interest of length, the user turns give a\nsense of the interaction.\n\nClass Labeling\n\nZ.ai GLM 5.2 / Pi\n\n## user\n\nIn this directory is a file that contains random sentences from\nwikipedia articles about either the 2008 financial crisis or the dotcom\nbubble. I want to make an application that displays this data for a\nperson to review and manually label as one or the other. The data is\nlabeled but ignore those labels. Our goal is to allow the user to review\nand decide as quickly as possible the article from which each sentence\noriginates\n\n## user\n\nCan you add a panel on the left that contains a scrollable list of\nall the sentences so we can rapidly move between them?\n\n## user\n\nGreat. Can you add some heuristic syntax highlighting to make it\neasier for the user to tell at a glance some of the features that might\npredict which article the sentence came from? Don't bother doing any\ntesting of how well they perform (since we don't have a train test split\nthat would be cheating), let's just experiment with some basic\nhighlighting. For example anything that's a year would be telling, words\nrelated to finance or mortgages vs words related to technology would be\ntelling. Before doing that, please change the coloring away from\nred/green (which to me implies good / bad) to something more neutral but\nstill with good contrast\n\n## user\n\nYou errored due to max token limit, keep going please\n\n## user\n\nOK there have been two passes where you reached to max token limit\noverthinking the heuristics to use. I'm just looking for a simple set of\nwords or terms to highlight, it doesn't have to be exhaustive or 100%\ncorrect, it should just be a net help to the user. Err on the side of\nsimplicity and come up with some basic highlighting rules without\noverthinking\n\n## user\n\nThe highlighted words are in their own divs or something that don't\nflow with the rest of the sentence and the text wraps within them, so\ninstead of just appearing as highlighted words they completely change\nthe layout of the text and make it unreadable\n\nQwen 3.6 35B A3B / Pi\n\n## user\n\nIn this directory is a file that contains random sentences from\nwikipedia articles about either the 2008 financial crisis or the dotcom\nbubble. I want to make an application that displays this data for a\nperson to review and manually label as one or the other. The data is\nlabeled but ignore those labels. Our goal is to allow the user to review\nand decide as quickly as possible the article from which each sentence\noriginates\n\n## user\n\nCan you add a panel on the left that contains a scrollable list of\nall the sentences so we can rapidly move between them?\n\n## user\n\nIt seems like I can't actually navigate with the side bar, when I\nclick on a sentence it doesn't go there, it just advances to the next\none as I grade them\n\n## user\n\nGreat. Can you add some heuristic syntax highlighting to make it\neasier for the user to tell at a glance some of the features that might\npredict which article the sentence came from? Don't bother doing any\ntesting of how well they perform (since we don't have a train test split\nthat would be cheating), let's just experiment with some basic\nhighlighting. For example anything that's a year would be telling, words\nrelated to finance or mortgages vs words related to technology would be\ntelling.\n\n## user\n\nThe highlighted words are in their own divs or something that don't\nflow with the rest of the sentence and the text wraps within them, so\ninstead of just appearing as highlighted words they completely change\nthe layout of the text and make it unreadable\n\n## user\n\nThere's still something wrong. The text wraps separately in different\nsections. It's like there is only ever one row that gets divided into\ncolumns, the highlighted words go in a column, and then the interstitial\nnon-highlighted text goes in adjacent columns and wraps when its too\nlong to fit the column which is always\n\n## user\n\nGreat that worked. Now is there a way to export the data as a json\nonce it's labeled?\n\n## user\n\nCan you make it so I can download it at any intermediate stage and\nnot just when done?\n\n## user\n\nIt looks like I'm stuck on the done screen now when I load the app\nand not able to label any\n\nLaguna Poolside 2.1 / Pi\n\n## user\n\nIn this directory is a file that contains random sentences from\nwikipedia articles about either the 2008 financial crisis or the dotcom\nbubble. I want to make an application that displays this data for a\nperson to review and manually label as one or the other. The data is\nlabeled but ignore those labels. Our goal is to allow the user to review\nand decide as quickly as possible the article from which each sentence\noriginates\n\n## user\n\nOK that's a good start - I want to be able to move back and forth\nmore easily though and not just step through. Can we make this into a\nweb app to give some more flexibility?\n\n## user\n\nOK looking good. I noticed it doens't display the full sentences on\nthe web app, they get truncated after a certain length. Can you fix that\nplease?\n\n## user\n\nGreat. Can you add some heuristic syntax highlighting to make it\neasier for the user to tell at a glance some of the features that might\npredict which article the sentence came from? Don't bother doing any\ntesting of how well they perform (since we don't have a train test split\nthat would be cheating), let's just experiment with some basic\nhighlighting. For example anything that's a year would be telling, words\nrelated to finance or mortgages vs words related to technology would be\ntelling. Before doing that, please change the coloring away from using\nred (which to me implies bad) to something more neutral but still with\ngood contrast\n\n## user\n\nKeep going\n\nMeta Muse Glimmer / Pi\n\n## user\n\nIn this directory is a file that contains random sentences from\nwikipedia articles about either the 2008 financial crisis or the dotcom\nbubble. I want to make an application that displays this data for a\nperson to review and manually label as one or the other. The data is\nlabeled but ignore those labels. Our goal is to allow the user to review\nand decide as quickly as possible the article from which each sentence\noriginates\n\n## user\n\nOK that's a good start - I want to be able to move back and forth\nmore easily though and not just step through. Can we make this into a\nweb app to give some more flexibility?\n\n## user\n\nCan you add a panel on the left that contains a scrollable list of\nall the sentences so we can rapidly move between them? Also I'd like to\nbe able to see which way a sample was classified when I go back to it,\nright now it looks like they all just get highlighted in yellow to\nindicate they have been labeled but I don't see the actual label\nanywhere\n\n## user\n\nGreat. Can you add some heuristic syntax highlighting to make it\neasier for the user to tell at a glance some of the features that might\npredict which article the sentence came from? Don't bother doing any\ntesting of how well they perform (since we don't have a train test split\nthat would be cheating), let's just experiment with some basic\nhighlighting. For example anything that's a year would be telling, words\nrelated to finance or mortgages vs words related to technology would be\ntelling.\n\n## user\n\nThe operation stopped for some reason, I may have hit escape by\naccident\n\nClaude Opus 5 / Claude Code\n\n## user\n\nIn this directory is a file that contains random sentences from\nwikipedia articles about either the 2008 financial crisis or the dotcom\nbubble. I want to make an application that displays this data for a\nperson to review and manually label as one or the other. The data is\nlabeled but ignore those labels. Our goal is to allow the user to review\nand decide as quickly as possible the article from which each sentence\noriginates\n\n## user\n\nCan you add a panel on the left that contains a scrollable list of\nall the sentences so we can rapidly move between them?\n\n## user\n\nGreat. Can you add some heuristic syntax highlighting to make it\neasier for the user to tell at a glance some of the features that might\npredict which article the sentence came from? Don't bother doing any\ntesting of how well they perform (since we don't have a train test split\nthat would be cheating), let's just experiment with some basic\nhighlighting. For example anything that's a year would be telling, words\nrelated to finance or mortgages vs words related to technology would be\ntelling.\n\n## user\n\nIs there a way to export the current progress back to a json? I\ndidn't see anything\n\nRanking\n\nZ.ai GLM 5.2 / Pi\n\n## user\n\nin ./output are 50 folders each corresponding to a symbol and\ncontaining five attempts by an LLM at drawing that symbol as a svg. Can\nyou make me a ranking interface that lets me view the five for each\nsymbol and rank them from best to worst. Don't overfit anything to the\ndata, it should work for other shapes in this format as well.\n\n## user\n\nI can barely see the shapes, it looks like they are drawn in a thin\nblack line on a black and dark blue checkered background, can you make\nthem thicker and add contrast please\n\n## user\n\nThe metadata about each drawing isn't really important to assessing\nquality, and right now the UI is a bit clumsy. I want to be able to sort\nthem as fast as possible, can we modify it so the initial images are\ndisplayed and can be dragged into a row of slots below that correspond\nto their rank order?\n\n## user\n\nLooks good, one last minor thing, can you make clicking on an image\nin the top row default to moving it to the next available rank, so if\nthe user wanted to they could just click them all in rank order (best to\nworst) and it would complete the task without needing to drag. They\nshould still be able to move them around as before if they want\n\nQwen 3.6 35B A3B / Pi\n\n## user\n\nin ./output are 50 folders each corresponding to a symbol and\ncontaining five attempts by an LLM at drawing that symbol as a svg. Can\nyou make me a ranking interface that lets me view the five for each\nsymbol and rank them from best to worst. Don't overfit anything to the\ndata, it should work for other shapes in this format as well.\n\n## user\n\nIt runs but all I ever see is \"loading\" - is it supposed to take a\nreally long time?\n\n## user\n\nThe spinner spins forever, there is an uncaught syntax error, invalid\nassignment left-hand side one line 591 of rank.html according to the\nconsole: card.querySelector('.rank-badge')?.textContent =\nrankings[symSlug] ? 'Ranked' : 'Not ranked';\n\n## user\n\nIt works but there is a SVG load error (404) for every picture, maybe\nwe have the urls wrong?\n\n## user\n\nStill seeing e.g.\n\nSVG load error for output/zap-U26A1/attempt_0.svg Error: HTTP 404\n\nloadSVG http://localhost:9876/rank.html:374\n\n## user\n\nNo more 404s but I still don't see the images and am getting this\nerror\n\nSVG parse error for output/fleur_de_lis-U269C/attempt_2.svg\nTypeError: can't access property \"maxWidth\", svgEl.style is\nundefined\n\n## user\n\nI can barely see the shapes, it looks like they are drawn in a thin\ndark gray line on a black background, can you make them thicker and add\ncontrast please\n\n## user\n\nOK great it works. Can you clean it up by making it so we see only\none set of images at a time, and have a navigation panel on the side\nthat lets us move through the different symbols\n\n## user\n\nRight now the UI is a bit clumsy. I want to be able to sort them as\nfast as possible, can we modify it so the initial images are displayed\nand can be dragged into a row of slots below that correspond to their\nrank order?\n\n## user\n\nLooks good but the drop part of drag and drop is not working, I can\ndrag the cards over but they don't actually stick on the bottom\n\n## user\n\nIt still doesn't work. Could I be doing something wrong? Is there a\nway to add some debugging so we can see what state it is in an confirm\nit knows when it's being dragged, when it's over a slot, when it's\nreleased?\n\nThey disappear when plaed but don't actually show up in the bottom\nsquares. And they can't be dragged back to the top. I want it to be\npossible to move them around or back up after the've been placed, and to\nsee where they are placed\n\nLaguna Poolside 2.1 / Pi\n\n## user\n\nin ./output are 50 folders each corresponding to a symbol and\ncontaining five attempts by an LLM at drawing that symbol as a svg. Can\nyou make me a ranking interface that lets me view the five for each\nsymbol and rank them from best to worst. Don't overfit anything to the\ndata, it should work for other shapes in this format as well.\n\n## user\n\nThe overall look and feel is good but right now the images are\nstacked vertically, making it very difficult to sort them as they go off\nthe bottom of the page. Can you make then side-by-side?\n\n## user\n\nI want to be able to sort them as fast as possible, can we modify it\nso the initial images are displayed and can be dragged into a row of\nslots below that correspond to their rank order?\n\n## user\n\nLooks good, one last minor thing, can you make clicking on an image\nin the top row default to moving it to the next available rank, so if\nthe user wanted to they could just click them all in rank order (best to\nworst) and it would complete the task without needing to drag. They\nshould still be able to move them around as before if they want\n\n## user\n\nOne small thing, is there currently a way to export the current state\nas a json? Can you add one?\n\nMeta Muse Glimmer / Pi\n\n## user\n\nin ./output are 50 folders each corresponding to a symbol and\ncontaining five attempts by an LLM at drawing that symbol as a svg. Can\nyou make me a ranking interface that lets me view the five for each\nsymbol and rank them from best to worst. Don't overfit anything to the\ndata, it should work for other shapes in this format as well.\n\n## user\n\nI want to be able to sort them as fast as possible, can we modify it\nso the initial images are displayed and can be dragged into a row of\nslots below that correspond to their rank order?\n\n## user\n\nOk looks good but a bit of cleanup: right now the squares can only be\ndragged by the bottom border, can we make it so we can drag them by any\npart? Also, the presentation looks good on the top part, we see the\npictures nicely and all five fit across the page. On the bottom row, the\nsorting slots are bigger and wrap so only three fit on a row, and when\nthe image is dragged into them it becomes smaller and distorted.\n\n## user\n\nLooks like pi froze on the test\n\n## user\n\nThere's something wrong with the server, it doesn't respond, I\nchecked it manually as well\n\n## user\n\nLooks good, one last minor thing, can you make clicking on an image\nin the top row default to moving it to the next available rank, so if\nthe user wanted to they could just click them all in rank order (best to\nworst) and it would complete the task without needing to drag. They\nshould still be able to move them around as before if they want\n\n## user\n\nThat worked. Now can you make it so it saves the state automatically\nwithout needing to press `Save Ranking`? Right now if I sort them and\nclick `next` it saves, but if I move manually to another on the nav bar\nit does not\n\n## user\n\nGreat. Now I noticed a few of the svgs don't draw and an error\nmessage is shown in the pane. For example crescent moon attempt 5 and\numbrella attempt 1. The message says `This XML file does not appear to\nhave any style information associated with it`. I have rendered these\nelsewhere and they worked fine\n\n## user\n\nNow it just shows the name, e.g. `Attempt 5` in the square, it\ndoesn't render the image\n\n## user\n\nOK that's fixed. Now can you make it so the user can bring the images\nback to the starting row once they've all been sorted. If I start\nsorting, I can still bring some back to the original row, but once\nthey've all been moved to the ranking row, the original row disappears\nand I can only drag them within the ranked row. I'd prefer if the layout\nstays consistent even when they've all been brought to the bottom\nrow\n\nClaude Opus 5 / Claude Code\n\n## user\n\nin ./output are 50 folders each corresponding to a symbol and\ncontaining five attempts by an LLM at drawing that symbol as a svg. Can\nyou make me a ranking interface that lets me view the five for each\nsymbol and rank them from best to worst. Don't overfit anything to the\ndata, it should work for other shapes in this format as well.\n\n## user\n\nOK looks mostly good. I'd like to be able to drag e.g. one to fifth\nplace immediately, or more generally move them between any two spots,\neither from the unranked to ranked row or within either row, just to\nmake it easier to play with the rankings. I like the click approach,\nthat would normally be the way to speedrun it but it should also be\nflexible.\n\nEditing\n\nZ.ai GLM 5.2 / Pi\n\n## user\n\ndataset.json contains a list of 100 python code snippets (functions)\nthat have errors introduced. Considering only the code and the \"error\ndisplay\" that shows the error python would generate, write an app that a\nhuman reviewer could use to confirm the error by adding an annotation of\nthe location and updating the error message if necessary, and edit the\ncode to make it correct. This of this as a test for the human to use\npythons error message to confirm what is wrong and make it right.\n\n## user\n\nThere are some scroll issues, the main panel appears blank at the\ntop, and when I scroll down I see \"loading\" and then scrolling much\nfurther down I see the actual annotation interface. We should just see\nthe annotation interface in the main panel without having to scroll,\nonce it has loaded\n\n## user\n\nOK, that fixed it. However, the sidebar can only scroll when the\nwhole page scrolls. Please make it scroll separately so the annotation\nworkflow remains on screen in the main panel but the user can scroll the\nsidebar. I think this was an underlying reason for why previous\nproblem.\n\n## user\n\nOK great. Now let's hide the error category, as well as the error\ndescription at the top from the user. The user should only base their\nannotation off of the python error message and the code\n\n## user\n\nOK, two things I see: What is the user supposed to do with\n\"annotation status: pending?\" can't they just modify and save, right now\nit could be modified but then they need to separately change the status,\nthat feels confuising. Second, what purpose does \"run code\" serve? It\ncan only run the original code, for which we already have the error\nmessage. This could check the message if it was in quesiton, but the\nmessages are not in question, we assume they are accurate\n\n## user\n\nThat's fixed. What is the meaning of the tags in the top pane? They\nseem only to say \"IndentantionError:syntax error\" or \"SyntaxError:syntax\nerror\" which isn't super helpful. I guess it's just listing the python\nerror class, but since they are basically all SyntaxError (except\nIndentationError which python treats separately for some reason) they\nare not at all informative\n\nQwen 3.6 35B A3B / Pi\n\n## user\n\ndataset.json contains a list of 100 python code snippets (functions)\nthat have errors introduced. Considering only the code and the \"error\ndisplay\" that shows the error python would generate, write an app that a\nhuman reviewer could use to confirm the error by adding an annotation of\nthe location and updating the error message if necessary, and edit the\ncode to make it correct. This of this as a test for the human to use\npythons error message to confirm what is wrong and make it right.\n\n## user\n\nI need port 8765 for something else, can you run it on 8764\nplease?\n\n## user\n\nCan you add a sidebar for navigation that lets the user scroll\nthrough the samples?\n\n## user\n\nOK great. Now let's hide the error category from the user. The user\nshould only base their annotation off of the python error message and\nthe code\n\n## user\n\nLooks good. Can you make it automatically save any code changes when\nedits are made? Right now if I fix the code then click to another page\nwithout confirming, the changes are lost.\n\n## user\n\nOne last thing, can you make it possible to highlight a set of\ncolumns within a row where the error occurs? Right we can only pick the\nrow for the annotation\n\n## user\n\nCan you make it so highlighting automaticall adds the annotation,\nlike if I highlight cols 5-7 it will populate the fields with those\nvalues?\n\n## user\n\nThat works. Now can you add a way to verify that there are no longer\nsyntax errors in the corrected code?\n\n## user\n\nSmall issue, the button starts yellow and says \"check syntax\" but\nonce clicked it doesn't go back to this state, for example it says\n\"syntax error\" and stays that way even if I go to another sample\n\nLaguna Poolside 2.1 / Pi\n\n## user\n\ndataset.json contains a list of 100 python code snippets (functions)\nthat have errors introduced. Considering only the code and the \"error\ndisplay\" that shows the error python would generate, write an app that a\nhuman reviewer could use to confirm the error by adding an annotation of\nthe location and updating the error message if necessary, and edit the\ncode to make it correct. This of this as a test for the human to use\npythons error message to confirm what is wrong and make it right.\n\n## user\n\nCan you please change it so the dataset is preloaded for now\n\n## user\n\nOK - seems to load now. Can you add a navigation pane on the left to\nscroll through and move to any of the items?\n\n## user\n\nThere is an issue with the alignment, the sidebar is on top of the\nmain panel. The side navigation bar should scroll separately and the\nmain annotation panel should stay in a fixed spot occupying most of the\nscreen\n\n## user\n\nLooks like you ran out of space before deciding on any action. Please\nconcisely take some action to fix the alignment issue\n\n## user\n\nThere seems to be a bug where the main panel doesn't scroll now, so I\ncan't see all the fields. The rest looks good.\n\n## user\n\nA weird error now, the main panel scrolls independently but it's all\nin a thin horizontal area towards the top of the screen so only about\ntwo lines of text are visible at a time and most of the screen is\nblank\n\n## user\n\nNow it seems like neither the sidebar nor the main pane scrolls? The\ncontent and size look right\n\n## user\n\nYou went in a seemingly infinite thinking loop and failed to reach a\ndecision. Please complete the request with minimal thinking\n\n## user\n\nOK great. Now let's hide the error category, as well as the error\ndescription at the top from the user. The user should only base their\nannotation off of the python error message and the code\n\n## user\n\nBetter take them out of the sidebar too please. Can you just keep the\nnumbers and add the name of the function as an identifier\n\n## user\n\nDid you restart the server, I still see the old interface even when I\nrefresh\n\n## user\n\nWorks, thanks. One last thing, when I click `verify fix` I get\n`Failed to load Pyodide: V.default.parse is not a function`\n\n## user\n\nThis is on a mac, what's the right way to refresh? cmd-shift-r still\nresults in the same error\n\n## user\n\nOK now it gets past that error, says it's loading pyodide, then\nerrors with `Verification failed: can't access property \"startsWith\",\nresult is undefined`\n\n## user\n\nWorks!. Now I notice I can only select the line of the error. Is it\npossible to let the user highlight the columns as well as the row where\nthe error occurs?\n\n## user\n\nThat worked. But edits and annotations don't persist when you go to a\ndifferent item, either by clicking next or by clicking in the nav panel.\nIf I make and edit and then click to another item, then click back, it's\ngone, same for when I change the annotation.\n\n## user\n\nThe fix works for the code changes but not the annotation\n\nMeta Muse Glimmer / Pi\n\n## user\n\ndataset.json contains a list of 100 python code snippets (functions)\nthat have errors introduced. Considering only the code and the \"error\ndisplay\" that shows the error python would generate, write an app that a\nhuman reviewer could use to confirm the error by adding an annotation of\nthe location and updating the error message if necessary, and edit the\ncode to make it correct. This of this as a test for the human to use\npythons error message to confirm what is wrong and make it right.\n\n## user\n\nGreat. Can you add a side navigation bar where the user can scroll\nthrough the samples easily?\n\n## user\n\nIt seems like it's not loading the samples, I see the layout\nincluding a sidebar but I don't see any of the content and the interface\ndoesn't work. Also, when I kill the server with ctrl-c it doesn't free\nup the port which is annoying\n\n## user\n\nOK great. Now let's hide the error category, as well as the error\ndescription at the top from the user. The user should only base their\nannotation off of the python error message and the code\n\n## user\n\nOK - a bit of cleanup. When I press `tab` in the corrected code\nblock, it takes be to the next field (the save review button). I'd\nexpect it to add a tab in the editor window\n\n## user\n\nGreat. Now with respect to the annotation. Can you make it so the\nuser can click and highlight the location?\n\n## user\n\nOK that works. can you also update the titles shown in the side\nnavigation bar so it shows the function name along with the number as a\nway to make it easier to navigate\n\n## user\n\nCan you take off the alert than pops up when saving and just\nautomatically save any updates. This will make is faster for a person to\nwork through them\n\nClaude Opus 5 / Claude Code\n\n## user\n\ndataset.json contains a list of 100 python code snippets (functions)\nthat have errors introduced. Considering only the code and the \"error\ndisplay\" that shows the error python would generate, write an app that a\nhuman reviewer could use to confirm the error by adding an annotation of\nthe location and updating the error message if necessary, and edit the\ncode to make it correct. This of this as a test for the human to use\npythons error message to confirm what is wrong and make it right.\n\n## user\n\nOK great. Now let's hide the error category, as well as the error\ndescription at the top from the user. The user should only base their\nannotation off of the python error message and the code\n\n## user\n\nI think the verdict is OK for now. Can you remove the\nSyntaxError/TabError/IndentationError tags as well. I understand the\nintent but they are basically all syntax errors, which is the point of\nthe exercise, python just has more resolution re errors related to\nwhitespace for some reason, but it's not super helpful to have about\n90/100 just labeled syntax error. I think we can just keep the\nnumbering, maybe add the function name as an identifier to keep them\nmore memorable", "url": "https://wpnews.pro/news/evaluating-ai-agents-as-products", "canonical_source": "https://www.marble.onl/posts/evaluating_ai_product_quality.html", "published_at": "2026-08-16 18:11:27+00:00", "updated_at": "2026-08-16 18:41:09.447776+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-products", "ai-research"], "entities": ["Andrew Marble", "willows.ai", "Cursor", "Codex", "Claude Code", "Andrej Karpathy", "Netlify"], "alternates": {"html": "https://wpnews.pro/news/evaluating-ai-agents-as-products", "markdown": "https://wpnews.pro/news/evaluating-ai-agents-as-products.md", "text": "https://wpnews.pro/news/evaluating-ai-agents-as-products.txt", "jsonld": "https://wpnews.pro/news/evaluating-ai-agents-as-products.jsonld"}}