AI Tip #2: Twenty directions and a page to compare them

A developer outlined a workflow for working with coding agents that replaces turn-by-turn iteration with parallel exploration: prompting the agent to generate 20 variations of a screen or flow and build a single HTML comparison page that displays them side by side, lets sections be swapped between variations, and adds a notes field under each. The approach, drawn from Thorsten Ball's 15 rules for working with agents, also covers asking for multiple theories on a problem before redesigning and demanding concrete proof such as screenshots at 375, 768 and 1440 px, a full click-through, and accessibility checker output rather than the agent's own summary.

Context In Tip 1 I said my day-to-day is mostly exploration. My skills push the UI in many directions, I pick one, and I take it back to users. This one is about what happens on either side of “I pick one”: how I compare the directions, and how I know the one I picked works. Two things I read cover it. Thorsten Ball’s “How I work with Agents,” and Hiten Shah’s Early Access Playbook. Thorsten builds Amp. His list is 15 rules for working with coding agents, as of October 2026. Most of them are about code. One reads like a design crit: Don’t iterate with the agent. Let it build tools to iterate. Not “let’s try this. ... ah no, let’s try that.” Instead: “Generate 20 variations and a little page that lets me combine and test them.” He uses the same move on the problem, not just the solution: “Give me 4 theories on why this is slow, then check all four.” Most of us work with a model the first way. Ask for a mock, say what’s off, ask again. Every round starts from the last guess, so twenty minutes in you’re still polishing direction one. Why does it work? A chat thread shows you one option at a time, and each fix is built on the option before it. A comparison page shows the whole range at once. You can spot that the header from variation 4 works with the layout from variation 11, and no single round of feedback would have produced that pairing. It also moves your effort to the part only you can do. The model makes 20. You judge 20. Judging is where taste lives. The variation page, the prompt I’d start with: Generate 20 variations of screen or flow for user type trying to task . Vary them on layout / hierarchy / density / tone , not just color. Then build one HTML page that shows them side by side at mobile and desktop widths, lets me swap individual sections between variations, and has a notes field under each one. Label each variation with the one decision that makes it different. How to do it Rung 1: the variation page. The prompt above. Ask for range on purpose, or you’ll get 20 copies of the most likely answer in different colors. Rung 2: theories before redesigns. Thorsten’s second example is about the problem. Before you redesign a drop-off, ask for several explanations and have the model test every one. Our metric drops at step . Give me 4 theories for why, using session data / funnel events / support tickets / call transcripts . For each theory, say what we’d see in the data if it were true, then check. Rank them by how well the evidence supports them, and tell me which one you couldn’t test. Rung 3: proof, not a summary. Thorsten calls asking for irrefutable proof his “biggest game changer.” Agents are good at producing it: end-to-end walkthroughs, demo videos, before/after numbers. He puts “this is not evidence” in his AGENTS.md, so the agent stops reporting its own confidence as a result. For a screen, proof is screenshots at real breakpoints, a recorded click-through of the whole flow, and an accessibility checker’s output. A paragraph saying the design “improves clarity” proves nothing. Before you tell me this works, show me: screenshots at 375, 768, 1440 px, a click-through from entry point to goal covering every state including errors and empty states, and the output of accessibility checker . If you can’t produce one of these, say which and why. A description of the design is not evidence. When should I use this? The tell is round four of “make it more X.” That’s iterating with the agent. Use the page when you’re choosing a solution, and the theories when you’re still not sure what the problem is. The mistake to expect: treating the agent’s proof as proof the design is right. A click-through shows the flow works. It can’t show that anyone wants it. That’s Hiten’s half. Proof from people, not just the agent Hiten’s playbook opens at KISSmetrics, where his team spent roughly eighteen months on two product directions that missed. His line for the AI version of that problem: “The faster you can build, the faster you need to learn.” Twenty variations in an afternoon is the speed he’s warning about. Steve Blank, whom Hiten quotes, has a name for what it can produce: the Initial Untested Product. It looks finished before anyone has checked that people need it. Close cousin of Katie Dill’s zombie UI from Tip 1. https://johnmaartifacts.substack.com/p/ai-tip-1-design-for-the-metric-not Hiten names four ways customer research misleads you: - Feature requests become the roadmap. - Enthusiasm gets mistaken for demand. - Repeated complaints look like market evidence, when everyone complaining belongs to one narrow group. - One successful task becomes proof of a business. His fix is to start with things that already happened. The question he opens with is “Tell me about the last time you tried to do this.” Then what happened first, which tools they used, and where it got hard. He’s blunt about AI here. Synthetic interviews can help you prepare questions. They can’t tell you what a customer did. In his words: “You can generate a hundred plausible customer conversations without learning what anyone will do with your product.” Grade your research the way you grade the agent The most useful page in the playbook is an evidence table. Each kind of signal, what it tells you, and what it doesn’t: I collect my user transcripts with Granola, and this ladder is how I’d sort them. Here are number user transcripts: transcripts . List every claim users made about problem or feature . Tag each one with the strongest evidence behind it, using this ladder: talked about it / committed time / completed a real task / succeeded again when the need came back / seen in a second group. Keep what someone said they’d do separate from what they described doing last time. Then tell me which of my current design decisions rest only on the bottom two rungs. Keep the learning next to the decision Hiten adds a four-line note whenever something important happens. He calls the running list a Learning Thread: We believed: what we expected, including the biggest unknown. We saw: what someone reported or what we observed, with a link to the evidence. We decided: what we’ll do next, and why over the alternatives. We’ll check: the next real outcome that could confirm or change the decision. Thorsten lands in the same place from the agent side. Threads are company memory, and “Why did we do this?” should have an answer you can find. For each decision we made in this thread, write a Learning Thread entry: We believed / We saw link the evidence: variation number, screenshot, transcript line / We decided / We’ll check. Flag any decision where “We saw” is only your own reasoning. Test your own agent files too One more of Thorsten’s rules, for anyone keeping a design system for agents. Knowledge piles up in AGENTS.md files, skills, scripts and docs, he says. But do agents actually use it? “Spawn five blind agents, control vs. treatment, and find out.” The design brain I built for my team at Meta started as 12 skills and 5 design agents, with a RULES.md read on every run. If you have a setup like that, this test tells you which files earn their place. Run the same design task 5 times without RULES.md / skill name and 5 times with it, in fresh sessions that can’t see each other. Compare the outputs on criteria . Tell me which rules visibly changed the result and which made no difference. Too long, didn’t read - Don’t iterate with the model. Ask for 20 variations and a page to compare and combine them. - Ask for theories before redesigns, and have the model check every one. - Agent proof screenshots, click-throughs, checker output shows a design works. Only people show it’s wanted. - Sort what users tell you by what it proves, starting from “the last time you tried to do this.” - Write down what you believed, saw, decided and will check, right next to the decision.