Hi HN,
I'm Adam Ginzberg, one of the cofounders of Subframe, a code-first design tool.
One of the things I'm really curious about is understanding LLM performance on subjective domains like design.
As an experiment I built this game which presents two designs of the same screen, one of which was generated by AI. First you pick which one you like better and then which one you think was made by AI.
Most design work happens in an existing product, so we set up this game to test that. As context, the AI gets 4 screenshots from the existing product, the product's icons + fonts, and a written brief about the screen it is asked to build. It then writes HTML which is screenshotted identically to the real screen. To get the data, we signed up for the products with test accounts and added dummy data. All of the inputs were manually checked to make sure nothing about the target screen was leaked in the prompt. You can read more about the methodology at https://www.realdesignbenchmark.com/methodology.
After a few thousand submissions, the surprising result is that people prefer the AI design - 57% prefer AI, 33% prefer the real design, 10% say they are about the same. People are also not great at identifying the AI design, only 59% get it right.
The initial results were all generated with Opus 5.5 (medium effort). We just added GPT-6.1 Sol (medium effort) tonight.
Give the game a try. Feedback on the approach is appreciated. Or any ideas on what we might improve or try next.
Comments URL: [https://news.ycombinator.com/item?id=50099581](https://news.ycombinator.com/item?id=50099581)
Points: 1