cd /news/artificial-intelligence/can-ai-generate-good-designs-from-ra… · home › topics › artificial-intelligence › article
[ARTICLE · art-173475] src=nngroup.com ↗ pub= topic=artificial-intelligence verified=true sentiment=· neutral

Can AI Generate Good Designs from Raw User Data?

AI-generated interface designs built from synthesized research findings and recommendations met 8.4 percentage points more scorecard criteria on average than designs built from raw user comments, a statistically significant difference (p < 0.001), according to a study by Nielsen Norman Group. The study gave an AI model two design tasks — a one-page form for rebooking a canceled flight and a multistep checkout flow for an online store — with roughly 100 raw comments per task, generating 8 designs per task and context condition and scoring them blindly by a human grader and an LLM judge that agreed on about 96% of scoring decisions. The gain was larger for the flight-rebooking task (11.4 percentage points) than the checkout flow (5.5 percentage points), suggesting teams still benefit from synthesizing research before handing context to AI design tools.

key takeaway

Feeding an AI design model synthesized findings and recommendations instead of ~100 raw user comments raised scorecard criteria met by 8.4 percentage points on average (11.4 for flight rebooking, 5.5 for checkout).

by read7 min views2 publishedNov 20, 2026
Can AI Generate Good Designs from Raw User Data?
Image: Nngroup (auto-discovered)

AI can now analyze user data, provide design recommendations, and generate interfaces all in one go. So, is it still worth synthesizing research and developing recommendations first, or can teams hand AI the raw data and let it decide which user problems matter and how to solve them? The answer matters more and more as AI-generated prototypes become part of everyday design work.

To find out, we asked an AI model to generate interfaces from different types of research context. Across two design tasks, it produced better designs from synthesized findings and recommendations than from raw user comments.

About the Study #

We gave an AI model two design tasks: designing a one-page form for rebooking a canceled flight and designing a multistep checkout flow for an online store. In each task, the prompt and the design-system information remained the same, while the type of research context we provided to the model changed.

While the actual study design was more complicated, in this article we will focus on two types of context that the AI model received:

  1. Raw user comments , which included approximately 100 comments for each design task
  2. Synthesized findings and recommendations based on the same raw user comments (A researcher analyzed the comments with an AI tool and made the final decisions on which findings to include and what to recommend).

Since AI tools are nondeterministic, we generated 8 designs per task and context condition. Then, a human grader and an LLM judge blindly scored each design against a predefined scorecard. The score of the design was the percentage of scorecard criteria it met (e.g., “every checkout step shows the items being bought”).

We used the human grader's scores in the results. The LLM judge scored every design independently as a check, and after calibration, the 2 graders agreed on about 96% of scoring decisions in both tasks.

Details on the study methodology are available in the accompanying sidebar.

Synthesized Findings and Recommendations Led to Better Designs #

Across both design tasks, the AI-generated designs based on synthesized findings and recommendations met more scorecard criteria than designs based on raw user comments. The average difference was 8.4 percentage points and was statistically significant (p < 0.001).

The improvement was larger for the flight-rebooking task (11.4 percentage points compared to 5.5 percentage points for the checkout flow), but both tasks benefited. The finding suggests that design generation with AI can benefit from the work of synthesizing research data into findings and recommendations.

The flight-rebooking task shows what this difference looked like in practice. A few raw comments complained about loyalty miles, yet none of the designs generated from raw comments included information on whether the cancellation would affect passengers’ miles. In contrast, every design generated from synthesized research included that information.

The online-checkout task showed a smaller gain for the synthesized research but the same pattern. Several user comments complained that items disappeared from the cart during checkout, yet fewer than half of the designs based on raw data kept the items and total visible on every checkout step. Nearly all the designs made from synthesized research did. The smaller gain may be due to the task prompt, which was the same in both conditions and already pointed to some of the issues raised in the comments: It described a first-time shopper with a discount code who needed a gift within 2 days. Because the prompt itself flagged concerns such as applying a discount and receiving the order on time, even the designs made from raw comments tended to address them, leaving less room for synthesized research to make a difference.

The 8.4-percentage-point advantage of synthesized findings and recommendations may underestimate their benefit in more complex situations. First, our data was likely cleaner than typical user feedback. Although cleanliness is hard to measure, more than half of the comments in each task supported a finding, whereas research on app-store reviews found that only about a third of reviews contained information useful to product teams. Second, the dataset was small, about 100 short comments per task. Third, both tasks (designing a form and a checkout flow) are common patterns that AI models have likely seen many times. We expect the advantage of synthesized research to grow with larger, messier datasets and less familiar design problems, and we plan to study this hypothesis in the future.

Why Might Synthesized Research Improve AI-Generated Designs? #

Raw Data Contains Observations Only

Raw user data describes problems, such as what frustrated users and what they wished worked differently, but it rarely says what to build in response. For example, a comment such as "Items disappeared from my cart" names a problem but does not mention a solution. Even when users suggest a fix, the suggestion isn’t a reliable guide. Users are good at describing their problems, but they’re poor at predicting what they would actually use, and what they say doesn’t always match what they do. Raw data also tends to be messy. It can include comments unrelated to the design, and different user comments may contradict each other.

Synthesized Findings and Recommendations Add a Layer of Interpretation

When working only with raw data, the AI needs to decide which data points matter and what changes to make. Those decisions are difficult because user feedback is only one input. In real life, designers weigh user feedback against business needs, technical constraints, product goals, and what the team has already tried. AI tools struggle with those tradeoffs without human guidance.

Through findings and recommendations, researchers apply their judgment to turn raw data into design decisions. They add a layer of interpretation that can help the AI generate the right design. Without them, you leave those decisions to the AI.

Findings and recommendations play different roles, though. Findings interpret the data, while recommendations turn that interpretation into design decisions. Because our analysis compared raw user comments with synthesized findings and recommendations, we don’t know at this point whether the improvements were caused by the findings or by the recommendations. We are working on determining the effect of each in a separate study.

Implications for Product Teams #

Research artifacts now have two audiences: your teammates and the AI tools used in the design process. Along with artifacts created for people, we also must create artifacts that guide AI tools.

Don’t Assume Raw Research Data Is Sufficient Context

AI tools can process substantial amounts of user-research data, but that doesn't mean they will automatically identify the most important information and use it appropriately in a design. Consider what interpretation and guidance the AI tool needs rather than assuming access to the underlying data will be sufficient.

UX-Context Design: Think About What UX Knowledge AI Needs

As AI becomes more involved in generating interfaces, we will increasingly need to figure out what guidance and knowledge to provide to these tools. The context that you give an AI tool shapes what it creates. UX-context design is the practice of selecting, organizing, and maintaining the UX knowledge that guides AI-generated design. This study offers evidence that the content of this context can affect design quality.

Ongoing Research #

This study raised several questions that we’re still exploring.

  • Do synthesized research and recommendations improve design quality only on the specific issues addressed by the findings and recommendations, or in other respects too?
  • Do the results hold for other types and larger volumes of raw data than short customer comments, such as interview transcripts, usability-test notes, or survey responses?
  • Would other AI models produce the same results?
  • How much of the benefit comes from the synthesis of the findings versus from including explicit design recommendations?
  • Does the format of the file (e.g., Markdown, PNG, PDF) in which the context is provided make a difference?

We’ll keep studying how teams can give AI better UX context, so look out for future articles on these topics. To start applying these ideas now, see our Context Engineering: Shaping AI-Generated Designs course.

Conclusion

As AI takes on more of the design work, steering what it produces matters more. Our findings suggest that not all context is equally useful. Providing AI with synthesized research findings and recommendations leads to better designs than providing it with raw user comments alone. At a time when AI is raising questions about the future of UX, this finding is important. Generating an interface is just one part of design. Understanding which user problems matter for your product and translating that understanding into design recommendations remain an essential contribution of UX professionals, whether the designs are created by humans or AI.

References

Ning Chen, Jialiu Lin, Steven C. H. Hoi, Xiaokui Xiao, and Boshen Zhang. 2014. AR-miner: Mining informative reviews for developers from mobile app marketplace. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 767–778. https://doi.org/10.1145/2568225.2568263

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @nielsen norman group 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/can-ai-generate-good…] indexed:0 read:7min 2026-11-20 · —