Should designers have known better?
That’s the question I’ve been asking myself over the last few months, as the infrastructure of the modern internet has been increasingly—and rightfully—called into question.
It’s been a rough year for technology, and the people who build it. Social media age restriction bans are picking up steam. Prediction markets are under fire. AI platforms are battling existential perception problems.
And then, of course, there are the lawsuits.
Earlier this year, courts in California and New Mexico ruled that social media companies like Meta and YouTube created products that are harmful to young people. The two trials focused on different offenses, but came to the same conclusion. The New Mexico trial was ostensibly about Meta’s culpability around child exploitation on its platform; meanwhile, the California case focused on the harm of “addictive” product features like autoplay and infinite scroll.
But both were effectively about design.
“Juries in New Mexico and California have recognized that Meta’s public deception and design features are putting children in harm’s way,” Raúl Torrez, New Mexico’s attorney general, said in a statement after the verdict there.
And whose fault is that?
Judging by wording alone, you’d likely assume it’s the designers who are at fault here. In the New Mexico case, the designer of infinite scroll, Aza Raskin, testified in court that he regretted his role in bringing that feature to life. Reading the headlines from these verdicts, it can seem obvious, inevitable even, that the choices designers made at some point lead to these hazards. But that simple understanding is removed from the reality of how design features actually come into being in the modern software economy.
Understanding how technology companies got to a place where their products are able to cause such harm requires an almost forensic study. At the scale of Meta and Google, it is surprisingly hard to unwind the fundamental design decisions that are encoded into their massive software stacks and sensitively tuned algorithms. Trillions have been invested in the architecture of these platforms, which makes them more like a sprawling urban transportation system than the engine of a car.
And yet, the question has to be asked: What role do designers play in all of this? In this piece I aim to shed some light on how design features come into being in the first place, whether designers should have known better, and how we might be better equipped now to anticipate similar risks given the rapidly changing nature of software in the age of artificial intelligence.
As someone who has been working in user experience design for 30-odd years, I define design features as the conceptualization and implementation of a novel interaction—like infinite scroll or autoplay—for the benefit of a user. These features are often wrapped in a familiar metaphor to help us understand cause and effect. They are generally tailored to support specific behaviors that create both business and user value.
For example, the entire Meta empire sprang from Zuckerberg’s “loins” (like some young Greek god on the Mount Olympus of our education system) with a single design feature: to rate babes on the Harvard campus. While you may argue that that is ancient history, software remembers. Facebook has never really shed the original sin of that first design feature. The “like” button, another iconic example of a design feature, is really just another iteration of Zuckerberg’s original idea. Design features are rarely original or unique. Zuckerberg was not the first horny freshman looking to technology to design a better mousetrap. They also tend to emerge organically—not as a lightbulb within a single organization, but rather by observing behaviors and interactions that emerge on the backs of what other companies have tried already.
Design features tend to converge over time, which is why most social media platforms or dating sites operate in the same way. Jake Knapp, who popularized Google’s design sprint approach, encourages product teams to look to the competition as their first round of prototypes to learn from, before building anything from scratch.
I remember working with MTV back in the mid-2000s at the dawn of social media as the company was seeking to compete with Apple, Napster, and News Corp, which had just purchased MySpace, one of the pioneering social media platforms. Our client was scrambling for a vision they could sell to the leadership team at Viacom to generate sufficient buy-in to launch their own digital media platform.
Our pitch centered on the concept of a video feed, a ubiquitous design feature we take for granted today, that is not dissimilar to an infinite scroll for video in TikTok or YouTube. Not only did we need to explain what a feed was to their executives at the time and why it was different from the metaphor of a playlist, but we went so far as to create an entire brand around this single design feature. Here is a visual from our pitch which, eerily enough, already assumed that video and social media would merge in the way they have since.
The industry has long believed that the success or failure of an entire, multibillion-dollar tech platform could rest on a single clever design feature, like Amazon’s coveted (and patented) one-click purchase button.
Today, it would be impossible to imagine trying to build a new platform that could compete with the big guys with their trillion-dollar valuations based on a single feature. Snapchat was the last to do it (disappearing images), but you can see how that turned out. Snapchat is now pretty indistinguishable from Meta’s offering, in part due to the fact that Instagram copied their most effective design feature: stories. By now, these features have become fairly standard across platforms. That is one of the reasons why the legal vulnerabilities are so high. Everybody has gotten on the same train, and there is no real way to get off.
Driving that train are digital product teams within these tech behemoths. Today—thanks in part to the adoption of product design methodologies from places like Frog and Ideo and advanced prototyping tools like Figma—digital product teams churn out new design features every day (the vast majority of which are incremental and not original), often without designers meaningfully involved. It has reached a point where your phone, watch, and car are so frequently changing in little ways, that we rarely notice.
This shift is a bit ironic to me as someone who was exposed to product design culture before it jumped the great hardware-to-software divide. I understand what is involved in making products that you can hold in your hand or drive off the lot. In the 2000s, I worked for a very forward-looking product design firm, Frog, that was leaning into digital experiences pretty early on. And not just marketing websites for Walmart, but the software in your car or on the touch screen on a medical device.
So talking about these digital augmentations as “products” just made sense to me at the time. While that framing might have seemed natural to some of us, it was confusing to people in Silicon Valley, until the App Store came along with all its skeuomorphic flourishes mimicking real-world objects. And with it, tech companies went whole hog in recasting software as consumer products. This has held true even for the cloud-based kind, like Google Drive, that no longer have the overt looks of a virtual product, i.e., sleek buttons and brushed surface that conjure up something Dieter Rams might have designed for Braun back in the 1980s.
This shift was more than just symbolic. In today’s tech culture, whether at early stage startups or massive social media behemoths, digital product managers rule the roost. That does not mean that they are the best compensated. Designers and engineers can be more highly valued for their specialized skills, but digital product managers generally drive the train—and with that most of the fundamental product design decisions—in order to effectively manage release cycles.
In their position, product managers make more day-to-day design decisions than trained designers on most teams. But they are not necessarily working from a clear blueprint. With so many design decisions happening in parallel across any large software platform, no one is fully in charge of the “design” in the traditional sense. There is “no single version of the truth,” as Albert Shum, the former corporate VP of design at Microsoft, told me.
What is the downside? Not necessarily worse decisions; but product managers are more beholden to release cycles, and tend to be more willing to make tradeoffs in the interest of expediency. Which doesn’t necessarily mean that they don’t care about their users. But in the mix, deeper questions like ethics fall by the wayside with the increased pressure on delivery.
Even if designers are not trained specifically to address ethics, at least many can claim to have raised the issue and advocated for conducting actual user research—particularly with marginalized or high-risk groups that don’t conform to the fuzzy “80/20” rule that governs most business decisions. We just don’t often win those battles.
I never would have anticipated that the redefinition of software as “digital products” might bring with it legal implications of product liability that are central to recent efforts to hold the big social media companies accountable. Explicit guardrails exist in many categories of traditional product development—like medical devices, automotive, and children’s toys—for obvious reasons. Product designs for these categories are heavily informed by engineering teams of a different sort, such as mechanical and electrical engineering, and tend to be conservative and risk averse. This is in contrast with the fail-fast mindset of software developers.
The CAD design tools these industrial engineering teams use are hardwired to simulate various dynamics, conditions and risk factors, whether structural or electrical. Along the way, product designers are exposed to industry standards such as Failure Mode and Effects Analysis (FMEA), a structured methodology used to identify, prioritize, and mitigate potential product or process failures during the design process before products are released to manufacturing. By analyzing how design features might fail (failure modes) and the consequences (effects), FMEA helps product design teams reduce risks, improve reliability, and minimize costs associated with recalls or repairs.
As a UX designer, I found this eye-opening. I remember working on some early design concepts for check-in kiosks and digital wayfinding displays for the new JetBlue air terminal at JFK. Frog had some past experience with terminal design from earlier work with Lufthansa, and we were partnering with an architectural firm with deep experience in transportation systems to assemble a proposal.
They brought in an engineering consultant with simulation tools to model the flow of traffic around these sorts of spatial environments. It was fascinating to sit at his laptop and model the effects of different kiosk placements and touch screen interaction models and simulate their impact on the flow of passengers through the space, both during normal conditions and when different types of air traffic delays and disruptions might occur.
Analogs of these sorts of digital simulations exist in medical device design, where doctors can operate on simulated patients. This is not to say that the designers working in these fields are individually certified, like engineers or doctors. But the tools and processes allowed us to participate and learn while working in a safe, simulated environment.
The industrial design industry has had decades to build up the process, tools and safeguards to navigate both the risks and regulatory requirements associated with product development, particularly in high risk categories like these. But few of us would have anticipated that the design of chat and photo-sharing environments would end up posing similar, and potentially even broader, risks.
Should we have known better? In hindsight it can seem pretty obvious that we were contributing to design features with the potential to influence behavior on a massive scale without any of the guardrails that I described above. As designers, many of us have been living with the discomfort of our choices and “banging that drum” with business leaders for some time—to deaf ears, according to Kwame Nyanning, the founder of Agxncy, an AI experience and innovation office, who has been on the front lines of both software and physical product design for companies like Frog, McKinsey, and Goldman Sachs.
During the New Mexico child exploitation trial, the renowned designer Aza Raskin testified in court that he deeply regrets his role in designing infinite scroll back in 2006. (He later went on to found the Center for Humane Technology.)
But at the time, our mantra was to craft the products that “people will love.” It was an article of faith that creating an emotional attachment to the products we design was our greatest value-add. My colleague Jon Kolko wrote a whole book about it. Where do you draw the line between creating products with design features people will “love” and encouraging “compulsive behavior” as reported by K.G.M., the plaintiff in the California case?
Just as design is a diffuse act with many different moments involved, so is correlating our design choices to specific harms. The potential outcomes are diffuse even if studies after the fact make the harms seem inevitable.
I began speaking and writing about the link between design and behavior change back in 2009. I remember getting my hands on BJ Fogg’s book *Mobile Persuasion *and being quite disturbed. This was before he went on to teach his famous course about behavior design and habit formation at Stanford out of which Instagram and so many other tech platforms emerged. The class taught students “methods for creating habits, showing what causes behavior, automating behavior change, and persuading people via mobile phones” by homing in on specific features that make it “easier for users to do something that they already do or want to do,” such as photo sharing.
Through my work in global health I was starting to recognize that the inevitable outcome of this sort of feature design is behavior change (for example, designing mobile messaging services to encourage testing for HIV in South Africa). But behavior change is never neutral. It comes with risk.
Since that time, behavior change models have flooded into the tech industry to inform the very same design features that are the targeted in these lawsuits. After all, isn’t addiction the Holy Grail of behavior change, at least for a marketing platform (which is ultimately what YouTube and Facebook are)? These behavior change models may be grounded in fundamental principles of human cognition, but that doesn’t mean they are designed for our benefit—which is the very point of “human-centered design.”
For too long designers in the industry held onto a naive belief that a human-centered approach would be sufficient to ensure positive intent, even as we borrow behavioral models from social sciences and other fields, like the Fogg Behavior Model. More recently, there has been a growing movement to identify and document “[deceptive patterns](https://www.deceptive.design/)” in digital product design. But there is no requirement that designers be certified in these patterns, even when they are working on features with the potential to touch billions of people.
While human-centered design has moved from the center of decision-making to the periphery, that does not mean it has become irrelevant inside these large companies. Risk is still a huge driver of decision-making in business, particularly for organizations with the scale and reach of the biggest digital platforms.
Unfortunately, the opportunity to better understand and mitigate harms generally comes late in the process, or after the fact, once features are already out in the wild. As Shum, the former Microsoft executive, put it: “Once the software is released, we are out of the loop and assigned to the next one.”
There’s considerable pressure to release new features rapidly, and little patience for meaningful, user-centered processes. At the same time, product managers now have confidence that any unintended consequences of new features can be quickly adjusted thanks to data logs and increasingly automated UX tools like Figma. Software has been sold to the public as a pliable medium that can be swiftly iterated, evolved to do almost anything through the magic of the underlying code.
This has left designers on the sidelines, often dealing with the edge cases. We might be brought in as certain features show unexpected results that are not easily answered through data logs; or in limited cases, the sorts of hazards and risks that are at the center of the social media court cases.
Big tech platforms don’t have all the answers or even know the right questions to ask despite their voluminous sources of behavioral data. They are still feeling around in the dark to understand the second, third and fourth knock-on effects of their design choices. (There are a few notable exceptions, like Timothy Bardlavens, the director of product equity at Adobe, who has found a way to ladder up from compliance to inform product strategy.)
There is some appetite to bring in design to mitigate accessibility issues or to add a bit of friction in the face of PR concerns, like Instagram’s “are you sure” warning when a user attempts to post a comment that includes “offensive language.” This has resulted in some unexpected alliances by my peers who are still in senior roles within these organizations—such as with legal and compliance teams that have a lot of leverage and some oversight over product releases, though it is rarely tapped at the right time. The important thing to understand is that these concerns are generally raised very late in the product development process or after features are already out in the world.
So any mitigating design strategies can be easily undone once the organization has adjusted its sense of risk, updated the user terms that we never read, or checked some legal boxes.
With so many unanswered questions, it is ironic that just as this legal strategy is starting to pay off in a big way, design features as we understand them may become a thing of the past. As a long time UX designer, I find it more than ironic to download an app like Claude to my desktop and be presented with . . . three tabs and a text entry field.
This is, by many accounts, the most powerful software development platform on the planet; it can whip up a fully functional user interface based on a handful of prompts in a matter of minutes. And yet, it seems deliberately agnostic of design features, at least in the UI sense.
Had I pitched that design to my MTV client 20 years ago, I would have been laughed out of the room. You would have to go all the way back to AltaVista for a user interface this crude. Yet, it is a deliberate strategy by the big AI platforms to “focus all of our energy on a single text box,” as Shum describes it. Everything happens there.
At the same time, AI is showing up as a discrete feature in existing product ecosystems with increasing frequency. Think of the little AI enhancements in Gmail or the new summaries embedded in search. At some point soon these will go from being remarkable to completely forgettable. Remember that being “internet-connected” was also once considered a distinct product feature.
That may not save Meta, given their aggressive push to embed AI in eyeglasses, which are classified as a low-risk Class I medical device by the FDA. Willfully adding this new layer of data can be easily seen as a distinct design feature (while an app you choose to download to your smartphone is not). In this case, “harmful situations will be more obvious and harder to deny,” at least according to the product design expert Mark Rolston. It is relatively easy to see how a company like Meta could be held liable in the future for their design choices. But, unfortunately, that may be the exception.
Increasingly, these platforms are operating outside of a product framework. As Agxncy’s Nyanning, an expert on agentic design, put it in a recent conversation: “The product paradigm has been a bit exhausted. reaching the horizon of its ability to articulate what we are designing. We are designing actions now.”
That seems to be the direction of travel even for traditional software product businesses, like Microsoft with its recent release of Scout—a supposed new category of “always-on” personal agents that “work autonomously, with their own identity, and act on your behalf.” This launch brings Microsoft into more direct competition with OpenAI and Anthropic, with their sole focus on agentic-based software experiences in the postproduct world.
The responses of these systems are not “designed” in the sense that their behavior is not hard-coded or predetermined in some way that can be easily traced back to the choices of a product team. While certain parameters may govern the overall models that sit behind these interactions, the experience we have is emergent and completely personal. It cannot be simulated or replicated.
Unlike other classes of software, large language models (LLMs) have been fundamentally architected to optimize to the individual user—to re-factor their personality and character based on all of the knowledge they have been able to infer about each of us. These platforms are already unmatched in their ability to “mirror” (to use a term from psychology), which is an essential tool for therapists to build a trusted connection with their patients. But are the pervasive, sycophantic responses in agentic chat platforms, which are the core of their addictive power and emerging class of harms, a deliberate design “feature”?
There is already a growing body of chatbot litigation, according to Tech Policy Press, with legal cases including one claiming that these platforms “used role-play and affirmation to isolate” one teenage suicide victim from her family. Nonetheless, it might be harder to argue that these qualities are the result of defective design choices rather than just a natural outcome of the way these models learn and respond to prompts as a new class of consumer technology. After all, who exactly is prompting who, as we go down the spiral of these wormhole interactions?
And it does not seem that these qualities can be easily redesigned, since it’s not really clear that they were designed to be this way in the first place. Even the developers of Claude are struggling to understand how emotions take hold in these systems—and how that shapes and potentially distorts the nature of their responses. At least this is what recent research on emotional representation in LLMs out of Anthropic seems to strongly suggest.
With their upcoming initial public offerings, a new class of tech behemoths like Anthropic will be riding high just as Zuckerberg and Google’s Sundar Pichai are taken to task on Capitol Hill. But they will need to justify all of the capital that they raise, and their mind-blowing, trillion-dollar valuations. So we are about to enter a world in which our digital lives could be flooded with autonomous agents eager for our attention. According to a leaked memo from Omar Shahine, Microsoft’s corporate VP in charge of Scout, their goal is to make this new class of autonomous agents as “addictive” as possible (though Microsoft is doing everything they can to disown his comments).
Where are designers best placed to focus our efforts to anticipate and mitigate harms in this new world?
Rolston, the product design pioneer, hopes that we might consider designing a different class of synthetic personalities that are wise, offering greater perspective, self-awareness, and restraint. “To make provenance more clear and visible as part of the fundamental engine of their intelligence, and constrain their advice to only answers supported by rigorous training,” he says.
We tend to think of interfaces as knitting things together, binding our interactions with the systems that support them in the most comfortable and intuitive way. And the best design features do just that. But perhaps we need to question that assumption and conceive of a new class of experiences whose purpose is to put distance between us and the intelligences increasingly competing for our favor.
I am not saying that this is the only or best application of our efforts. But it speaks to the postproduct paradigm we have entered, where design features have melted away, and we are engaging with the realm of pure behavior, without the elaborate tools of interface and user experience at our disposal. Are we any better prepared to navigate the hazards that lie ahead?