In 2025, researchers studied experienced endoscopists who had begun using AI to help detect precancerous growths. When the doctors later performed colonoscopies without AI assistance, their detection rate had fallen from 28.4 percent to 22.4 percent. The study was observational, and so the researchers couldn’t say why the decline occurred. But it surfaced a possibility designers know well: a product can perform exactly as intended while changing the people and systems around it in ways nobody intended.

Every product team asks: “Does it work?” The team behind our Unintended Consequences tool wanted to make it easier to ask a second question: “What happens when it works?”

What skill might a person stop practicing? Who is affected by a product—even if they didn’t use it? What behavior does it reward by default? How might a prediction tool change the outcome it was meant to predict?

These effects are often hard to spot just from interacting with, or even designing, a demo. They emerge as people and institutions use a tool, often when the decisions behind it are too expensive to unwind.

We have long believed in designing with AI’s unintended consequences in mind. But too often, consequence thinking gets tacked on at the end of a project as a workshop, a checklist, or a review. Those forms can be valuable. But they can also make ethics feel like a gate: something a team passes through after the idea has already largely formed.

We wondered what would change if teams had a tool to pressure-test an idea while they were still shaping it. And then they used it again every time the idea changed—treating consequence-spotting as a potential source for design, rather than a filter.

AI engineers have their own practice for systematic scrutiny: the eval. They assemble test cases, run a system against them, and examine the results. We wanted to see if that basic pattern could be used as another way to approach our more human question: 

What are the consequences of this design—intended or unintended?

Building a critical tool

An obvious tension arises when we use an AI system to examine the consequences of AI-enabled design: it may perpetuate existing limitations, gaps, or unintended consequences within its own system. But we decided to work inside it and use the tension to better understand both our discomfort and our enthusiasm.

We reached for a large language model because the process we designed requires considering several kinds of information together: the details of a concept, varying perspectives, changing scenarios based on human needs, research, and a set of critical lenses. You can use a static checklist to apply a standard, and a workshop to create a responsive conversation. This prototype tries to do something in between: apply a consistent structure while responding to the specifics of the design under consideration.

That consistency is useful, but it’s different from expertise. We’ve designed the tool to explicitly include a set of perspectives rooted in varying identities and frameworks for considering harms. But like any other form of input, it has limitations and shouldn’t be used in isolation. The tool doesn’t know as much about a client's or consumer’s history with a product as the design team. It can be overconfident, flatten nuance, and make a speculative chain of events sound much more inevitable than it is. 

So we designed and positioned its outputs as provocations, not scores. They are there to support the team’s analysis, explicitly pointing them to sources and signals, calling out assumptions and why the system made them. People choose and edit the scenarios, decide which critical lenses to include, and are encouraged to challenge the analysis by considering who they ought to learn more from. 

To test it, we ran an early workshop version of the tool that eventually became Soluna, a youth mental health platform that IDEO helped launch with the state of California and Kooth. (This was purely to test the tool itself; the Soluna project was already complete.)

When the Unintended Consequences team ran a scenario where the Soluna app provided clear next steps to users through the tool’s Skill & Human Capability lens, the analysis warned that promise might cause users and referrers to “treat the product’s urgency judgment as definitive.” It also noted that routing crisis cases elsewhere “transfers demand rather than eliminating it” and that a shared entry point could become “a concentrated failure mode.”

As it turned out,  before Soluna launched, the internal design team had already addressed these concerns by establishing clear crisis boundaries, keeping direct crisis pathways visible, incorporating clinical oversight, and using trained people to recognize risk and manage escalation. (The Soluna design team grounded all its work in clinical measures like GAD-7 and PHQ-4 and 9, and worked with a clinical oversight board to oversee the products that they were making.) A strong design principle emerged: keep one welcoming front door, but preserve many safe exits.

The testing experience reinforced something we believe about designing with AI: preserving human judgment isn’t ambitious enough. The goal should be to enhance our judgment and critical thinking, and we can only do that if the interaction gives us something meaningful to judge.

Generative critique tests like this one changed our understanding of the tool. We expected it to help teams find risks. And it did—but the more interesting moments came when a critique opened a new creative direction.

In one session, a team used the exercise on an active design challenge and told us it changed how they framed the work. They described the process as generative, not only cautionary. Other testers found questions they had sensed but had not yet been able to articulate.

In the Soluna project test, we also used the Who Is Affected Beyond the User lens. The tool raised a privacy flag, outlining a potential scenario where it could affect others beyond the primary user: “Persistent activity across visits may expose sensitive information on shared devices unless local privacy controls accompany the stated privacy explanation.” Policy-level confidentiality does not prevent a parent, partner, sibling, or roommate from seeing an app, notification, journal entry, or open session.

When we shared our findings with the original project team, they were inspired to consider future versions of the tool that could offer clear, optional, user-controlled protections, allowing each young person to choose the balance of discretion, convenience, and accessibility that fits their circumstances.

The prototype of the Unintended Consequences tool was most useful when it behaved less like a compliance officer and more like a good design critic: providing scenarios specific enough to respond to, structured enough to reveal a pattern, and unfinished enough to invite the team back into the work.

That led us to three principles we are carrying forward:

1. Start testing for consequences before the idea feels finished. It’s most valuable while the brief, defaults, roles, and business model can still change.
2. Trace every concern to a design choice. A team cannot redesign society, but it can redesign an interaction, an incentive, a boundary, or a handoff.
3. Use critique to create options. The goal is a wider field of possibilities for creating a positive impact for more people.

Moving consequence thinking upstream

The prototype we built can be used with a design idea at almost any level of fidelity: a brief, a concept, a flow, or a more detailed product description. A team then creates plausible scenarios involving different people, contexts, and conditions. The tool projects how the design might behave in each scenario and examines the result through selected lenses, such as human capability, power and agency, effects beyond the user, or feedback loops. 

The point is to give a design team more futures to react to.

Building new habits

This tool and these experiments will not eliminate unintended consequences. Complex system behavior cannot be fully anticipated. But we can change when teams look for those consequences, how often they look, and what they do with what they find.

Usability testing was once treated as specialists’ work. Now it is an ordinary part of making many products. We think consequence testing can make a similar journey: from an occasional expert review to a repeatable design habit that strengthens the viability lens.

The prototype itself is not a public tool; it carries IDEO methods and research developed through years of practice. But the question at its center is one any team can use:

If this product works exactly as intended, what else might it do?

Ask it while you can still do something about the answer.

Interested in an upcoming workshop on responsible innovation? Fill out the form at the bottom of this story, and we'll be in touch!

‍

‍

Curious about the lenses the tool uses?

Skill & Human Capability

• What skills, knowledge, or judgment might people stop developing or maintaining because of this system?
• If this system were removed after years of use, could people still function effectively without it?
  

Power, Authority & Agency

• Who gains influence or decision-making power from this system, and who loses it?
• When this system makes a recommendation, how easy is it for someone to say no?
• Does this system concentrate control in ways that are hard to reverse?
  

Who Is Affected Beyond the User

• Who is affected by this system but never directly uses it or consents to it?
• Whose labor, data, or attention does this system depend on—and are they fairly compensated and acknowledged?
  

Feedback Loops & Self-Reinforcing Dynamics

• Could this system’s predictions or outputs influence the very outcomes it tries to predict?
• Are there ways this system could narrow choices over time—homogenizing behavior, culture, or options?
  

Human Relationships & Social Fabric

• What human relationships or interactions does this system replace, mediate, or change?
• Could this system erode trust between people, or between people and institutions?
  

Identity, Memory & Sense of Self

• Does this system shape how people see themselves, their abilities, or their future?
• Could the system create a gap between what people genuinely want and what it leads them toward?
  

Scale & Systemic Risk

• What happens at the margins—the edge cases no one designed for?
• If millions of people use this simultaneously, what macro-level effects emerge that wouldn’t be visible at small scale?
  

Transparency & Accountability

• Can users understand why this system made a particular recommendation or decision?
• When something goes wrong, is there a clear path to identify what happened and who is responsible?
• How might people repurpose this system? Are there avenues that could lead to exploitation or misuse other than intended purposes?

Misuse & Adaptation

• How might people repurpose this system? Are there avenues that could lead to exploitation or misuse other than intended purposes?

‍

‍