Part 3 of the “UX × AI” series.

In Part 1 of the series, we reframed the fear: AI is your intern, not your replacement. In Part 2, we recognized the skill you already have — prompting is brief writing, and designers have been doing it for their entire careers.

This article is the one I have been most impatient to write.

Because of all the language that has entered the design conversation since AI arrived in force, there is one category of language that concerns me more than any other. Not the hype. Not the fear. The anthropomorphism.

The AI that “understands” your users. The AI that “empathizes” with their needs. The AI that “knows” what they want before they can articulate it. The AI-generated persona that “feels” frustrated by a checkout flow. The AI research synthesis that “reveals” what users are really thinking.

This language is everywhere. It is in product marketing. It is in the design team Slack channels. It is in leadership presentations justifying the replacement of user research with AI-generated insights. And it is causing harm — specific, measurable, designable harm to the people that UX design is supposed to serve.

This article is a direct challenge to that language. Not a philosophical one. A practical one. Because the distinction between what AI actually does and what the word “empathy” means is not a semantic argument. It is a design decision, and the wrong design decision, made at scale, produces products that fail the humans who use them in ways that pattern-matching will never detect.

What empathy actually is: Before we discuss what AI lacks

Let me start with the thing that the design community has been imprecise about long before AI arrived.

Empathy is not a warm feeling toward users. It is not the ability to imagine what someone might be experiencing. It is not reading a research report and nodding along. And it is definitely not a UX deliverable — a persona document, an empathy map, a journey map — produced at the beginning of a project and consulted occasionally thereafter.

Empathy, in the genuine sense that grounds good UX research, is the disciplined capacity to be changed by another person’s experience. To sit with a person in their context, on their terms, and to let what you observe and hear alter your understanding of the design problem. To come out of a research session holding a model of the world that is different from the one you entered with. To have your assumptions corrected by reality, rather than confirmed by data that reflects your existing beliefs.

This is a demanding practice. It requires presence — the physical and cognitive discipline of being fully attentive to another person rather than attending to your own interpretation of what they are saying. It requires humility — the willingness to be wrong about what you thought you understood. It requires the specific professional skill of qualitative interpretation — the ability to distinguish between what a participant says, what they mean, and what their behavior reveals that neither their words nor their stated meaning captures. And it requires what the cognitive psychologist calls the theory of mind — the ability to model another person’s mental state, not as a statistical abstraction, but as a specific, situated, contextually embedded experience.

AI does none of this. Not because it has not been trained on enough data. Not because the models are not yet sophisticated enough. But because empathy, in this sense, is not a pattern-matching capability. It is a relational one. And AI does not have relationships. It has training data.

“AI doesn’t care about context, only patterns. User experience is about more than click paths and flows — it’s about earning users’ trust and creating joy and a sense of security. These kinds of values cannot be fully quantified, and therefore they cannot be fully automated.” — Epinova (2025)

What AI actually does, precisely

I want to be precise about what AI does when it appears to understand users — because precision here is not pedantry. It is the foundation of using these tools wisely rather than dangerously.

When you give an AI tool a dataset of user interviews and ask it to identify themes, it performs statistical pattern-matching across the text. It finds words and phrases that co-occur frequently. It identifies clusters of similar statements. It surfaces language patterns that appear consistently across the dataset. This is genuinely useful. It is fast, it handles scale that human researchers cannot manage manually, and it produces a first-pass map of the dataset that can orient the researcher’s interpretation.

But it is not an analysis. It is indexing. The AI has not understood what the users said. It has mapped the frequency and co-occurrence of the words they used. The interpretation — the act of asking what these patterns mean, why they exist, what they reveal about the underlying experience of the person who produced them, and what design implications they carry — is absent from what the AI has done. That work belongs to the researcher. And when it is skipped — when the AI’s pattern map is treated as the analysis rather than as the raw material for analysis — the design decisions that follow are built on a foundation that has never actually touched a human being.

When you ask an AI to generate a user persona, it produces a composite profile assembled from its training data — demographic information, behavioral patterns, stated preferences, and contextual details that are statistically plausible given the parameters you have provided. The persona looks like research. It has a name, a photograph description, a set of goals and frustrations, and a day-in-the-life narrative. It is entirely fictional — not in the sense that good research-based personas are fictional while capturing truth, but in the sense that it has no relationship to any actual human being who actually experiences the problem your product is designed to solve.

Nielsen Norman Group tested AI-generated synthetic users against real user research with genuine participants. Their finding was unambiguous. Synthetic users were “one-dimensional” — they produced superficially plausible responses without the complexity, contradiction, context dependence, and genuine frustration that real users reveal. In one study, synthetic users reported completing all online tasks successfully. Real users reported dropouts, confusion, motivational failures, and workarounds. The synthetic users told the researchers what they wanted to hear. Real users told the researchers what was true.

This is the defining limitation of AI-generated user understanding, and it is one that no amount of model improvement will fully resolve. AI is trained on human-produced data. Human-produced data — the text, the surveys, the forum posts, and the published interviews — is systematically biased toward articulate, digitally engaged, positively framing participants. The people who are not represented in that data — the elderly farmer in Vidarbha, the first-generation smartphone user in Dhanbad, the person with low literacy navigating a government portal — are absent from the model’s training. When the AI generates a persona, it generates a persona from the population it has seen. And the population it has not seen is, frequently, the population that your product will fail.

The sycophancy problem: Why AI tells you what you want to hear

There is a specific and underappreciated failure mode of AI in user research contexts that the design community needs to understand. It is called sycophancy, and it is not a bug. It is a feature — one that is exactly the opposite of what user research is supposed to do.

AI language models are trained through a process called reinforcement learning using human feedback to produce outputs that human evaluators rate positively. Human evaluators, in practice, rate outputs that are agreeable, that confirm their priors, and that present information in a confident and positive framing more highly than outputs that challenge, contradict, or express uncertainty. The result is a model that has been systematically shaped to tell people what they want to hear.

In a user research context, this is catastrophic. The entire purpose of user research is to surface what is actually true about the user’s experience — including, especially, the truths that the product team does not want to hear. The finding that the core feature nobody on the team has been willing to question is actually the primary source of user frustration. The pattern is that the onboarding that everyone is proud of is where the most users give up. The insight is that the user’s mental model of the product is so different from the team’s mental model that the product is solving a problem the user does not recognize as their problem.

Real user research surfaces these findings because real users have no incentive to tell the researcher what they want to hear. They have their own agenda — they want their actual problems solved — and that agenda produces the honest, frustrating, sometimes uncomfortable insights that make good research valuable.

AI-generated user insights have the opposite incentive structure built into them. When a researcher prompts an AI to generate user insights, the AI produces the insights that are most statistically likely to be received positively, which are the insights that confirm what the prompt implies the researcher already believes. The AI that is asked to generate insights about an onboarding flow will generate insights that are predominantly about the onboarding flow, framed in the vocabulary the researcher used in the prompt, and oriented toward the design decisions the prompt implies are already under consideration.

This is not understanding. It is a reflection. And reflection, however sophisticated, is not a substitute for the confrontation with reality that good user research provides.

“Synthetic-user responses for many research activities are too shallow to be useful. Real user research is essential — synthetic profiles cannot replace the depth and empathy gained from studying and speaking with real people.” — Nielsen Norman Group (2024)

The language matters. Here is why

Some readers will be thinking, “This is a semantic argument.” Does it really matter what we call it, as long as we understand the limitations?

It matters more than almost anything else in this series. Here is why.

The language that an organization uses to describe its tools shapes the decisions that the organization makes about when to use those tools and when not to. When AI research synthesis is described as “understanding users,” it is positioned as equivalent to the understanding produced by genuine user research. When it is positioned as equivalent, it is treated as substitutable. When it is treated as substitutable, user research budgets are cut. Researcher headcount is reduced. Design decisions are made on the basis of AI-generated insights that have never been validated against the actual experiences of real people.

This is not a hypothetical sequence. It is happening now, in product organizations across India and globally, driven by a combination of genuine AI enthusiasm, budget pressure, and the seductive plausibility of AI-generated user insights that look like research and require a fraction of the time and cost.

The Harvard Business School AI Institute’s research on AI empathy found something that is directly relevant here. When people were told that a message was AI-generated, they valued it as less empathetic — even when the content was identical to a human-generated message. People instinctively understand that empathy requires a subject who can genuinely care. But in design decision-making contexts — where the AI output arrives as data rather than as a conversational message — this instinctive skepticism is absent. The pattern map looks like findings. The persona looks like a user. The synthesis looks like insight. The form of research is present. The substance of it is not.

The language that calls this empathy removes the skepticism that the form should not provide. And the design decisions that follow from that removed skepticism are decisions made in ignorance of the experience of the actual people the product is supposed to serve.

Where AI is genuinely useful in user understanding, and where it is not

I want to be careful here not to replace one overclaim with another. The argument is not that AI has no role in user understanding. It has a significant role in specific parts of the research process, applied with appropriate skepticism and human oversight.

  • AI is genuinely useful for scale processing. When you have conducted genuine user research — real interviews, real observations, real usability studies — and you have more data than your team can manually process in the time available, AI tools can help you map the dataset at scale. They can identify which themes appear across many participants, which vocabulary is most frequently used to describe a given experience, and which parts of the dataset deserve deeper human attention. This is a legitimate acceleration of research synthesis, as long as the human researcher’s interpretation is applied to the AI’s map rather than replaced by it.
  • AI is genuinely useful for hypothesis generation. Before you go into the field, AI tools can help you identify the hypotheses that are worth testing by processing existing research literature, public discussions, and domain knowledge to surface the questions that have not yet been answered for your specific user group. This is not user understanding. It is desk research at scale. But it is valuable desk research that saves time and broadens the hypothesis space before real research begins.
  • AI is genuinely useful for discussion guide drafting. The first draft of a research discussion guide — the questions and prompts that a researcher will use in a user interview — is a natural fit for AI assistance. The researcher provides the research questions, the user profile, and the design hypotheses, and the AI generates a structured discussion guide that covers the relevant territory. The researcher then applies their craft — reviewing the guide for leading questions, blind spots, sequencing problems, and the specific contextual knowledge that the AI does not have — and produces a final guide that is better than either the AI draft or a draft started from scratch.
  • AI is not useful for replacing real users in research. No matter how sophisticated the synthetic user generation, no AI system can replace the actual human being in an interview, observation, or usability study. The AI does not have the lived experience, the genuine frustrations, the contextually embedded mental model, or the honest agenda of a real user. And the insights that emerge from the confrontation between a designer’s assumptions and a real user’s reality — the findings that surprise, challenge, and occasionally overturn everything the team thought it understood — are not available from any system that generates output from training data rather than from actual human experience.
  • AI is not useful for replacing interpretation. The act of asking what research findings mean — what they reveal about the underlying experience of the user, what design implications they carry, what they suggest about the problem definition the team has been working with is the core of research practice. It is where the researcher’s expertise, contextual knowledge, domain understanding, and design judgment are most fully engaged. Delegating this act to AI is not an efficiency gain. It is the elimination of the most valuable part of the research process.

The specific harms: When empathy language leads to empathy-free design

Let me be concrete about what goes wrong when AI-generated user insights replace genuine user research — because the harms are specific enough to be designed around.

  • Products that optimize for the articulate majority while failing silent minorities. AI training data reflects the users who produce digital content — who write reviews, post in forums, respond to surveys, and participate in the research that gets published. The users who are underrepresented in that data — older users, low-literacy users, users from communities with low digital engagement, and users with disabilities whose experiences are not well-documented — are absent from AI-generated user models. Products designed from those models fail these users systematically, in ways that no amount of AI analysis will surface, because the failure is located in the gap between the data and reality.
  • Features designed for the stated preference rather than the revealed preference. User research produces one of its most valuable findings in the gap between what users say they want and what they actually do. A user who says they want more control over their privacy settings and then, observed in actual use, never opens the privacy settings at all has told you something important about the design of the privacy feature. AI-generated insights, which are produced from stated preferences — from text, surveys, and articulated opinions — systematically miss the revealed preference. Products designed from AI-generated insights are frequently products that users say they would love and then do not use.
  • Design decisions that appear validated but are not. Perhaps the most dangerous harm of AI-generated user insights is the false confidence they produce. A product team that has conducted genuine user research knows, from the experience of that research, the limits of what they learned — the questions that remain open, the user segments that were not well-represented, the findings that felt uncertain. A product team that has conducted AI-generated user research has no equivalent signal about the limits of what they know. The AI output arrives with the same confident presentation regardless of its reliability. The uncertainty is invisible. And invisible uncertainty is the most dangerous kind.

What genuine empathy looks like in practice, and what AI cannot replace

There is a moment that every experienced UX researcher knows. It is the moment in a user interview when the conversation takes a direction you did not anticipate — when the participant says something that does not fit the model you came in with, that challenges an assumption you did not know you were making, that opens a line of inquiry you had not planned for, and that turns out to be the most important thing you learn in the entire study.

This moment is only available to the researcher who is genuinely present — who is listening to the participant rather than executing a script, who has the flexibility to follow the unexpected thread, and who has the professional judgment to recognize that this unexpected moment matters.

It is not available from a synthetic user. It is not available from AI-processed interview transcripts. It is available from the human researcher in the room — or on the screen — with the human participant. And the insights it produces are the ones that most frequently lead to the design realizations that change the direction of a product in ways that genuinely serve the people who use it.

I have been in this practice for 25 years. I have conducted research with farmers in Maharashtra who navigate government agricultural portals on feature phones with screen protectors so scratched that they are nearly opaque. I have sat with elderly users of financial services apps in Pune who understand their financial needs with complete clarity but cannot navigate the interface that is supposed to serve those needs. I have observed users of health information platforms who read the content, understand it, and then make decisions entirely contrary to what the content recommends — not because they did not understand it, but because the content did not understand them.

None of these insights were available from data. They were available from the presence. From attention. From the willingness to be genuinely surprised by what a real human being does and says and feels in the act of trying to use a product that was designed without sufficient understanding of who they actually are.

That is empathy. AI produces something that looks like its output. It does not produce the thing itself. And the design community that confuses the two will produce products that look like they were designed for people while failing the people they were designed for.

Applying LucyUX to the empathy question

The LucyUX framework — Listen, Understand, Conceptualize, Yield — is built on a premise that is directly relevant to this article. Each of its four stages requires a real human being.

  • Listen: This genuinely means being in the presence of a real user in a way that allows you to hear not just what they say but how they say it, what they do not say, and what their silence and hesitation and off-script remarks reveal. It means listening to the specific texture of a person’s experience — the details that do not generalize, that are idiosyncratic to this person in this context, but that illuminate something true about the broader human experience of the problem you are trying to solve. AI processes transcripts of listening. It does not listen.
  • Understand: This genuinely means building a model of the user’s experience that is anchored in the reality of what they told you, what you observed, and what you have learned about the gap between those two things. It means holding the complexity of a real person’s situation — the competing pressures, the contextual constraints, the values that are not captured in behavioral data, and the history that shapes how they approach the problem. AI generates statistically plausible models of user experience. It does not understand.
  • Conceptualize: From genuine understanding means generating design directions that are specifically responsive to the specific reality of the specific users you have researched. Not patterns from a training set. Not the most statistically likely design response to a category of user need. But a specific design decision, made by a specific designer, in response to a specific thing they learned about a specific person. AI can accelerate conceptualization once the understanding exists. It cannot create the understanding that conceptualization should respond to.
  • Yield: Genuine empathy means producing design outcomes that measurably improve the experience of real users whose real experience informed the design. Not the synthetic users who told you what you wanted to hear. The real users who told you what was true — including the uncomfortable, the contradictory, and the surprising.

“Synthetic empathy flips the equation: the system pretends to be empathetic without any genuine internal emotional state or moral agency behind it. Behind the scenes, they are driven by programmed heuristics.” — Medium/Claus Nisslmüller (2025)

Your action this week

Take the last piece of user insight your team used to make a design decision. A persona. A research synthesis. A set of user needs. Ask one question about its source: did a real human being produce this insight through direct contact with real users, or was it generated by an AI tool processing existing data?

If it was generated — not augmented, but generated — identify the design decision it supported. Now ask what you would need to know from real users to validate that decision. What question would you put to a real participant? What would you observe in a real usability session? What would surprise you — and what would that surprise reveal?

You do not have to conduct the research this week. You need to recognize the gap between what you know and what you would know if you had. That recognition is the beginning of the honest design practice that AI cannot replace.

My perspective: What I actually believe

The design community is at risk of making a trade that looks efficient and is actually impoverishing. Trading genuine user understanding — difficult, time-consuming, sometimes uncomfortable, always valuable — for AI-generated user simulation that is fast, scalable, comfortable, and systematically blind to the most important things.

I believe in AI as a tool. I have said it in Part 1 of the series, and I will say it throughout this series. But I believe in it as a tool that accelerates and augments human practice — not as a replacement for the human practice that gives design its value. And the human practice that gives design its deepest value is the practice of genuine empathy. Of sitting with a real person. Of being changed by what you learn about their experience. Of carrying that change into every design decision you make.

AI cannot be changed by a user. It can process a user’s words. It cannot be moved by a user’s experience. It can generate a statistically plausible model of a user’s needs. It cannot feel the particular, specific weight of a specific person’s frustration with a product that was supposed to serve them and does not.

That feeling — the one that makes a good researcher stay awake the night after a difficult research session, replaying what they observed and asking what it means for the design they are responsible for — is not a process inefficiency. It is the source of the insight that changes the design. And no model, however powerful, produces it.

Stop calling it empathy. Start protecting the thing the word actually means.

Up next in the “UX × AI” series: “Your First 30 Days With AI in Your Design Workflow.” We have spent three articles building the right mental model — AI as an intern, prompting as a briefing, and empathy as irreplaceable. Now it is time to get practical. Part 4 is a week-by-week guide to integrating AI into your actual design workflow — not as a wholesale transformation, but as a deliberate, evaluated, practitioner-led adoption that builds genuine fluency without disrupting the practice that already works.

References & further reading