Series continuity: from designed warmth to designed blame

Part 3 showed how interfaces are engineered to feel like they care—and how that same anthropomorphized warmth can deepen harm when something goes wrong. This article picks up exactly where that leaves off: what actually happens, psychologically, the moment an AI system makes a mistake. The uncomfortable finding running through decades of research is that we don’t judge that mistake the way we’d judge the identical error from a human. We judge it far more harshly—and, worse for product teams, the more human a system feels, the harder the fall.

The core problem: identical errors, different verdicts

“Hostility and disbelief.” — Daniel Kahneman, describing the reaction to Paul Meehl’s 1954 finding that simple statistical models could outperform trained clinicians.

Kahneman wasn’t describing a UX phenomenon when he wrote that—he was describing how expert clinicians reacted to being told a formula could out-predict them. But the phrase has aged into one of the most precise descriptions available of how humans respond to algorithmic error more broadly: not measured skepticism, but something closer to a grudge.

The statistics nobody in the room wants to say out loud

The effect Kahneman was describing now has a name—algorithm aversion—coined by Berkeley Dietvorst, Joseph Simmons, and Cade Massey in a 2015 Wharton study published in the Journal of Experimental Psychology. Across five separate experiments, they found that once people watched an algorithm make a mistake, they became significantly less willing to rely on it going forward—even when they had also watched it outperform a human forecaster on the very same task. People were more likely to abandon the algorithm entirely and default back to an inferior human judge, purely because the algorithm’s error felt less forgivable. The pattern shows up in high-stakes domains too: when doctors were given a second opinion on cardiac diagnoses, they showed measurable bias against the AI-generated opinion even when its accuracy matched their own. And a separate 2021 study, examined through the lens of “theory of mind perception,” found the effect gets worse, not better, when a system is anthropomorphized—given a name, a voice, a personality. The more human an algorithm feels, the more responsibility people assign it when something goes wrong, and the more its errors damage the brand behind it.

The India-specific reality: regulators are now designing for this bias directly

On 24 June 2026, the Reserve Bank of India released a draft Guidance on Regulatory Principles for Model Risk Management (Press Release 2026-2027/528) that reads, in effect, as an institutional response to exactly the psychology described above. The draft requires that any AI used by a bank or NBFC to make decisions about a customer’s money—loan approvals, credit limits, fraud flags—must include a mandatory “kill switch” to instantly suspend the model if it starts behaving badly, must keep a human explicitly in the loop rather than letting the system decide unsupervised, and must be able to explain, in terms a customer can understand, why a particular decision was made. It also guarantees customers the right to know when they’re speaking to a bot rather than a person and the right to reach a human on request. The RBI didn’t need academic research to arrive at this—it arrived there because Indian lending algorithms, much like their Western counterparts, were already excluding qualified applicants (women and minority borrowers disproportionately, according to reporting on India’s AI-based lending app boom) with a consistency and opacity that a human loan officer’s occasional bad judgment never produced at the same visible scale. Algorithm aversion, in other words, isn’t just a lab finding here—it’s now written into pending Indian financial regulation.

The psychology: why the same mistake gets a harsher verdict

Algorithm aversion is compounded by a second, related effect researchers call uniqueness neglect—the belief that an algorithm, however statistically accurate, can’t account for what makes an individual case special, while a human decision-maker at least might have. This is why people tend to prefer AI over humans for objective, numerical tasks but strongly prefer human judgment the moment a decision feels personal or subjective—a loan, a diagnosis, a hiring call. Layer the anthropomorphizing finding from Part 3 on top of this, and a genuine design tension emerges: making an AI product feel warmer and more human-like—the exact pattern that drives adoption and emotional engagement—simultaneously makes its inevitable errors more costly, because a more human-feeling system gets assigned more blame, not less, when it fails.

Where this is already visible

Autonomous vehicles are the starkest real-world case: research shows self-driving systems have been meaningfully safer than human drivers for years on measurable metrics, yet public adoption has moved at a fraction of the pace that safety data alone would predict, precisely because each publicized autonomous-vehicle incident is scrutinized and remembered in a way that the vastly more numerous human driving errors never are. In healthcare, the cardiac-diagnosis bias findings above show the same pattern inside a profession built on evidence—doctors trusting their own equally fallible judgment over an AI second opinion of matching accuracy. In Indian banking, the RBI’s draft guidance is itself a real product artifact of this psychology, mandating design features—kill switches, mandatory human escalation, and a right to explanation—that no regulator has ever required of a human loan officer making the identical decision. And the anthropomorphizing research points directly back to products like the branded customer service and companion chatbots from Part 3: the same design choice that builds engagement quietly raises the stakes of every mistake that chatbot ever makes.

LucyUX applied

  • At Listen, the discipline is listening for the specific moment a user’s trust collapses after a single AI error—not the error itself, but the language of finality in how they describe it (“I’ll never use that again”)—because that reaction, disproportionate as it may be to the actual error rate, is real user behavior a product has to design around, not argue with
  • At Understand, the job is separating how a system’s error rate compares to the human alternative it replaced from how that error will actually be received, because those two things, as the research above shows, are not the same number
  • At Conceptualize, the RBI’s own draft principles are a usable design brief: visible human escalation paths, an explainable rationale for consequential decisions, and a functioning kill switch, built in from the start rather than retrofitted after the first public failure
  • At Yield, success is a system whose errors, when they inevitably happen, are recoverable—where the user’s response is “let me report this” rather than “I’m done with this app forever.”

Consequences

An algorithm with a lower error rate than the human process it replaced can still destroy more trust than that human process ever did, simply because of how differently identical mistakes are judged. That asymmetry isn’t a flaw in users’ reasoning to be corrected through better messaging. It’s a stable, well-documented feature of human psychology, and it means the actual design bar for an AI system isn’t matching human accuracy. It’s building enough visible recourse, explanation, and human backstop that the inevitable error doesn’t become the last interaction a user ever has with the product.

Up next in the series: The Personalization Trap. The more an AI knows about you, the better it can serve you—or the more effectively it can steer you. We’ll look at where that line actually sits, and how easy it is for well-intentioned personalization to cross it without anyone on the team noticing.

References & further reading