Series continuity: what the first article left unfinished

Part 1 of the series established that interfaces have stopped waiting—that agentic systems now act on our behalf, and that trust in them is far more fragile than most design teams assume, with real survey data showing a majority of users abandoning a product after a single AI mistake. This article goes one layer deeper into why that trust is so easy to break. It isn’t because AI is wrong more often than we expect. It’s because when it’s wrong, it doesn’t sound wrong. It sounds exactly like it does when it’s right—and design has almost no pattern anywhere for communicating the difference.

The core problem: confidence and correctness are not the same signal

“People conclude too early that they understand it.” — Eliezer Yudkowsky, AI researcher

He was speaking about AI generally, but the line describes precisely what happens inside a single AI response. A large language model doesn’t retrieve verified facts from a database—it predicts the most statistically plausible next word, based on patterns in its training data. When it hits a genuine gap in what it knows, it doesn’t pause or hedge. It fills the gap with something plausible-sounding and delivers it in the same fluent, even tone it uses for things it’s actually certain of. There is no tremor in a fabricated answer. That absence of a tell is the entire design problem this article is about.

The statistics nobody in the room wants to say out loud

MIT researchers found, in a widely cited January 2025 study, that AI models are 34% more likely to use confident language—words like “definitely,” “certainly,” and “without a doubt”—when they are generating incorrect information than when they are stating verified facts. The wronger the model is, the more certain it sounds. The scale of the underlying problem is just as stark: Stanford’s AI Index 2026 tested 26 leading models on a new accuracy benchmark and found hallucination rates ranging from 22% to 94%, and OpenAI’s own GPT-5 system card reports a 47% hallucination rate on the SimpleQA benchmark with web search switched off—nearly half wrong, delivered with total fluency. In academic publishing, a GPTZero scan of ICLR 2026 submissions found more than 50 hallucinated citations across roughly 300 papers, and a separate scan of NeurIPS 2025 accepted papers found over 100 fabricated references among 4,841 papers examined. These aren’t obscure edge cases. They are the baseline behavior of the systems being embedded into products right now.

The India-specific reality: when confident fabrication reaches a courtroom

On 2 July 2026, the Supreme Court of India delivered what may be its most consequential ruling yet on this exact failure mode. In Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd. (2026 INSC 668), a bench of Justices P.S. Narasimha and Alok Aradhe set aside orders of the National Company Law Tribunal and its appellate tribunal after discovering that the tribunal had relied on AI-generated case citations—three of six cited judgments did not exist at all, and the other three contained invented paragraphs. The Court held that citing such material is “misconduct on the part of an advocate” and that a decision resting on even a fragment of hallucinated material is “no decision in the eyes of the law.” It wasn’t an isolated incident: a tax tribunal in Bengaluru had to recall a roughly ₹669-crore order after discovering four non-existent case citations inside it, and the Bombay High Court quashed a nearly ₹28-crore tax assessment that rested on three AI-invented precedents. Every one of these documents would have been read on the page, exactly as authoritative as a real one. That’s the design failure—not that the system was wrong, but that nothing in how the output was presented gave a trained legal professional any reason to doubt it.

The psychology: automation bias and why fluency reads as truth

There’s a well-documented phenomenon in human factors research called automation bias—the tendency for people to over-rely on an automated system’s output, even when contradictory evidence is available, simply because the system has proven reliable often enough in the past. Automation bias is especially dangerous with generative AI because the interface gives no cue at all to distinguish a well-grounded answer from a fabricated one—both arrive in the same font, the same tone, and the same confident sentence structure. Human beings are not built to independently verify every claim a fluent, articulate source makes; we use fluency itself as a proxy for reliability, a heuristic that worked reasonably well for most of human history because confident delivery used to correlate, at least loosely, with actual expertise. AI has severed that correlation completely, and most interfaces have done nothing to replace the cue it used to provide.

Where this is already visible

The case that first made this failure mode a matter of public record was Mata v. Avianca in the United States: a New York attorney submitted a legal brief containing six ChatGPT-generated case citations, all fabricated, all syntactically perfect—correct court names, plausible docket numbers, realistic legal reasoning—and was sanctioned by the presiding judge. Moffatt v. Air Canada (2024 BCCRT 149) showed the same failure mode outside the courtroom entirely: Air Canada’s customer-service chatbot confidently told a grieving customer he could apply for a bereavement fare retroactively; he booked accordingly, and when the airline refused to honor its own chatbot’s promise, a tribunal held Air Canada liable, ruling that a chatbot’s words carry the same weight as any other page on a company’s website. Google’s AI Overviews feature in search has drawn similar scrutiny—one survey found 42.1% of web users had personally experienced inaccurate or misleading content in AI Overviews, delivered with the same confident, synthesized-answer tone Google uses for its verified results. And back in India’s own courts, the Andhra Pradesh High Court had to independently detect and confirm that citations submitted in a petition were AI-hallucinated fakes before it could even rule on the underlying case. Four completely different products, four different countries, and one identical design failure: nothing in the interface distinguished the confident truth from the confident fabrication.

LucyUX applied

  • At Listen, the discipline is listening specifically for the moment a user says, “I assumed it was right because it sounded so sure”—a phrase that shows up constantly in AI product research and is rarely coded as the actual root cause it is
  • At Understand, the work is mapping exactly which categories of query a given system is unreliable on—citations, specific numbers, niche or local facts—because uncertainty is not evenly distributed across a model’s outputs, and users deserve to know where the ice is thin, not just that it exists somewhere
  • At Conceptualize, the design task is building visible calibration into the interface itself—confidence indicators, sourcing, and “verify this” prompts on exactly the categories of claim most likely to be fabricated—rather than a blanket disclaimer that trains users to ignore warnings altogether
  • At Yield, success is a user who has developed accurate, calibrated trust: skeptical exactly where the system is unreliable and confident exactly where it’s earned that confidence, rather than uniformly trusting or uniformly paranoid

Consequences

A wrong answer costs a correction. A confidently wrong answer costs a ₹669-crore order, a sanctioned attorney, a grieving customer’s fare dispute, or a litigant’s case thrown out for relying on a citation that never existed. The damage scales with how convincing the fabrication was, not with how significant the underlying error actually was—which means the products most at risk aren’t the unreliable ones. They’re the fluent ones.

Up next in the series: The Empathy Machine That Feels Nothing. Part 3 turns to simulated warmth: interfaces that sound caring, and what happens to trust when a user can no longer tell care from performance.

References & further reading