Series continuity: from delegated to opaque
Part 6 looked at what happens when the user isn’t even human. This article turns back to the human who is still there, staring at a decision an AI system just made—a loan denial, a flagged trade, a rejected application—with absolutely no idea why. Delegation was Part 6’s problem. This one is older and more fundamental: even when a person is directly in the loop, most AI systems give them nothing resembling a reason.
The core problem: a black box is a design failure, not a technical inevitability
“Models are opinions embedded in mathematics.” — Cathy O’Neil, author, Weapons of Math Destruction
O’Neil’s point cuts against the most common excuse for opacity in AI products—that the model is simply too complex to explain. Complexity is real. But every model, however complex, encodes choices someone made about what to optimize for, what data to trust, and what trade-offs to accept. Those choices are explainable in principle. Whether a product actually surfaces them to the person affected by them is a decision the design team makes, not a limit imposed by the mathematics itself.
The statistics nobody in the room wants to say out loud
Gartner predicts that by 2028, the growing importance of explainable AI will drive LLM observability investment to 50% of generative AI deployments, up from just 15% today—meaning the large majority of AI systems currently in production still can’t reliably show how they arrived at a given output. That gap is scaling alongside the market itself: the global GenAI models market is expected to exceed $25 billion in 2026 and reach $75 billion by 2029, meaning the volume of unexplainable decisions being made is growing several times faster than the infrastructure built to explain them. Separate research directly tying explainability to customer experience found that a lack of explanation leads directly to user scepticism and disengagement—not eventually, but as an immediate, measurable reaction the first time a system’s reasoning is withheld.
The India-specific reality: two regulators, one demand—show your work
India’s regulators have now converged on explainability from two completely different directions in the same year. Article 4 of this series covered the Reserve Bank of India’s draft model-risk guidance requiring explainable lending decisions; the Securities and Exchange Board of India has since applied an even sharper version of the same principle to algorithmic trading. Under SEBI’s 2026 retail algo-trading framework, effective 1 April 2026, every automated strategy must carry a unique Algo ID, run inside a broker-controlled environment, and include a functioning kill switch—but the most telling provision is this: any “black box” algorithm, where the underlying logic is hidden from the end user, now legally requires the provider to hold a SEBI Research Analyst licence, the same regulatory bar applied to a human giving paid financial advice. The stakes behind that rule are stark. SEBI’s own data shows more than 90% of retail futures-and-options traders lost money, with net losses widening 41% to ₹1.05 lakh crore in FY25—a scale of harm the regulator is now treating an opaque algorithm as no less accountable for than an opaque human advisor would be.
The psychology: the illusion of explanatory depth
Cognitive scientists Leonid Rozenblit and Frank Keil documented a bias they named the illusion of explanatory depth—people consistently overestimate how well they understand a complex mechanism, right up until they’re actually asked to explain it step by step, at which point their confidence collapses. This bias explains a genuine design trap: users routinely report feeling like they understand why an AI system made a given recommendation, based on nothing more than the system’s confident tone and familiar interface, without ever having been given an actual explanation to evaluate. The illusion feels like comprehension. It isn’t. And because it feels like comprehension, users rarely ask the follow-up question a truly opaque system should be prompting—which is exactly why SEBI’s black-box rule doesn’t rely on users to demand transparency themselves. It mandates it regardless.
Where this is already visible
SEBI’s black-box RA licensing requirement is itself a real, live design artifact: any retail trading algorithm sold in India without a transparent decision trail now needs a human-accountable license sitting behind it. The EU AI Act takes a parallel but broader approach, classifying credit-scoring and other high-stakes AI systems as “high-risk” and mandating meaningful transparency about how they decide before they can be deployed at all. FICO, the dominant credit-scoring provider in Western markets, has for years published explicit “reason codes” alongside every credit score specifically to satisfy this exact demand—a direct precedent for what financial-sector explainability actually looks like in a shipped product, not just a regulation. And RBI’s draft principles, requiring a human-understandable rationale behind every consequential lending decision, complete the pattern: two of India’s most powerful financial regulators, working independently, have arrived at nearly identical conclusions about what a trustworthy AI product owes the person it’s making decisions about.
LucyUX applied
- At Listen, the discipline is listening for the specific moment a user says “I don’t know why it did that” about a decision they nonetheless accepted—the illusion of explanatory depth means this complaint is dramatically under-reported relative to how often it’s actually happening.
- At Understand, the job is separating explanations that satisfy a regulator from explanations that actually help a person—SEBI’s Algo ID and RBI’s rationale requirement are compliance floors, not a design ceiling.
- At Conceptualize, explainability has to be built as a first-class interface element—the reason-code pattern FICO pioneered, applied to whatever domain the product operates in—rather than a buried settings-page disclosure written for auditors instead of users.
- At Yield, success is a user who, shown the explanation, could accurately predict what the system would do differently under slightly different circumstances—the actual test for whether an explanation closed the illusion of explanatory depth or merely papered over it.
Consequences
A black-box decision that goes unquestioned isn’t a sign of trust. Per the psychology above, it’s frequently a sign that the illusion of explanatory depth is doing the work real transparency should be doing—right up until the decision affects someone badly enough that they finally ask why, by which point ₹1.05 lakh crore in losses, or a wrongly denied loan, has often already happened. SEBI and RBI have both concluded, independently, that waiting for that moment is too late to start requiring an answer.
Up next in the series: The New Illiteracy. Part 8 looks at what happens as these systems keep getting simpler on the surface while staying just as opaque underneath.
References & further reading
- Weapons of Math Destruction, Cathy O’Neil.
- Could Rising Demand for XAI Improve Trust Issues in CX? (Gartner 2026 forecast), CXM.
- Algorithmic Trading in India 2026: SEBI’s New Framework, Dev Community.
- Is Algo Trading Legal for Retail Investors? New SEBI Framework 2026 Explained, Liquide.
- Latest SEBI Regulations 2026: Key Changes Every Algo Trader Should Know, ShareIndia.
- Illusion of explanatory depth: Overview.
- What Is Explainable AI (XAI)? HPE.
- LucyUX, UXExpert, DesignImpulse.