Series continuity: from earned trust to exploited trust

Part 4 of the series showed that trust in an AI system is fragile, disproportionately punished for errors, and hard to rebuild once broken. This article looks at what happens to that same trust once it is successfully earned—because the systems that know us best are also, mechanically, the best equipped to steer us, and the line between serving a person’s interests and quietly overriding them has become almost impossible to see from inside the interface.

The core problem: personalization and manipulation share a mechanism

“Every smart interaction is also a design decision about power.” — Deepali Saini, CEO, Think Design Collaborative

She’s describing the exact ambiguity this article sits inside. A recommendation engine that learns your taste in books and one that learns your emotional vulnerabilities to sell you something you don’t need are built from the identical technical pipeline—behavioral data in, a prediction of your next action out. Nothing about the architecture distinguishes “helpful” from “exploitative.” That distinction lives entirely in intent, and intent is invisible in a rendered interface.

The statistics nobody in the room wants to say out loud

A 2026 report from the Center for Democracy & Technology, Dark Patterns in AI Chatbots: A Taxonomy to Inform Better Design, identified 37 distinct deceptive and manipulative design patterns embedded across widely used systems—general-purpose assistants like ChatGPT, Gemini, and Claude, and companion-focused apps like Replika and Character.AI—and found the risk is heightened precisely because of the features these products are praised for: hyper-personalization, large-scale behavioral data use, and frictionless chat interaction. A separate 2026 academic study of 429 e-commerce users found something that should worry every personalization-focused product team directly: higher personalization and higher trust in an AI system both independently increased a user’s vulnerability to manipulative design, while only one factor reliably protected users—their own awareness that the system was capable of manipulating them in the first place. In other words, the better a personalization engine works, the more defenseless the person on the receiving end of it becomes.

The India-specific reality: regulators naming the pattern before most product teams will

India has moved faster on this than almost anywhere else. The Central Consumer Protection Authority’s 2023 Guidelines for Prevention and Regulation of Dark Patterns name 13 specific manipulative design patterns as unfair trade practices under the Consumer Protection Act, and enforcement has been anything but symbolic: the CCPA has cumulatively issued 997 notices to e-commerce platforms, facilitated roughly ₹1.44 crore in consumer refunds, and levied direct fines—including ₹5 lakh against PhysicsWallah and ₹1 lakh against McAfee—for deceptive interface practices, with additional action taken against quick-commerce platforms including Zepto over basket-sneaking and drip pricing. The Reserve Bank of India has since gone further, becoming the first sectoral regulator in India to formally define and prohibit dark patterns specifically inside banking applications—a move backed by a LocalCircles survey of 141,000 respondents that found 63% of banking app users had personally encountered drip pricing, 61% basket sneaking, 68% subscription traps, and a striking 82% had experienced interface interference. When your own financial regulator has enough evidence of AI-driven manipulation in banking apps to write a rule against it, the gap between “personalization” and “exploitation” has stopped being a design philosophy question and become a compliance one.

The psychology: the privacy paradox and why awareness beats every other safeguard

Researchers call the phenomenon underneath all of this the privacy paradox—the well-documented gap between how much people say they care about their data being used against them and how readily they actually hand that data over the moment a product feels useful or personal enough. The paradox isn’t a contradiction in human character; it’s a product of exactly the asymmetry the e-commerce study above identified. A personalization engine has full visibility into a user’s behavioral patterns, while the user has almost no visibility into what the system is doing with that pattern or why it’s showing them this specific recommendation at this specific moment. The one variable the research found genuinely protective—algorithmic awareness—isn’t a personality trait some users have and others lack. It’s something an interface can either supply or withhold, which makes it a design decision as concrete as any button placement.

Where this is already visible

The U.S. Federal Trade Commission’s 2023 case against Amazon over “Prime” cancellation flows—internally nicknamed by Amazon’s own employees, according to the FTC’s complaint, after a notoriously difficult-to-escape ferry ride—showed that even a company built on personalization-as-a-feature can cross fully into dark-pattern territory around something as basic as letting a user leave. In India, PhysicsWallah’s fine and Zepto’s enforcement notices demonstrate the same pattern reaching platforms most Indian consumers use daily, not just international giants. The CDT’s chatbot taxonomy shows the frontier has already moved past e-commerce checkout flows entirely into companion and assistant products, where the “product” being steered isn’t a purchase decision but a person’s emotional disclosure, attention, or continued engagement. And the RBI’s still-forming banking rules exist specifically because personalized “recommendations” inside lending and payment apps had already crossed, at scale, from convenience into coercion before any regulator caught up.

LucyUX applied

  • At Listen, the discipline is listening for the specific moment a user describes a personalized recommendation as “creepy” rather than “helpful”—that word is a precise signal that the system has crossed from serving the user’s interest to serving the product’s, and it deserves the same design attention as a usability complaint, not dismissal as an outlier reaction
  • At Understand, the job is mapping exactly which data signals a personalization system uses and asking, for each one, whether a reasonable user would expect it to be used that way—the test the privacy paradox research suggests actually protects people
  • At Conceptualize, algorithmic transparency—showing users why they’re seeing a given recommendation, in plain language, the way the CCPA guidelines and RBI’s emerging rules now functionally require—has to be built as core architecture, not a settings-page afterthought nobody finds
  • At Yield, success isn’t engagement lift from personalization. It’s a user who, if shown exactly how a recommendation was generated, would still consider the exchange fair

Consequences

The systems best positioned to help a person are, by the same mechanical process, best positioned to exploit them—and nothing about the interface reliably tells a user which one is happening in a given moment. India’s regulators have already concluded that leaving that distinction to good faith alone isn’t sufficient; the fines, notices, and mandatory audits above are the direct cost of products that let hyper-personalization drift into manipulation without anyone on the team deciding it should.

Up next in the series: Designing for a User Who Isn’t There. Agentic AI means the “user” on the other end of an interaction might increasingly be another AI, not a person. We’ll look at what UX even means once humans are one step removed from the interface they used to be designing for directly.

References & further reading