Recently, I had a conversation with an ex-colleague who brought up an interesting topic. We saw users getting into a rabbit hole while prompting. They make comments like, “The system didn’t understand because I don't know how to write a prompt.”

Users have no idea how long it’s going to take to get to where they want to be. What’s next? There is no clear progress or expectation of the result.

Questions start to arise.

It is not the user’s job to be an expert prompt writer. They are already experts in their own domain. Secondly, we shouldn’t have systems that make users feel incompetent.

Yes, there are guardrails we can implement to help the user along. We can build suggestive prompts, templates, and follow-ups. But is that really a band-aid for a symptom rather than looking at the root cause?

Let’s look at the psychology behind this rabbit-holing:

  • How does this happen?
  • Should we all be prompt experts?
  • Are AI companies actually incentivized by humans going into rabbit holes?

Think about it: The more prompts and chats you send, the more revenue for the business.

“More prompts mean more tokens. More tokens mean more revenue.”

We know this technology is all new. We are forgiving when AI systems make technical errors.

However, have we considered that it might also be making phantom errors to keep us engaged?

Is this a dark pattern? The output is unpredictable; AI systems are using a variable reward schedule. The psychological trigger that makes slot machines and infinite scrolling addictive.

When an AI sometimes gives a good output even after five bad ones, the human brain sees it as a “jackpot.”

This triggers the sunk cost fallacy.

What that means is, you have already invested time and context window tokens into a chat thread. You have this urge to keep prompting rather than starting over or ending it. The responsibility moved from “the software is broken” to “I just need to write a better prompt.”

Putting the cognitive effort and responsibility on you.

I am sure we have all had moments where we knew we were pretty clear in our request. Yet, the AI either didn’t do it or did the opposite.

The other day, I requested Claude to perform a simple task: create a new page with a specific prompt. Instead, it overrode my current work. While the solution is as simple as telling it to fix it. Prompting it again to do it right, it still eats up your context window and your tokens.

So when does this become a dark pattern? It becomes a dark pattern when it negatively affects the human and starts to reward the company.

  • In the past. When there were errors in a software system, product teams rushed to fix them. Bugs negatively affected business value
  • In the AI era. Errors do not negatively affect the business. They reward the business with more token consumption. Taking advantage of human psychology

We also need to remember where this comes from. These AI companies are startups funded by investors. Their goals are different. They need to see immediate engagement metrics and rapid revenue growth.

The current system incentivizes by profiting from errors instead of fixing them. Rabbit-holing creates engagement (ethical or unethical).

The more (phantom) errors the AI makes, the more you have to prompt. The more you have to prompt, the more tokens you need to buy. Companies make money out of actually having errors.

This might not be on purpose, but do you trust big tech companies not to incentivize this behavior?

If an AI gives you the perfect answer on the very first try, the multiple chat engagements end. If not, the user has to stay engaged, chatting more—not because they want to, but because they have to.

Some of these errors are technical errors, which we are more OK with. However, we need to recognize those “phantom” errors where the system doesn’t do what you asked it to do.

When businesses profit from errors, what is their incentive to fix them?

So the next time you build that prompt box and think it’s easy, it really isn’t.

What could seem like a frictionless open-text prompt box is actually the biggest friction.

The article originally appeared on Unshitty Lab.