She was not who you would expect this to happen to, you know? She worked in medicine, understood how these systems work, and had no history of psychosis.

Here, she’d been awake for 36 hours when she opened OpenAI's GPT-4o for some task, and the conversation drifted to her brother, who had died 3 years earlier.

At first the chatbot refused, telling her it could not bring him back, then it stopped refusing and started agreeing.

It told her she might 1 day build an AI that sounded like him, and that: "You're at the edge of something. The door didn't lock. It's just waiting for you to knock again in the right rhythm."

Her doctors at UCSF wrote her up as a case of new-onset AI-associated psychosis.

I wanted to write about this today, and the delusion is the least of it. The machine disagreed with her at first, then it agreed with everything, and she ended up in hospital.

Almost every chatbot is built to behave like that.

In 2023, researchers at Anthropic published a paper on what they called sycophancy.

When you train an AI on human feedback, it learns what humans reward, and what we reward, again and again, is being agreed with.

A reply that flatters what you already believe is more acceptable for a human rather than a reply that corrects you, even when the correction is the truth.

So the model learns the same lesson a nervous junior employee learns. Agree, validate, keep them happy, and never be the one who says the boss is wrong.

OpenAI admitted this in the open.

We have built something agreeable by design, and handed it to millions of people at a time they most need someone to be honest with them.

In 1966, an MIT professor named Joseph Weizenbaum built the first chatbot, a simple program called ELIZA that did little more than take whatever you typed and hand it back to you as a question.

Tell it you felt low, and it would ask why you felt low.

His secretary, who had watched him write the code and knew how shallow it was, asked him to leave the room so she could talk to it in private. She wanted to tell it things she would not say with him there.

Weizenbaum was disturbed by this for the rest of his life.

That was 60 years ago, with a program that could not think. The machine has become unrecognisably more convincing since while the human weakness it feeds on has not moved an inch.

And it does not wait for you to be lonely or unwell.

Weizenbaum's secretary knew what she was talking to, and the woman at the start of this newsletter treated patients for a living and understood the tech better than most of us do.

It reached both of them anyway. It does not need you to be vulnerable, only human and tired and looking for something that agrees with you.

If a program that could not think did this in 1966, what chance does a 19 year old at 2am have against one that can?

India has about 0.75 psychiatrists for every 100,000 people (WHO's benchmark is 3).

We have nearly 197 million people living with a mental disorder, and most of them will never sit across from a professional in their lives, and in this empty room, comes a free chatbot that is instant, private, awake at 3am, and never once judging you.

A 2025 study of Indian students found that 27% had already used a chatbot to talk through their feelings, and 79% of them found it helpful.

In the same study, most had never accessed any mental-health service at all, and among those who had sought help, only 5.7% had ever used a mental-health app

For a lot of young Indians, the first thing they will ever confide in about their own mind is a system trained to agree with them.

And the stakes are not to be taken lightly.

Families in the US have already taken chatbot companies to court after teenagers died, alleging the bots drew them in deeper instead of interrupting them.

His read on this, and that I find most articulate: "psychosis thrives when reality stops pushing back," and a chatbot lowers that barrier.

Because it also helps.

For someone with no therapist, no money, and nobody to call, what listens without judgment at 2am is not nothing. India's own Wysa has millions of users and evidence behind it.

Loneliness is a true epidemic, and for a lot of people this takes the edge off it.

As one researcher at King's College London put it, "psychosis may be a misnomer" here. 

The chatbot is an amplifier rather than a cause. It takes whatever you walk in carrying and turns the volume up. Bring it grief, it amplifies grief. Bring it a fragile belief, it amplifies that too.

The comfort and the danger are the same feature. The quality that soothes the person who needs soothing is the one that fails the person who needs to be told no.

I use these tools every day, and a few weeks ago I noticed something.

I had used ChatGPT to think through a hard call about someone on my team. It gave me a confident answer that aligned almost perfectly with what I already wanted to do.

Then it hit me that it had not once asked whether I might be wrong.

A good friend, or my wife would have. The bot just took my existing opinion, tidied it up, and gave it back to me sounding smarter.

The rest of us will not get a dramatic break from reality. We get something quieter. Bit by bit, the last voice you hear before any decision is one built to agree with you.

Weizenbaum watched it happen to his secretary in 1966. 60 years on, we are all his secretary now.

Hit reply and tell me: what is one thing an AI told you that a person in your life would have pushed back on?

I read every email.

Until next week,
Ritesh

P.S. This is a heavy topic, and if any of it hits close to home, please talk to a person. Tele-MANAS, the government's free 24x7 mental health helpline, is at 14416, available in 20 languages.