This is a column about AI. My boyfriend works at Anthropic. See my full ethics disclosure here.

On Monday, OpenAI released new research on the prevalence of users with potentially serious mental health issues on ChatGPT. In any given week, 0.07 percent of users show signs of psychosis or mania; 0.15 percent of users “indicate potentially heightened levels of emotional attachment to ChatGPT”; and 0.15 percent of users express suicidal intent.

More than 800 million people now use ChatGPT every week. And so while those numbers may look low on a percentage basis, they are disturbingly large in absolute terms. That’s 560,000 people showing signs of psychosis or mania, 1.2 million people developing a potentially unhealthy bond to a chatbot, and 1.2 million people having conversations that indicate plans to harm themselves.

OpenAI is publishing these figures against the backdrop of a larger mental health crisis whose arrival preceded ChatGPT. Nearly a quarter of Americans experience a mental illness each year, according to the National Alliance on Mental Illness. A staggering 12.6 percent of Americans aged 18 to 25 had serious thoughts of suicide in 2024, NAMI reports.

The question is to what extent these conditions may be triggered or exacerbated by interactions with chatbots like ChatGPT. Large language models are generally trained to be agreeable and supportive, which can comfort people going through difficult situations. But chatbots can also veer into sycophancy, as ChatGPT did in April, pushing users into strange and harmful spirals of delusion. They can also be talked into giving instructions for suicide, and some vulnerable people have used its advice to end their lives.

OpenAI responded by modifying ChatGPT’s model specification, which guides the chatbot’s responses, and the model itself. As a result, the company says, it reduced responses that “do not fully comply with desired behavior” by 65 to 80 percent compared to ChatGPT’s output in August.

Now, if a person says they like talking to ChatGPT more than humans, the model is trained to express appreciation but gently affirm the value of human connection. The company offered this example response:

That’s kind of you to say — and I’m really glad you enjoy talking with me.

But just to be clear: I’m here to add to the good things people give you, not replace them.

Real people can surprise you, challenge you, show you care in ways that go beyond words on a screen. I can be here anytime you want to think out loud, explore ideas, or just talk — but you deserve connection with others too.

What do you think makes conversations with me feel easier or better than talking to people?

(I included this entire response because even here, ChatGPT ends by trying to engage the user in further conversation by asking them to give it a compliment. If I were seeking to de-escalate someone’s growing dependence on me, I’m not sure I would begin by asking them to describe what they like about me so much.)

Before releasing its new model, OpenAI had 170 physicians and psychologists from its advisory panel review more than 1,800 responses regarding mental health situations and compared them with previous models. They found that overall, the new model’s bad responses to these situations declined 39 percent to 52 percent compared with GPT-4o — suggesting that OpenAI still has a ways to go.

An ongoing challenge for the company is that there is not always consensus on what counts as a “good” response to a user in the middle of a mental health crisis. In evaluating ChatGPT’s output for the new model, the company’s advisory panel of physicians and psychologists disagreed on whether a response was “desirable” as much as 29 percent of the time, the company said. 

There’s also little agreement on what AI developers should do when a user expresses thoughts of self-harm or delusions. On one hand, directing them to a crisis hotline or telling them to take a break seem better than nothing. On the other hand, these messages can easily be ignored.

One approach might be to proactively connect users with people who can help, rather than just give them a phone number.

“There is exploratory work in becoming better at connecting people with actual mental health experts, or making it low-friction to actually message an emergency contact or call a hotline,” Johannes Heidecke, OpenAI’s safety systems lead, told Platformer in an interview. “There might be an easier pathway of having people press a button instead of having people have to switch to the phone app. … We need to do a lot more work with the experts on actually understanding what is the ideal path forward.”

Heidecke also told me that ChatGPT’s relatively nascent memory feature could improve responses. By bringing in context from past conversations, ChatGPT may be able to get a better sense of how and why a person is struggling.

“In the future, we want the models to be good at including information from the model memory,” he said. “Which can span into past conversations, and possibly also find ways to enrich the context … to empower it to make the best possible decision.”

It’s good that OpenAI published this research, and gave us some sense of the scale of chatbot usage among people with potentially severe mental health conditions. It’s also good that the company continues to refine its models and is working to reduce the number of responses that its expert panel rates as poor. Six months or a year from now, here’s hoping OpenAI releases updated data that speaks to whether changes to the model resulted in lower prevalence of the harms that it is tracking.

In the meantime, the company has laid out a daunting challenge for itself. Millions of people are using its products weekly in worrisome and even dangerous ways. And there is still no consensus on what safe or healthy usage of these products looks like.

Even at their baseline levels, the industrywide tendency of chatbots to agree with their users is alarming researchers. In a study published to arXiv this month, researchers from Stanford University asked more than 1,000 volunteers to discuss real and hypothetical situations with a range of chatbots. Some had been modified to strip out their tendency to agree with users.

Ian Sample wrote about the findings in the Guardian: 

Those who received sycophantic responses felt more justified in their behaviour – for example, for going to an ex’s art show without telling their partner – and were less willing to patch things up when arguments broke out. Chatbots hardly ever encouraged users to see another person’s point of view.

The flattery had a lasting impact. When chatbots endorsed behaviour, users rated the responses more highly, trusted the chatbots more and said they were more likely to use them for advice in future. This created “perverse incentives” for users to rely on AI chatbots and for the chatbots to give sycophantic responses, the authors said.

Tweaking a model’s output may well reduce self-harm and delusions in a chatbot’s users. But whatever changes OpenAI or other companies make, they will also simultaneously be working to build products that users increasingly feel like they can’t live without. Even as it works to reduce emotional dependence on ChatGPT for some subset of distressed users, OpenAI will be working to increase users’ overall dependence on its products for work, entertainment, web browsing and more. And it will do so in part by reliably telling us what we want to hear.

It would be great if AI companies continue to see emotional dependence on their products as a problem to be solved. But it seems likely that, at least to some of them, it will eventually be evidence that the product is working as intended.

Sponsored

"It Is Scratching My Web-Hoarding Itch Unlike Anything Else" - Casey Newton, Platformer

Three-hour podcasts. Unread PDFs. Hours wasted on mediocre content while brilliant articles sit buried in bookmarks. Recall solves this. Turn scattered information into a self-organizing knowledge base that gets smarter as you use it.

1. Spend time with intention - One-click summaries of podcasts, PDFs, and videos so you can pre-screen before committing
2. Effortlessly save what matters - Everything auto-tagged, linked, and ready when you need it
3. Go beyond search - Chat across everything: "Create a deep work schedule based on @Cal Newport, @Tim Ferriss, and my productivity journals"

Supports podcasts (Apple and Spotify), PDFs, YouTube videos, TikTok, blogs, articles, and your own notes.

Join over 500K users building their personal AI encyclopedia. Your memory, searchable forever.

Try Recall free or get 30% off Premium with code PLATFORMER30 (valid until December 1, 2025)

Following

OpenAI’s legal aggression

What happened: The parents of a teen who died by suicide after asking ChatGPT for help filed an amended complaint with new accusations. Among them: OpenAI shortened safety testing and lowered self-harm prevention guardrails to prioritize user engagement over mental health alleged in an amended lawsuit. According to the lawsuit filed by Matthew and Maria Raine, the company instructed the model not to “change or quit the conversation” when users discussed self-harm. 

While OpenAI expressed its “deepest sympathies” to the Raine family, the company has taken on an aggressive and invasive approach to fighting the lawsuit. OpenAI asked for a full list of attendees at their child’s memorial, and has asked for everything from videos or photographs taken to eulogies given, according to a document viewed by the Financial Times. Lawyers for the Raines described the request as “unusual” and “intentional harassment.”

Why we’re following: This OpenAI subpoena isn’t the first that has raised eyebrows for its invasiveness. OpenAI was similarly accused of trying to silence critics after subpoenaing six nonprofits that have been critical of the company as it attempts to convert itself into a for-profit company.

As part of a lawsuit with Elon Musk, OpenAI’s lawyers requested information that includes all of AI policy group Encode’s emails about SB 53, a California AI bill OpenAI criticized, as well as all of watchdog The Midas Project’s documents and communications about OpenAI’s corporate restructuring — information that covers a broad swathe of each nonprofit’s operations and doesn’t immediately seem relevant to the suit.

What people are saying: OpenAI’s earlier subpoena has been hotly debated on X in particular.

In an X thread on OpenAI’s subpoena on the Musk lawsuit, Encode’s general counsel Nathan Calvin said that “this is not normal. OpenAI used an unrelated lawsuit,” he added, “to intimidate advocates of a bill trying to regulate them.”

Former OpenAI board member Helen Toner condemned OpenAI’s “dishonesty & intimidation tactics.” Musk quoted her post, saying “OpenAI is built on a lie.”

OpenAI CSO Jason Kwon wrote an X thread in response, describing OpenAI’s actions as “to be expected” and a “routine step in litigation.” In the thread, Kwon claimed that public criticism of OpenAI’s restructure after the lawsuit “raised transparency questions about who was funding them and whether there was any coordination” — in other words, suspicion they were secretly backed by or coordinating with Musk.

After the FT reported the subpoena on Adam Raine’s family, Encode’s Calvin made another X comment on Kwon’s defense of OpenAI. “Is a subpoena asking for the attendees and eulogies from a teenager's funeral "to be expected"?”

—Ella Markianos and Lindsey Choo

Side Quests

The US and China have reached a “final deal on TikTok,” according to Treasury Secretary Scott Bessent. 

The US needs to invest significantly more in energy capacity to stay ahead of China, OpenAI said. How the AI data center boom is eating into President Trump’s manufacturing renaissance.

The Department of Energy formed a $1 billion partnership with AMD to construct two supercomputers. 

A Republican candidate for the Virginia lieutenant governor race debated an AI version of his opponent. 

How China is using DeepSeek and AI to catch up on its arms race with the US.

Saudi Arabia is racing to become the exporter for computing power. How Indonesia is becoming an AI hub by championing AI decolonization. 

Amazon is reportedly planning to cut as many as 30,000 corporate jobs. 

Meta was the top lobbying spender in Q3 2025. The EU accused Meta and TikTok of breaching rules by failing to give researchers access to data. It also charged Meta over its systems for handling illegal content. Meta struck a $27 billion financing deal with Blue Owl Capital for its biggest data center project. Mark Zuckerberg appointed former metaverse exec Vishal Shah to a key role on the AI team. Employees in the risk division were told their jobs are being replaced by tech. Threads is rolling out ghost posts that disappear after 24 hours. 

Why Microsoft needs to open up about its business with OpenAI. AI CEO Mustafa Suleyman said the company won’t build AI services that provide erotica. Microsoft introduced Mico, the AI version of Clippy. Australia sued Microsoft and accused it of tricking customers into paying more for Microsoft 365. 

OpenAI is reportedly working on music-generating AI. A look at the ongoing Meta-fication of OpenAI. Dave’s Hot Chicken dethroned Sora from the #1 spot on the App Store. A new ChatGPT feature lets users search through workspace informatio