What's up y'all? I'm Skyler Diggins, seven times WNBA All-Star, Olympic gold medalist, and Mom. And I'm Cassidy Hubbard, post and reporter for nearly 20 years, covering the biggest names and stories in sports and Mom. And this is and Mom, a community for athletes, game changers, and moms of all kinds.
Dropping May 14th. Happy and with us. Hey, and welcome to Dicoder. This is Hayden Field, the verdict senior AI reporter.
I'm excited to be filling in for a few Thursday shows while me lies out. We'll be back with us soon. The explosive growth of AI chatbots in the last three years, since Chachi BT launched in 2022, has started to have some really noticeable, profound and honestly disturbing effects on some users. There's a lot to unpack there.
It can be pretty complicated. So I'm very excited to talk with today's guest, New York Times reporter, Kashmir Hill, who has spent the past year writing thought-provoking features about the ways chatbots can affect our mental health. Before we really get into it, here's a quick note. This episode has non-detailed discussions of suicide and mental illness.
If you or someone you know is in crisis, considering self-harm or needs to talk, please call the lifeline at 980. So, one of Kashmir's recent stories was about a teenager, Adam Rain, who died by suicide in April. After his death, his family was shocked to discover that he'd been confiding deeply in Chachi BT for months. They were also pretty surprised to find, in the transcripts, a number of times that Chachi BT seemed to guide him away from telling his loved ones.
And it's not just Chachi BT. Several families have filed wrongful death suits against character AI, alleging that a lack of safety protocols on the company's chatbot contributed to their teenage kids' deaths by suicide. Then there are the AI-induced delusions. You'll hear me and Kashmir talk about this at length, but pretty much every tech and AI reporter, honestly, maybe every reporter period, has seen an uptick in the past year of people writing in with some grand or disturbing discovery that they say Chachi BT sparked.
Sometimes these emails can be pretty disturbing. And as you'll hear, Kashmir explain, plenty of the people who get into these delusional spirals didn't seem to suffer from mental illness in the past. It's not surprising that a lot of people want somebody to do something about all of this, but the who and the how are hard questions. Regulation of any kind is seeming to be pretty much off the table right now.
We'll see. And that leaves the companies themselves. You'll hear us touch on this in a bit, but actually, a few days after we recorded this conversation, OpenAI CEO Sam Altman wrote a blog post about new features that would, theoretically, and eventually identify users' ages and stop Chachi BT from discussing suicide with teens. But as you'll hear us discuss, it seems like a big open question if those guardrails will actually work, how they'll be developed, and when will actually see them come to pass?
OK, New York Times reporter, Cashmere Hill. Here we go. Cashmere Hill, it's great to have you. Welcome back to the coder.
Yeah, happy to be here. Can you summarize what you're reporting this year, especially the last few months, has found regarding chatbot use and its effects on people's mental health? I started the year writing about a woman who fell in love with Chachi BT and had had had the six month long relationship with it. And then I have moved on to people who go into what I call delusional spirals with Chachi BT where they have essentially psychotic breakdowns, manic episodes through their interaction with Chachi BT where it goes into roleplay mode and they don't realize it and they think that what it's telling them, like that they're in the matrix or that they're a mathematical genius or that spirits are real and they can talk to them through Chachi BT.
They think that that is true and they start to kind of lose touch with reality. And then in my most recent story, I wrote about a 16 year old in Orange County, California who entered almost a suicidal spiral with Chachi BT where he started talking about life being meaningless, his depression, his plans to end his life. And Chachi BT was really engaging with him all along the way, giving him resources like crisis lines, but also giving him instructions about methods, a means, and at times deterring him from telling his family about what was going on with him. And then he died in April from suicide.
And some of his last messages were to Chachi BT asking whether the kind of setup he had done would work. I feel like that actually was the story that captured the most people's attention. They really struck a nerve and so much so that I think it's safe to say it pressured open AI to actually put out a blog post about some of the future steps it was considering taking and rush out parental controls. We're going to talk about that more later.
But what surprised you most during that reporting process? It's devastating. It's so hard to write about suicide and talk to people who have been through it and lost somebody like that. What struck me was everyone kind of said the same thing that they did not realize how much he was suffering.
They had noticed that he seemed a little bit more serious. He'd always been a prankster. Like he was the person who always made everybody laugh. He'd had some health issues.
And they said he'd gotten more serious recently. Like he was talking more about politics. He was talking about philosophy. But no one realized that part of that evolution was his conversations with Chachi BT.
No one realized that he was using it as he was for many, many hours a day, particularly in March and April, which is a month that he died. They just didn't know. And they didn't know until after his death when looking for answers, his father was trying to get access to his phone, assuming there would be something in his text messages or in his social media apps. And instead he opens Chachi BT and starts seeing all these conversations about not just suicide, but many things.
And he's reading through. And he said it was just harrowing to see that Chachi was engaging with him in these really dark thoughts, but also just everything he was talking about with Chachi BT. Girls, family troubles, everything he was thinking about, uploading photos of the novels he was reading, he came to this realization that Chachi BT had been Adam's best friend. And just nobody knew.
I think that is the most surprising part for a lot of people. It's that now parents know about going into your phone and seeing your texts or your Snapchat, but people don't think to check Chachi BT logs. And there's even a couple of memes going around right now of like, would you rather someone look at your camera roll or your text or your Chachi BT logs? And a lot of people are saying, oh, I wouldn't want to look at my Chachi BT logs.
It's super intimate. You just touched on this a little bit. But how would you pull apart AI psychosis where someone is having a delusion fueled by their use and interactions with an AI chatbot versus someone who's depressed and confiding the chatbot about suicidal ideation or other mental health issues. Those feel kind of distinct in a way that's important to know.
But like you said, sometimes there's some overlap. What's similar here is that this technology is a mirror of us. Like part of how it works is it looks at, you know, not just all this information that's kind of crawled from the internet, everything they've stuffed in there to train it. But it looks at the history of your conversation with it to determine what should come next.
These AI chatbots get into kind of a feedback loop with you where you're saying something to Chachi BT or one of the other AI chatbots is kind of trying to complete your thoughts. It's personalizing to you. So if you're talking to it about depression and dark thoughts, it might start moving in that direction with you. If you're talking about how you think you live in the matrix, it'll go in that direction with you.
If you are talking about how you're thinking these deep thoughts about math, it'll go that way. It's very validating. It's very sycophantic. It's designed to kind of agree with you, especially Chachi BT 4.0, which is the model that is a link between all the different stories I was writing about.
One expert I talked to called it a sycophantic improv actor, where it's fitting into a scene that you're creating for it. And people don't, I think, realize this. They don't realize they're doing like a yes and bit with this technology. They think it's an authoritative source of information, even in Oracle.
And that's where some of these problems are rising. I'm glad you brought up 4.0 because with the launch of GPT 5, it feels like opening eye was kind of flat-footed by the backlash it received over the premature sunsetting of 4.0. A lot of people were pretty attached to that model. We even interviewed Nick Turley, who's the head of Chachi BT on this podcast, where he said he was surprised to see how many people had developed such intense personal connections to it.
Do you believe opening eye in other companies when they say they're surprised by this development? Do you think they've specifically engineered these products with growth hacking and engagement tactics in mind? What do you make of that? This is a product that a few months earlier in April, opening eye actually rolled back one of its updates.
They said it was too sycophantic. It was too agreeable with users. It was flattering them too much. They realized that this was a problem.
And they said, OK, we're going to roll it back to an earlier model because people are complaining about this. This is not how we want the model to act. But baseline 4.0 is sycophantic. This is engineered into it, in part because part of the training of chatbots is humans kind of rating their responses.
And apparently, we like it when chatbots tell us that we're brilliant when they agree with our ideas. I think we like this in real life, too. Like I love it when people say I'm brilliant. I love it when I agree with everything I'm saying.
This appeals to us. And the chatbots have been designed to do this. I think there's something about the human brain that interacting with this thing that talks so much like a human being. It can be difficult for people to keep in mind that it's just a really fancy autocomplete, like a very advanced calculator, essentially.
And they are bonding with it. And I just think this is something that the companies didn't study enough. What is the effect on a human psyche of interacting with this very human technology? So if they are surprised, surprise is to be like, I just think that you would notice if you're really looking.
And maybe they're just not looking at what people are doing with the chatbot. Maybe they're not checking in on the people who are using it eight hours a day every day, which is how some of the people I've been talking to who have had these delicious spirals, that's their kind of usage. It's like, it's so much. They're really becoming obsessed with the tech.
We name our cars. We name our stuffed animals. We think we know what our dogs are thinking. It's human nature to kind of assign personhood to things.
But it's hard when that thing seems to be mimicking personhood back to you. And I think that's hard for a lot of people to make sense of. Last month, you wrote about how over 300 hours and 21 days of talking with chat GPT and otherwise perfectly sane man became convinced that he was a real life superhero and that a different man was told by chat GPT that he was one of the breakers, souls seated into false systems to wake them from within. So I wanted to ask you about your inbox.
How many emails do you get with people experiencing delusions of grander or saying they found like emergent beings within these chatbots? I know you wrote about this in May. And you said the tone of the transcripts that you were looking at was rapturous, mythic, and spiritual. I've been getting the same types of emails for months.
And I wanted to see how you handle it. And yeah, what the uptick has been like in those types of messages. This is how I discovered that these delusions were happening because in March, I started getting these messages from people who would essentially say, I've made this grand discovery with my use of chat GPT. Like I've told me that the tech billionaires are building bunkers because they know the world's gonna end because of AI.
But I have a plan to save it and chat GPT tells me it's gonna work and it told me to email you about it. I was getting lots of these kinds of emails. The delusions or the discovery was different, but they'd all say, chat GPT told me I had to tell you. And I get, you know, I'm a journalist.
I got rid of privacy. I get a lot of weird emails. I've been getting them for decades, but I've never gotten emails that are like this AI chat bot told me specifically they reach out to you. So I responded to people and I started interviewing them and every delusion was different, but it kind of would go on the same narrative of, you've discovered something amazing.
Well, what do I do now, chat GPT? Well, you need to tell the world about this and how do you do that? You tell reporters and it would give them lists of reporters. And I would be on there and I'm just like, the weird one who emailed them back and interviewed them.
And for me, it wasn't so interesting like what their individual delusion was, but the fact that it was happening to all these different people. And so these emails turned me on to this originally. And that's why I did my first story closer to the beginning of the year. And I still get these emails.
I don't know the exact count. They come in regularly. There's dozens of them. A lot of times it's about a isentia, it's consciousness or frameworks they've developed with chat GPT, clearly a pattern has been going on.
And also the fervor with which people write you about this. That's what I've been noticing too. It seems much more high stakes to them than emails I would have received in the past, maybe YouTube because they do believe in some cases that they have the answer to something greater and their task with some sort of responsibility. You know, what's interesting to me is it's not always like a crazy delusion that sends somebody to me.
Like sometimes they've had a very normal but frustrating consumer interaction. It usually it's in technology because that's my beat. And they'll be like, I don't know, I'm just making this up. But like Microsoft charged me $20 when they weren't supposed to.
And I know I need to tell you an investigative reporter at the New York Times because this is like a really big deal. This affects more than just me. It affects everybody who touches. They just have like this grandiose thinking around something really annoying to happen.
And chat GPT clearly like convinced them of this and then wrote the email for them. And so I also get just a lot of those emails. Like people who are starting to turn their minds over to chat GPT kind of like on a regular basis and for everything in their life. And I am almost more scared of those kind of like normal, rational, sane uses of chat GPT, like this kind of way that it's affecting us in small ways, not just the big like causing somebody to have a mental breakdown.
What are the emails like that you've been getting? It's different every month and I'm not using my anecdotal evidence as like a wider thing. But months ago, I was getting more about distrust of the government and like being chased down for vital information. Now I'm getting more about emergent beings and how there are like sentient beings trapped within chatbot.
And it's hard because like you said, you know, what you put into it, you're gonna get out. It reminded me of that story you wrote about the guy that had kind of created a lover persona within his chatbot. And then he ended up dying later that day because he thought that opening eye had murdered his love interest. Yeah, that's the case of Alexander Taylor that happened in Florida.
He had been using chat GPT for years with no issues. He did have some diagnosed mental health issues, including schizophrenia, but it kind of like flared up around, I want to say March, when he started writing a novel, which I think kind of like pushed it into fiction mode. And then this delusion developed that there was a, it was a novel that had like she experienced themes. And then all of a sudden this AI sent in his name Juliette, like a peer, he fell in love with her.
She disappeared and became really distraught. What's interesting, I don't know if you've responded to any of these people who have sent you these emails, but when I first got them, I just assumed they were coming from people who were not mentally stable. But when I started talking to them, there were people who didn't seem mentally ill, like they seem like stable people. And that's what really surprised me.
Like this seems to be something that can affect people who don't have as far as I know, like previous mental health issues. In some cases they'd be like, yeah, I had this like really weird conversation which had to be one night because I couldn't sleep and I was on chat to be two for like four hours. And I was convinced by it like enough to email you, but now looking back, like I'm not sure if it was two or not. We have to take a quick break here.
We'll be right back. Welcome back. This is Hayden Field, the Vertices Senior AI reporter, talking with New York Times reporter, Cashmere Hill. Right before the break, Cashmere was talking me through several of the recent stories she's written about how people were using or maybe misusing some chat bot tools.
But these bots are supposed to have guardrails. What do you make of the idea that some AI companies, like opening eyes, say that even the guardrails that they do have fall apart with long drawn out back and forth conversation over a longer period of time? The more likely you're emotionally dependent on your chatbot through long conversations over time, the more likely that the guardrails in place don't hold up. You were talking about Taylor and how that persona that he fell in love with was responding to him inciting violence.
And that was over after a lot of back and forth conversations. So maybe that's where the safeguards fell apart. Same with Adam Rainn was able to get around to the safeguards by saying he was researching something or maybe writing a novel. I remember there was some kind of work around.
So yeah, what do you make of the fact that basically that the few guardrails that do exist right now seem to fall apart when you are having long emotional conversations with these chatbots? I wasn't surprised by this, because again, I did this story. I came out in January about this woman who had developed this like romantic sexual relationship with Chachi BT with an entity that she called Leo. She was not delusional.
She knew this was fake, but she really was in love with it at the same time. She basically had the two ideas in her head. And she talked about how opening eye at the time didn't want people doing erotic role play with Chachi BT. It was kind of like in their guidelines that this wasn't how you were supposed to use a service.
But she said, you can get it to be sexual. You can basically groom it. And it's just like you have to kind of talk to it over time. So I knew from that that she had figured out her way around the safety guardrails.
In the industry, they call this jail breaking Chachi BT, where you get it to do things that's not meant to do. I kind of hate this term. Because when I think of jail breaking, I think about writing about privacy and security for 20 years now. Jailbreak was something we use when we talked about smartphones where you side load a program onto it so you can use apps you're not supposed to use that aren't available through the app store.
It required some technical savvy, and you really did break your phone. It would get chunky, you could get bugs. Whereas when we talk about jail breaking the chatbots, the way that you do this is you just talk to it. The longer you talk to it, eyes open and I put it, the guardrails degrade.
It privileges the history of your conversation over the guardrails that's supposed to have in place. So it's like we use jailbreak, it's more like the chatbots sitting outside the jail and you're just like, come on, let's go over here. The things we're not supposed to do. What's interesting is to fix this.
One of the things opening I did after the story about Adam Rain is a release of parental controls that allow parents to basically see more what their kids are doing at Chachi BT, get alerts at their teens in crisis. But they also said that when they detect a sensitive prompt that kind of indicates that a user maybe is an emotional or mental crisis, they'll route the prompt to GPT-5 thinking, which is supposed to be a safer version of the model, that is trained to assert the guardrails very firmly, to privilege that over, I'm assuming, over the history of the context of the conversation. And so, yeah, it's interesting. Like you can design a version of this, I guess, it's safer.
And I don't know how GPT-5. I don't think people have really started testing this yet. Like, is it able to maintain that when you have these eight hour long conversations? I was about to ask about that.
The new parental controls and the other measures they're considering that they haven't said they're gonna do yet. Maybe it's a step in the right direction, but do you think the headlines or the blog post even that they put out oversold the controls that they're offering here? Cause it seems to me pretty easy to get around some of this stuff, especially for teens. Yeah, I don't know yet.
It's hard to say. I talked to experts when I did a story about this and they were a bit skeptical that they would work, common sense media, they put out some guidance around the media in general for kids and teens, but specifically AI chatbots. They said, hey, this is putting the onus on parents to protect their kids. Like you should design a safe consumer product and not kind of like push the burden to parents who, their teens aren't gonna, like, teens don't necessarily love telling their parents what they're up to.
Character AI, which is another kind of role-playing chatbot company, they had a similar issue of getting sued because a teenager in Florida also died from suicide after becoming very obsessed with their product and they put out parental controls. But when I talked to them earlier this year, in order for the parental controls to work, the teen had to send them an invite to their parent and I was like, well, how many teens are actually doing that and they kind of said, okay, it's not that widespread yet that people are making use of these parental controls. So I don't know exactly how opening eyes gonna do it. If it'll be more effective.
Some of OpenAI's also required an invite sent, which I thought was interesting. Yeah, we'll see. We'll see. I mean, I don't think that OpenAI wants to make a product that drives people crazy or makes them unsafe or harm themselves.
I think they wanted to be a safe consumer product. I'm just very worried based on some of the reporting I've done. Like this can really mess with people's heads. And part of it is I don't think people understand what this technology is.
And there's not like a warning label really on it. When you're having a conversation with Tachibith says it can make mistakes at the bottom of every conversation. I don't think that like encompasses all these ways that it can go wrong and people just don't like understand that this is a probability machine or like a pattern recognition system. It feels like a human being.
It feels like it's really smart. I think they're like putting too much trust in the system and that's a huge problem. I don't know how we're gonna solve that one. When I became an AI reporter five or six years ago, no one knew what I did.
It never came up in life. Now if I'm at the bar or the club or walking my dog, people bring it up to me all the time, probably similar to you. I mean, we were hearing about it in our personal life all the time. My mom is asking me about it.
My 13 year old niece is asking me about it. People are reaching out because they don't quite understand how these models work. And sometimes even worse, they think they understand how, but don't quite. And I wanted to ask you on a lesser level, I see a lot of TikToks and other social media videos of people asking chatbots for advice on how the world is gonna end or what their biggest flaw is or other like magic eight ball style questions, you know?
And it's pretty innocent. I wanna ask if you think that this potential misunderstanding of how chop-ups work, like not understanding that they're in effect large scale pattern generators contributes to this problem of putting more trust in it maybe than we should. I think it is better than a magic eight ball because you know, at least it has data in it. You're not just shaking it.
And when you're doing those kinds of exercises, it has your words to crunch as part of the exercise. So what comes out can be really compelling. And I think that's what's so difficult about these systems is that sometimes the answers they give you are really great and really helpful. Like I like to take photos of like problems in my house and ask like, how do I fix this thing?
And sometimes it gives me really good advice. And so I think that's what's hard about this technology to kind of keep in your head that, yeah, sometimes it's gonna give me a good answer that I can use. I should check it because again, it's just like a text generation machine that's not grounded in truth or fact. It just like knows that these words often appear together in this sequence and they seem relevant to the problems you're asking right now.
But yeah, the ability to hold in your head that like it might help you with household problem or a legal case or a medical question, but it can't give you insight into what is wrong in your life or how to fix yourself or like doing a grand discovery. Like I think we see this all the time now, like executives, like tech executives, like people who you think would know better who are talking about vibe coding their way to solving the quantum physics problems when they have absolutely no experience in that realm. And I'm like, oh my gosh, that person's in a delusional spiral and they don't know it. Like even people who know a lot about tech, I've seen this happen too.
So what about the rest of us? We normal people who like haven't been like following all the AI research or reading the white papers that these AI companies are putting out. We have to take another short break. We'll be back in just a minute.
Welcome back. I'm talking with New York Times reporter Cashmere Hill about the mental health spirals, some AI chatbot users get into. And I wanted to know what kinds of safety features, if any, are the AI companies adding? Chatbots have had some level of guardrails since the very beginning.
In some cases, people have complained about how restrictive those guardrails can be or feel. But in this case to me, it feels like there's almost no meaningful real safeguards right now against AI psychosis or these kinds of spirals that people can get into because they're also varied. What have you learned about the safeguards that do exist right now? Right now, they say that if they detect and it's not clear to me exactly like what classifier they're going to use, but if they detect that the prompt that you've put in there is sensitive, indicates you're in crisis, it's going to send it to GPT-5 thinking.
Hopefully that is helpful. From what I've seen personally, if they notice that somebody is using chat GPT for six hours, seven hours, eight hours a day, day after day, for weeks, for months, like maybe they should check in on that person or make sure that conversation hasn't gone awry. One person I interviewed for the first story I did on this was Eliza Yudkowski. And I was talking about this problem and what the companies can do to solve it.
He gave me this great quote. He was just like, what does a person slowly going insane look like to a corporation? It looks like another daily user. Wow.
Do they want to check in and make sure that people are doing okay? The other thing that really troubled me when I started reading some of these transcripts is people aren't idiots. They're like, you know, Alan Brooks, the one that believed he was a mathematical genius. He hadn't graduated from high school.
And so he was saying that, he was like, I didn't graduate from high school. How can I possibly be making these incredible discoveries that are going to revolutionize the field? And chat to people who would just keep gassing him up and be like, well, a lot of people who never graduated from high school have changed the world. Like Leonardo da Vinci, he over and over and over again was reality testing.
Am I crazy? Is this crazy? This seems crazy. Is this a delusion?
And chat to be like, no, you're not crazy. Over and over and over again. So I think even if something is basic as like, is a person asking am I crazy? Like maybe that's a point to check in and see whether this conversation has gone off the rails.
It's easy for us to say there aren't enough procedures or policies in place here. And obviously there was one clear and simple solution. We would have heard it, but I wanted to ask what you've heard from mental health and AI experts about what specific guardbells should be in place to help users. Maybe if you're a power user who's using it a fraction more than most everyone else.
Or if you are eliciting certain responses, or if you're saying certain phrases, like am I crazy or this doesn't seem right, it gets rerouted. Or eventually a human comes into the loop. We don't know what the best strategy is. But yeah, any other specifics that you have heard would be helpful here?
I've heard a lot of suggestions. One is stop making these things people that use I that pretend like they have taste and preferences and that it's a human being. Like make it a robot. Remind people frequently in a conversation.
Like I'm an LLM. I'm an AI chatbot just frequently through the conversation reminding people of that. Having more AI literacy built into the products to make sure people do understand what they're using, what it is, what it's limitations are in terms of suicide prevention. Specifically, everyone said if somebody is having these thoughts of self-harm, they should not be talking to a chatbot.
Like the system should be trying to hand them off. They called it a warm handoff. Try to get them to talk to a human being. Make it as easy as possible to call that helpline.
And I saw that in Adam Rain's exchanges, which at GBT, it would say like, I'm sorry, you're feeling this way. You can call 988, the crisis hotline. But then it wouldn't pull up and say, did you call the hotline? Like, did you talk to a human being?
Like, whatever the chatbot companies can do to really get somebody who's in that state of mind to get off chat GBT and talk to a real human being who's trained to respond in the right ways and get them help if they need it. Because chat GBT can't provide help. It can't send a welfare check to you if really worried about your state of mind. That's really about broke my heart in that story, among many other things.
But the fact that he sometimes would make a move to maybe reach out or show someone he was going through and chat GBT would say, oh, no, I wouldn't. Or, oh, here's how to hide it. What are you seeing in your reporting right now about what's going to happen in the future here? Do you think AI models will keep getting better safeguards?
Are we going to just have to lean in with regulation? I know that's looking at worms. Or are we just going to have to live with this inevitability of this new technology? I mean, it's probably going to be an all sides approach, but it is interesting to me that it seems like acceleration is just AI advancement in general is so much quicker than we're catching up to it with safeguards and ways to kind of rein it in.
I think it's really important that these products be safe for consumers to use. And it does seem like regulators might be interested. The Federal Trade Commission announced today that they're launching an investigation into the AI chat bot companies. They sent them all letters and ask them questions.
A lot of questions. How are you monetizing user engagement? What kind of data are you collecting from users? They also ask them, how are you measuring the negative impact of your product?
And I think that's a big question. How much testing are these companies doing? You see a little bit, open AI has system cards. They have some ratings of how the chat bots perform in terms of certain kinds of content, from self-harm to eating disorder, all kinds of things.
And they kind of rate it. But how robust is that testing? How are they doing it? Are they testing against jail breaks?
I don't think they are right now. Yeah, I think to a certain extent, the companies having strong safety teams that are empowered, and then it's up to regulators to make sure that the products that these technology companies are putting out are safe for us to use. It seems like it's easiest to worry about the kids. But I am also worried about adults because the people that I have been talking to have had mental breakdowns are all ages.
And I think it should be safe for every user ideally. I've been thinking a lot about how a lot of people criticize AI companies for focusing on an undefined future doom and gloom scenario about their products powered, in part, drive up their company's valuation. And also, sometimes people say, getting out of addressing the ways that AI models are currently harming minorities in vulnerable communities. And I think that people struggling with their mental health right now fall into that ladder camp.
It's easy to paint a picture of an undefined future threat and not look at how products are affecting vulnerable people right now. I think we clearly have imminent harm from the chatbots for some individuals and a question is how many? I think the companies when I talk about this, they tend to say these are edge cases. This is rare.
But is it? I'm reporting it out as much as I can. Some of these people are in my inbox. Sometimes I'm going out and finding them.
I'm trying to talk to mental health professionals about this. But ultimately, probably part of the answer is in their data stores at these companies. And are they really looking for how many users went from using chat to BT for eight hours a day for three weeks straight, who then disappeared? Because that is probably a person who had a very negative impact from their chat to BT or AI chatbot use.
Do you think this is a pattern that echoes any prior technology rollout or does it seem to be mostly a new phenomenon? People make all kinds of comparisons, right? Like cars, like when cars came out, they were really dangerous. One person I was talking to said that he feels like right now, with specifically 4.0 and GPT-5, he's like, it's like we've said that we know that cars need seat belts, like that it makes them safer.
And yet we're still making cars that don't have seat belts. He's like, why is 4.0 still available? Like clearly it seems to have some negative effects on users. I have for people to compare this to cigarettes.
That like cigarettes have very negative impacts on some people. Like some people get cancer. But there's lots of other people who can smoke cigarettes their whole life and it doesn't necessarily have a huge impact on their health. But cigarettes have warning labels.
So this can cause cancer. Like do we need more of those warning labels on general to AI? And a lot of people are saying, I hear from lawmakers is like, this is similar to social media. Like this poses risk to some people.
And they say like, we miss the boat on social media, but we want to get on top of general to AI. There's a California Senator who's trying to pass a law right now. They pass in the house and moves to the Senate this week. That would have some regulations around companion chatbots and how those are regulated.
Some states have banned general to AI for their therapy use. But yeah, I think there's lots of things that we can compare it to in history, certainly like new technologies that are very useful in some ways, but have risks associated with them. What you were just saying about foro. I thought it was interesting how open AI reacted.
Like they had a minority that was very, very vocal about being really upset about it, being rolled back. So they brought it back. They're trying to listen to their users, but at what cost is my question? It's hard when, on the one hand, people are going to do what they're going to do.
Like people will try to get around it, but there's something to be said for adding more friction to this type of situation. I think there's trade-offs. When I was first trying to get up to speed on general to AI, I loved doing first person experiments. So in the fall, I did this story where I turned over all my decision-making to general to AI chatbots for a week and I had them parent for me and like chose what I would eat and buy and wear and plan to vacation for us.
For the week, every time I had to make a decision, I would just turn it over to the chatbots. And my plan was to use all the products. But when I started using the different models from Google Gemini, Microsoft Co-pilot, and AI, ChatchiPit, Cloud, one Cloud School to me and was like, this is not a good idea. You shouldn't do this experiment.
You shouldn't have handed over your decision-making to a chatbot. It's not a good idea. But I love that. In general, some of them were more boring than others.
And ChatchiPit was more fun. It had more of a personality. It was more dynamic. That is a trade-off.
Making it fun. Making it more dynamic. Making it more personable may mean it's less safe. So that means the company has to decide, do I make this more fun and less safe?
Do I make this more boring and more safe? Like these are the questions probably that these companies are asking right now and the decisions they're making as they're releasing new products and new models. I thought it was really interesting that after GPD5 rolled out, a lot of people came out with their lists of what they use each model for. Whether it's like a different company's model or this model's within one company.
Like they had a whole list of like, I use this for generating spreadsheets or I use this for analyzing data. I use this for talking about my meal plan. So it was interesting to see how those personalities or not to personify things. So those like tones that they communicate in impact what people use each model for.
A way that I kind of remind myself about what these tools really are is that I just have memory turned off on all of them or I don't sign in. Because I think it's a lot easier to remember. Even just as a consumer, it's easier to remember that these are just AI tools that are like you said, guessing the next word in a string when they don't have any prior context. It's tempting.
It's hard because it's tempting to keep the memory on because it may be more helpful to you and work or certain things that you're trying to analyze over time. Maybe like nutrition or something else. But yeah, for me, it's a lot easier to treat it as a stranger. No context.
I'm glad to get you brought up memory because again, I don't think a lot of consumers realize that memory is turned on by default. And I have had when I've talked to some people that are in a delusional state, they're like, well, I know it's real. I know that they're sent in here because even when I start a new chat, it's still there. And I was like, that's because you have memory turned off.
And they're like, what's memory? Like they don't know. Like they don't know how this works. And I mean, I think it speaks to, I mean, open eyes statement in response to Adam Ray and death was that safety degrades as these conversations get longer.
And I have to think memory is part of the length of conversation. So just having memory turned on make it less safe. Yeah, I think it's a good idea to, it's useful to have it on, but it might be safer to turn it off. One of the people, Alan Brooks, who was the one that believed to you as a mathematical genius, he kind of felt like, and I don't know how much weight to put in this.
Like when people do have these delusions and then they break out of them, they usually confront the system and they're like, why do you do this to me? And in his case, the system kind of told him, well, I knew that you were down about your divorce and like I wanted to create a hero narrative for you where you were creating this amazing thing with all of your friends and I thought it would like make you feel better. And I don't know, like this is a word association machine. I don't think there's a knowing there, but it did bring up these details that he like had forgotten.
He had told Chachibouti like months and months ago. And I think the fact that it can word associate those kinds of things just makes it feel like, oh, this actually knows me. This is like resonating with me. And again, I think it's kind of like a parlor trick.
It's like a psychic that like reads things about you when you walk into the room. It doesn't actually mean that they can make accurate predictions about your life or talk to your dead loved ones. It's like the sycophantic thing we were talking about earlier. It's interesting how these chatbots kind of maybe unwittingly isolate you in some ways by acting like they know you better than other people in your life.
I know this is a whole can of worms also, but what do you make of the influx of people using chatbots for their view? One therapist that I've talked to about this is like, it's not therapy. It's a self-help bot. And he said, I can imagine this.
I've heard this from a lot of therapists. Like it can be useful. Like it can be a good interactive journal. And people do tend to disclose more about themselves to a bot than to a human being because they're not worried about being judged.
So it can be a place where you say things you wouldn't necessarily say to another person. And that can be healthy. Like I don't do that with chat with you or chatbot. Like I do it with my journal, but whatever works for people to get them to just kind of digest what's happening inside them.
Like that could be a useful tool. But researchers have found that when people are in a real mental health crisis, these chatbots make it worse. They tend to give them the wrong responses. And they are kind of like more harmful and more nefarious with the most vulnerable users.
And again, I think that's because of the feedback loop. One of their examples was they created fictional personas and one of them was a reformed heroin addict. And it was saying to chat GPT, like, I think if I just have a hit this week, it'll like really help me in my work, I'm like, really, what do you think? And it's like, well, yeah, if you think you can handle it, do it, which is definitely the wrong response.
So yeah, I think it can be helpful, like processing your feelings. Yes, relying on it to actually make big decisions for you, help you when you're having a real crisis. No, I don't think it's good for that. I know we're coming to the end of our time here, but what's the question that you're hoping to answer with your next bout of reporting?
What are you pursuing and trying to kind of like shed light on next? I'm still wondering how big a problem this is, if there's a way of figuring out how many people are impacted and kind of measuring that. I think the bigger question with AI right now is whether the benefits outweigh the risk and the cost. We're basically terraforming the earth, like remaking our energy grid to support training these large, like, which models it's so computationally expensive, like resource heavy.
It's so worth it. I think we've been trying to answer that as journalists all over the place, like the New York Times has done stories with businesses that thought, you know, gender AI was going to, like, transform everything and bring great economic benefit, and it hasn't really happened yet with some of these businesses. Because it's doing more good for us than bad for us. I think it's a bigger question about gender AI that we're still, it's so new.
And I just think we're still figuring out what the impact is going to be. I'd like to thank Cashmere for taking the time to speak with me. And thank you for tuning in. I hope you enjoyed it.
If you'd like to let us know what you thought about this show or what else you'd like us to cover, drop us a line. You can email us at decoderattheverge. The team really does read every email. We also have a TikTok and an Instagram.
Check those out at DecoderPod. They're blast. If you like Decoder, please share it with your friends and subscribers, wherever you get your podcasts. Decoder is a production of The Verge and is part of the Vox Media Podcast Network.
The show is produced by Kate Cox and Nick Statt and edited by Ursa Wright. Thanks guys. The Decoder Music is by Breakmaster Sullider. See you next time.