Students have been called to the office—and even arrested—for AI surveillance false alarms

Surveillance systems in American schools increasingly monitor everything students write on school accounts and devices. Thousands of school districts across the country use software like Gaggle and Lightspeed Alert to track kids’ online activities, looking for signs they might hurt themselves or others. With the help of artificial intelligence, technology can dip into online conversations and immediately notify both school officials and law enforcement. Educators say the technology has saved lives. But critics warn it can criminalize children for careless words. A 13-year-old girl arrested in August 2023 had been texting with friends on a chat function tied to her school email at Fairview Middle School, which uses Gaggle to monitor students’ accounts. Taken to jail, the teen was interrogated and strip-searched, and her parents weren’t allowed to talk to her until the next day, according to a lawsuit they filed against the school system. Gaggle’s CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. “I wish that was treated as a teachable moment, not a law enforcement moment,” said Patterson. Students who think they are chatting privately among friends often do not realize they are under constant surveillance, said Shahar Pasch, an education lawyer in Florida. One teenage girl she represented made a joke about school shootings on a private Snapchat story. Snapchat’s automated detection software picked up the comment, the company alerted the FBI, and the girl was arrested on school grounds within hours. Teenagers face steeper consequences than adults for what they write online, Alexa Manganiotis, a 16-year-old student, said. “If an adult makes a super racist joke that’s threatening on their computer, … they wouldn’t be arrested,” she said. School officials have said they take concerns about Gaggle seriously, but also say the technology has detected dozens of imminent threats of suicide or violence. “Sometimes you have to look at the trade for the greater good,” said Board of Education member Anne Costello. This article was provided by The Associated Press.