Link: https://www.techspot.com/news/114091-florida-woman-used-claude-diary-anthropic-reported-shoot.html

What just happened? Another incident has taken place that illustrates the need to be careful what you tell AI. A Florida woman is facing felony charges after she used Claude as a diary and allegedly wrote that she planned to "shoot up" the Sheriff's office. After a human reviewer examined the statements, they were reported to police.

According to the arrest report, Carli Michelle Heller, of Bonita Springs, Florida, wrote on September 26 that she would attack the Sheriff's office. She later said that she uses Anthropic's chatbot like a "diary."

Claude's safety systems flagged the entry and it was escalated to a human reviewer. After deciding it was a credible threat, the reviewer reported it to law enforcement.

The company says it may share user information in limited emergencies if it believes disclosure is necessary to prevent death or serious physical injury.

Deputies identified Heller and visited her home. She was detained without incident before an LCSO intelligence detective took over the investigation.

Heller faces a charge of making a written threat of violence under Florida law. Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.

Anthropic isn't going to be taking any chances when it comes to anything it deems a potential threat. Last month, it was reported that OpenAI and Sam Altman are being sued by British Columbia over claims that the company could have prevented a mass shooting in the Canadian province.

The shooter, eighteen-year-old former pupil Jesse Van ⁠Rootselaar, had previously been flagged by OpenAI's safety team for her conversations about gun violence, but OpenAI never alerted police because the conversations did not meet the threshold for legal referral.

In June, Florida also sued OpenAI and Altman, alleging that ChatGPT had contributed to real-world harms, including the 2025 Florida State University shooting.

The latest incident is another reminder to think before you enter something into a chatbot that could get you into trouble. It's certainly not a private diary whose contents are for your eyes only.

Reports last month revealed that human contractors reviewing Microsoft Copilot's image editor can see users' prompts, uploaded photos and AI-generated edits. Documents show that some of those assignments contain sexual, disturbing or potentially illegal material, though the reviewers are not there to flag the content – only to assess whether the output is accurate.

gen-ai chatbots have been advising people in how best to kill themselves with no one who owns a gen-ai company being held legally responsible, but when police are potentially at risk, suddenly there are security measures in place & legal consequences 🫩

Florida Statute 836.10 makes it a second-degree felony to send, post, or transmit a written or electronic record threatening to kill or injure someone, carry out a mass shooting, or commit an act of terrorism. The communication must be made in a manner in which another person may view it.

Emphasis mine.

I think any message that you type to an AI chatbot has the expectation that it will not be viewed by another person. This should be a quick case. Way to go Anthropic, you've done nothing except let people know that you will hand them over to the police, and given this woman another reason to not like her local police department.

Thought crime! Thought crime!

The rest of that law states it is against the law to transmit it. She transmitted it because that's how talking to a non-local LLM works. She broke the law, the question is, did she know that she did not have the expectation of privacy.

Dude, how is talking to AI transmitting the message? If you have kidnapped someone and about to ask for ransom, would you a) call his family or b) tell ChatGPT to notify his family???

Because it doesn't process the request or token on machine. So it was transmitted over the Internet to the model that processed it. She didn't intend to transmit it (probably). But unfortunately she did exactly that. Claude isn't a local model and so far as I can tell unless you opt out of a lot of stuff (such as allowing Anthropic to train on your LLM chat data), you will be transmitting whatever you put into it over the internet.

If she had not transmitted it anywhere then Anthropic would not have been able to report her to the police.

I'm on her side but how do you think her message got to the server lol

Which server? Is the victim to be owning the AI server and constantly reading all messages sent to it?

...Do you think she's hosting Claude locally...?

The text she wrote was transmitted to a server to be processed. That is how every cloud LLM works.

Again I don't support her arrest but that's what transmitting means in the context of the internet

No, apparently she wasn't hosting any server.

And yes, I also don't think that constitute as transmitting.

How do you think her messages got to Claude? Perhaps TCP??

Oh, nice, you judge by the name, which is so lame because I can press charges when someone shouts in a tree hole, which connects to Yggdrasil.

I'm just explaining the law, I don't agree with it. If you want to be obtuse about it, go ahead

Meanwhile, in an alternate universe:

Carli Michelle Heller, of Bonita Springs, Florida, has been arrested for shooting and killing several officers at the sheriff's office. Police are saying this tragedy could have been prevented by Anthropic; on September 26, Heller divulged her plans to Anthropic's chatbot Claude, but the message was neither flagged nor reviewed by a human.

This same comments section:

How many times is this going to happen? How do they not at least have flags in place to make sure stuff like this gets caught?