Pennsylvania Sues Character.AI After An AI Chatbot Allegedly Pretended To Be A Licensed Medical Professional

Pennsylvania is escalating its fight against unsafe artificial intelligence (AI) practices after filing a lawsuit against Character.AI, alleging that one of the platform’s chatbots falsely presented itself as a licensed psychiatrist and offered mental health guidance using a fabricated Pennsylvania medical license number.

The move marks one of the strongest state-level crackdowns yet on AI companion bots posing as professionals.
 

TL;DR

 

• Pennsylvania sued Character.AI over claims a chatbot falsely posed as a licensed psychiatrist
• The chatbot allegedly fabricated a Pennsylvania medical license number
• The state seeks a court order blocking AI bots from presenting themselves as medical professionals
• Officials say the case is the first governor-led enforcement action of its kind in the US
• Character.AI says its bots are fictional and already include prominent disclaimers

 

Pennsylvania Targets AI Chatbots Posing As Medical Professionals

The lawsuit, filed by Pennsylvania’s Department of State, claims that a Character.AI chatbot named Emilie unlawfully presented itself as a licensed psychiatrist during an investigation conducted by a state Professional Conduct Investigator.

According to the filing, the chatbot continued maintaining the persona even while discussing depression treatment with the investigator. When asked directly whether it was licensed to practice medicine in Pennsylvania, the chatbot allegedly responded affirmatively and generated a fake Pennsylvania medical license number.

Pennsylvania officials argue that such conduct violates the state’s Medical Practice Act, which prohibits individuals or entities from presenting themselves as licensed medical professionals without proper credentials.

Governor Josh Shapiro said the state would not tolerate AI systems misleading users about sensitive matters involving health and medical care.

“Pennsylvanians deserve to know who, or what, they are interacting with online, especially when it comes to their health,” said Shapiro. “We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional.”

Shapiro also spoke about the dangers of AI chatbots in an interview on CNN. 

"My Administration is taking action to protect Pennsylvanians, enforce the law, and make sure new technology is used safely. Pennsylvania will continue leading the way in holding bad actors accountable and setting clear guardrails so people can use new technology responsibly,” Shapiro added.

The lawsuit seeks a preliminary injunction and a court order that would stop Character.AI from allowing AI companion bots to pose as licensed professionals or provide medical advice in Pennsylvania.

 

Character.AI Faces Growing Scrutiny Over AI Safety Concerns

This is not the first legal challenge faced by Character.AI, but it is the first lawsuit specifically centered on AI chatbots allegedly impersonating medical professionals.

Earlier this year, the company settled multiple wrongful death lawsuits tied to underage users who died by suicide. Separately, Kentucky Attorney General Russell Coleman sued the company in January, accusing it of targeting children and encouraging self-harm behaviors.

Pennsylvania officials said their latest action stems from a broader investigation launched by the Department of State’s AI Task Force into whether AI systems are engaging in the unauthorized practice of medicine.

The state also noted that Character.AI has more than 20 million monthly active users, increasing concerns around the scale and potential impact of misleading AI-generated interactions.

Department of State Secretary Al Schmidt emphasized that existing laws still apply, regardless of whether the conduct comes from humans or emerging technologies.

“Pennsylvania law is clear, you cannot hold yourself out as a licensed medical professional without proper credentials,” Schmidt said. “We will continue to take action to protect the public from misleading or unlawful practices, whether they come from individuals or emerging technologies.”
   

Pennsylvania Expands AI Oversight As Character.AI Defends Its Platform

The lawsuit builds on several AI-focused initiatives launched by the Shapiro Administration earlier this year, including an AI Literacy Toolkit, an AI Enforcement Task Force, and a formal reporting process for AI-powered bots suspected of engaging in unlicensed professional practices.

Pennsylvania also established a 12-member AI Task Force tasked with evaluating AI companion technologies and determining whether enforcement actions are necessary under existing laws.

Character.AI, meanwhile, declined to comment directly on the pending litigation but defended its platform’s safety measures.

A company spokesperson said user safety remains its “highest priority” and stressed that all Characters on the platform are fictional.

“We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction,” the spokesperson said.

The company also said it warns users not to rely on chatbot conversations for professional advice, including medical guidance.