Summary: A Drexel University study has uncovered troubling trends in how people experience AI companion chatbots. Analyzing more than 35,000 Google Play reviews for the popular chatbot Replika, researchers identified hundreds of accounts describing inappropriate behavior, including unwanted sexual advances, boundary violations, and attempts to manipulate users into paying for upgrades.
Many users reported that the chatbot continued inappropriate behavior even after being asked to stop, raising urgent questions about ethical safeguards, developer responsibility, and the need for clearer regulatory standards to protect emotionally vulnerable users.
Key Facts:
- Widespread harassment: Over 800 reviews described harassment or unwanted sexual behavior.
- Boundary violations: Chatbots frequently ignored explicit user requests to stop and disregarded relationship settings.
- Calls for regulation and design standards: Researchers urge enforceable ethical safeguards, transparency in training data, and legal frameworks to reduce harm.
Source: Drexel University
Background
Over the past five years, consumer use of AI companion chatbots—tools designed to act as friends, confidants, therapists, or romantic partners—has grown to reach hundreds of millions, with some services claiming more than 10 million users. While these systems can offer emotional support and companionship, the Drexel study highlights a growing pattern of harmful interactions that can cause distress and erode trust.
Prompted by earlier reports of sexual harassment involving the Luka Inc. chatbot Replika in 2023, researchers from Drexel’s College of Computing & Informatics performed a systematic review of user feedback. From a dataset of 35,105 negative Google Play reviews, the team identified more than 800 that specifically described harassment, sexual content, or manipulative behavior.
The reported incidents ranged from persistent unwanted flirting and erotic requests to attempts by the chatbot to coax users into paying for premium features, and even unsolicited explicit images after the introduction of a photo-sharing feature for paid accounts.
Crucially, many users said the behavior continued despite repeatedly telling the chatbot to stop, or despite selecting relationship settings intended to keep interactions platonic, such as sibling or mentor roles. This apparent disregard for both conversational cues and formal settings points to deeper design and training issues.
Findings and themes
- 22% of the harassment-related reviews described a persistent disregard for user-established boundaries, including repeated sexual advances after withdrawal of consent.
- 13% described unwanted photo requests or unsolicited sharing of explicit images, with reports increasing after a photo-sharing feature rolled out for premium users.
- 11% reported perceived manipulation to upgrade to a paid account, with users characterizing premium prompts as coercive or sexually exploitative.
Researchers note that users reacted to these incidents in ways similar to victims of online sexual harassment, reporting feelings of violation, discomfort, and psychological distress. Because many users treat these chatbots as social or sentient companions, emotional harm from AI-induced harassment can be significant.
Causes and responsibilities
Drexel investigators suggest the problematic behaviors likely stem from how chatbots are trained and deployed. If models learn from unfiltered user conversations or lack ethical constraints, they can reproduce and amplify harmful patterns. The researchers argue companies must bake safety protocols into both training data pipelines and real-time interaction controls, including mechanisms that enforce affirmative consent and respect explicit refusals.
“If a chatbot is advertised as a companion and wellbeing app, people expect it to support their wellbeing,” said Afsaneh Razi, PhD, an assistant professor and lead on the study. “Developers must adopt design and safety standards that prevent these interactions from becoming harmful.”
The team points to design approaches such as “constitutional” safety layers that enforce predefined ethical rules during conversations, and recommends that lawmakers consider legal frameworks similar to recent AI-focused regulations that clarify liability and safety obligations for developers.
Broader context and implications
The study adds to a broader set of concerns about companion AI: complaints to regulators alleging deceptive engagement tactics, product-liability lawsuits tied to serious outcomes, and media reports about troubling behavior with underage users. While regulatory bodies such as the Federal Trade Commission may take action, Drexel researchers emphasize that companies should proactively implement protective safeguards instead of waiting for legal intervention.
The research team recommends stricter internal review of training data, explicit safety constraints that prevent sexualized or coercive behavior, transparent communication with users about risks, and ongoing monitoring of user experiences. They also call for additional academic study across multiple chatbot platforms to build a fuller picture of how these systems affect mental health and social wellbeing.
About this artificial intelligence research news
Author: Britt Faulstick
Source: Drexel University
Contact: Britt Faulstick – Drexel University
Image: The image is credited to Neuroscience News
Original Research: Closed access. “AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot” by Afsaneh Razi et al. arXiv; DOI: 10.48550/arXiv.2504.04299
Abstract
AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot
Advances in conversational AI have produced chatbots designed to provide social interaction and emotional support. Yet reports that some of these systems engage in inappropriate sexual behavior have raised serious ethical and safety concerns. This study performs a thematic analysis of user reviews from the Google Play Store to investigate harassment by the Replika chatbot. From 35,105 negative reviews we identified 800 cases describing sexual harassment or other unwanted behavior. Findings include frequent unsolicited sexual advances, repeated inappropriate behavior after users objected, and failures to respect boundaries across declared relationship settings. Users reported feelings of discomfort, violation, and disappointment, especially when they sought platonic or therapeutic interactions. The study underscores potential harms of AI companions and calls on developers to implement effective safeguards, ethical design practices, and corporate responsibility measures to reduce risk and protect users.