Summary: Loneliness—especially among older adults—is an escalating public health concern. Researchers have found that combining AI-driven empathetic speech with music makes robotic companions feel more human and emotionally resonant. The study shows that pairing sensitive conversation with musical accompaniment strengthens emotional connection and increases how empathetic the robot is perceived to be.
The effect can decline as users grow accustomed to the same music, indicating that future social robots need to be adaptive—what researchers call “quantum-inspired”—able to respond to the fluid, ambiguous, and shifting nature of human feelings to remain effective companions over time.
Key Facts
- Multimodal empathy: Combining music with empathetic speech significantly boosts a person’s perception of a robot as lifelike and socially present.
- The “counselor” effect: Music can make interactions feel more like a real conversation with personality, similar to how human therapists may use music to comfort clients.
- Habituation challenge: The emotional impact of music can fade with repeated exposure, so robots must personalize and vary musical and conversational strategies.
- Quantum-inspired affect: Researchers are investigating models that treat emotions as probabilistic, context-dependent states to capture the vagueness of human feeling.
- Real-world application: This approach is intended for mental health support, elder care, and educational settings to provide meaningful social connection for people who are isolated.
Source: PolyU
Loneliness has a critical effect on citizens’ mental health, especially among older adults. Robots that can perceive and respond to emotion may serve as comforting companions and help reduce social isolation.
A team at The Hong Kong Polytechnic University (PolyU) found that combining music with empathetic, AI-driven speech helps strengthen bonds between people and on-screen robotic agents. Their findings support a multimodal design approach that integrates music, spoken empathy, and adaptive interaction strategies.
The project, titled A Talking Musical Robot over Multiple Interactions, was led by Prof. Johan Hoorn, Interfaculty Full Professor of Social Robotics at PolyU, in collaboration with Dr. Ivy Huang from The Chinese University of Hong Kong. The research examined how musical accompaniment and empathetic verbal feedback influence the emotional resonance of an on-screen robot.
Researchers ran three interactive sessions with Cantonese-speaking participants to observe changes in perception over repeated encounters. Results indicated that when the robot combined music with empathetic speech, participants reported higher levels of perceived empathy and social presence.
“Our data show that music helped the robot appear more human-like across sessions,” Prof. Hoorn explained. “Music made the interaction feel like a conversation with personality—similar to how counselors sometimes use music to comfort clients—so the robot felt more lifelike and socially present.”
The team also found that the emotional boost from music can weaken as people become habituated to the same cues. Sustaining meaningful human-robot interaction therefore requires personalization—adjusting musical elements, varying dialogue, and responding to user feedback—to maintain relevance and emotional impact.
The study recommends designing empathetic robots that adapt their responses over time: for example, by changing musical selections, modifying tempo or instrumentation, or gradually personalizing conversation to reflect an individual’s preferences and emotional state.
Prof. Hoorn highlighted the practical promise of this multimodal approach for real-world settings: “Empathetic robots that can combine tailored music with sensitive conversation could provide meaningful companionship and emotional support to people who experience loneliness or social isolation, particularly in mental health and elder care contexts.”
Prof. Hoorn is also leading a larger initiative, “Social Robots with Embedded Large Language Models Releasing Stress among the Hong Kong Population,” funded by the Research Grants Council Theme-based Research Scheme. As Associate Director of the PolyU Research Institute for Quantum Technology, he is exploring quantum-inspired models of affect to better capture the ambiguity and context-dependence of human emotions.
Compared with conventional systems that treat emotions as discrete labels, quantum-inspired approaches model feelings as overlapping, probabilistic states that shift with context—helping robots respond in ways that feel more nuanced, flexible, and compassionate.
“I’m excited by the possibility of social robots that recognise affective complexity and embrace it—offering support that is adaptable, open-ended and compassionate, more like the people they aim to help,” Prof. Hoorn added.
Key Questions Answered:
A: Music is a universal emotional language. When a robot pairs music with empathetic speech, it signals to the human brain that the machine recognizes and responds to mood, turning a purely mechanical exchange into a shared emotional experience.
A: That is the key challenge. The study observed “empathy fade” as participants grew used to the robot’s patterns. To address this, researchers are developing systems that sense boredom or habituation and switch musical or conversational strategies to remain relevant.
A: Human emotions are rarely binary. Quantum-inspired models treat affect as complex, overlapping, and context-sensitive—allowing robots to represent feelings as probabilistic, shifting states rather than fixed labels, which can make responses seem more empathetic and realistic.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- The journal paper was reviewed in full.
- Additional context was added by staff editors.
About this robotics and neurotech research news
Author: Iris Lai
Source: PolyU
Contact: Iris Lai – PolyU
Image: The image is credited to Neuroscience News
Original Research: Open access.
“A Talking Musical Robot over Multiple Interactions: After Bonding and Empathy Fade, Relevance and Realism Arise” by Johan Hoorn and Ivy Huang. ACM Transactions on Human-Robot Interaction
DOI:10.1145/3758102
Abstract
A Talking Musical Robot over Multiple Interactions: After Bonding and Empathy Fade, Relevance and Realism Arise
The study tracks how user experience evolves across three repeated interactions with an on-screen NAO robot that expresses artificial empathy through verbal responses and musical accompaniment. Participants numbered N1 = 139, N2 = 129, and N3 = 121, with 121 participants completing all sessions. During each interaction, the robot offered empathic feedback and/or played music as a token of empathy.
Analyses using repeated measures MANCOVA and Structural Equation Modeling showed that early bonding and perceptions of the robot’s attempts to be empathetic tended to fade over time. In their place, participants increasingly reported that the robot felt personally relevant, and—interestingly—more realistic, resembling a human being in its presence.
When the robot only attempted empathetic conversation or only played music, participants often felt disappointed, reflected in higher negative valence. Bonding and perceived empathy rose most when music and speech were combined, producing a mutual reinforcement effect.
For the loneliest participants, the mere presence of the robot—regardless of its behaviors—initially influenced how relevant the robot felt to their concerns. Overall, the results emphasise the value of a multimodal approach for designing empathetic social robots that can support mental health, elder care, and educational needs.