Summary: New research shows that context plays a central role in how people perceive others’ emotions.

Source: UC Berkeley

Context Matters: Faces Alone Don’t Always Reveal True Emotions

In the film 127 Hours, actor James Franco appears to smile as he records a video diary. Only when the camera pulls back to reveal his arm pinned beneath a boulder does the smile’s true meaning become clear.

That cinematic example illustrates a key finding from a recent study at the University of California, Berkeley: visual context — including background, scene elements, and actions — is as important as facial expressions and body posture when people interpret emotional states.

The study, to be published in the journal Proceedings of the National Academy of Sciences, challenges long-standing assumptions that reading emotions depends primarily on decoding facial micro-expressions for categories like happiness, sadness, anger, fear, surprise, disgust and contempt. Instead, the research shows viewers frequently rely on surrounding cues to infer what someone is feeling.

“Our study reveals that emotion recognition is, at its heart, an issue of context as much as it is about faces,” said Zhimin Chen, the lead author and a doctoral student in psychology at UC Berkeley.

Chen and colleagues blurred faces and bodies in dozens of short, muted clips drawn from Hollywood films, documentaries and home video. Even with characters visually obscured, hundreds of participants were able to judge emotional states accurately by attending to contextual information — where people were positioned, how they interacted, objects in the scene, and changes in the environment.

To capture these dynamics, Chen developed an “affective tracking” method that records viewers’ moment-to-moment ratings of characters’ emotions while they watch video. The tool tracks cursor movements across a rating grid superimposed on the video, allowing researchers to collect fine-grained, time-locked ratings from many participants quickly.

Chen noted the method could be useful beyond basic research. It may help chart how people with conditions such as autism or schizophrenia process emotional cues in real time, revealing whether deficits stem from facial decoding, contextual reading, or both. That insight could eventually inform diagnostics and interventions.

“Some people might have difficulty recognizing facial expressions but can read emotion from context. For others, it’s the opposite,” Chen said.

The findings also have implications for emotion-recognition technology. Many machine learning systems are trained on cropped faces and therefore learn to classify emotions solely from facial appearance. The Berkeley results indicate such systems are likely missing crucial information if they ignore scene context.

“Right now, algorithms trained only on faces can only read emotions from faces,” Chen said. “Our research shows faces don’t always reveal true emotions very accurately, and systems that aim to identify a person’s state of mind should incorporate context as well.”

The research team, led by Chen and senior author David Whitney, a vision scientist and psychology professor at UC Berkeley, tested nearly 400 young adults across three experiments. Stimuli included movie scenes, documentary footage and candid home videos selected to display natural emotional responses in realistic settings.

In the first experiment, 33 participants watched movie interactions in which one of two characters was blurred and rated the emotions of the blurred person. Viewers used interpersonal cues and background elements to infer the blurred character’s emotional state, demonstrating that context and interaction dynamics provided sufficient information.

In a larger follow-up, roughly 200 participants viewed clips under three conditions: fully visible scenes, scenes with blurred characters, and scenes with blurred context. Results showed that removing contextual information degraded emotion recognition to a similar extent as blurring faces, demonstrating that context is as essential as facial information for decoding emotional states.

The final experiment involved 75 participants who rated documentary and home-video clips to confirm the effects in more naturalistic settings. Again, context proved critical: viewers relied on background cues and action patterns as much as facial expressions and gestures to infer emotions.

“Overall, the results suggest that context is not only sufficient to perceive emotion, but also necessary to perceive a person’s emotion,” said David Whitney. “Face it, the face is not enough to perceive emotion.”

Source: UC Berkeley (Yasmin Anwar)
Publisher: NeuroscienceNews.com (organized coverage)
Image credit: Jeffrey Zhang
Original research: Study to appear in Proceedings of the National Academy of Sciences (PNAS)

UC Berkeley. “Our Faces Don’t Always Reveal Our True Emotions.” NeuroscienceNews, February 26, 2019. Original research published in PNAS.

Feel free to share this article and its findings about how context shapes emotion recognition.