How Our Preconceived Ideas Shape the Way We See Emotions on Faces

Summary: A new NYU study reveals that how we interpret facial expressions depends on our preconceived concepts of emotions.

Source: NYU

A study from New York University finds that our perception of other people’s facial expressions is shaped not only by the visual cues on a face but also by the concepts and beliefs we bring to the moment. Published in Nature Human Behaviour, the research provides new evidence about how conceptual knowledge influences emotion perception—an ability that plays a central role in social interaction, negotiations, and everyday communication.

“Reading emotion on a face often feels immediate and objective, yet our visual impressions can differ depending on the conceptual beliefs we hold about emotions,” says Jonathan Freeman, senior author of the paper and an associate professor in NYU’s Department of Psychology and Center for Neural Science. “People vary in the facial cues they rely on when identifying emotions, and those differences appear tied to how they conceptually relate different emotions.”

The study, led by Freeman with NYU doctoral student Jeffrey Brooks, examined how people’s internal organization of emotion concepts affects their moment-to-moment perception of facial expressions. Participants first reported how similarly they thought pairs of basic emotions were related in their mind—specifically the six commonly studied emotions: anger, disgust, happiness (joy), fear, sadness, and surprise. For some individuals, anger and sadness might feel similar because both are associated with actions like crying or physical expression; for others, those emotions may feel distinct because they are associated with different experiences and outcomes.

Researchers then tested whether those conceptual relationships predicted how participants perceived faces. In a series of experiments, participants viewed facial expressions and made rapid judgments about which emotion the face displayed. To capture subtle, unconscious influences on these judgments, the team used an innovative mouse-tracking method developed by Freeman. This approach records the trajectory of a participant’s hand movement during forced-choice responses, revealing momentary competition between alternative emotion categories that a simple final choice would hide. Because responses must be made quickly, the mouse trajectories expose implicit tendencies rather than post-hoc rationalizations.

Across experiments, the results were consistent: when a participant judged two emotions to be conceptually similar, faces belonging to those emotion categories were also perceived as visually more similar. Mouse trajectories showed that the hand often veered toward both category responses simultaneously—indicating a concurrent activation of the two emotion concepts—even though each face was intended to depict a single emotion.

In the final experiment, the researchers used reverse-correlation to create visual prototypes that reflect how each participant mentally represents the six emotions. Starting from a neutral face, many variants were generated by overlaying patterns of random noise. On each trial, participants chose which of two noisy images looked more like a specific emotion (for example, “anger”). Averaging the noise choices produced a personalized facial prototype for each emotion. Consistent with the mouse-tracking findings, when a participant considered two emotions more conceptually similar, the resulting visual prototypes for those emotions were also more physically similar.

“These findings suggest that perceived facial expressions reflect both the information in the face and the perceiver’s conceptual knowledge about emotions,” Freeman explains. “For example, when someone views fear and anger as conceptually close, the two emotions will appear more alike on faces to that observer. That means people may rely on different facial cues depending on their personal understanding of what each emotion signifies.”

The study challenges classic theories that assume discrete, universally recognized facial expressions for each emotion. Under those traditional accounts, a standardized facial pattern—for example, a scowl for anger—should trigger the same emotion label for everyone, regardless of individual beliefs. In contrast, this research indicates that individual differences in conceptual knowledge systematically shape perceptual processing.

Beyond human perception, the authors note potential implications for artificial intelligence and machine learning systems that perform facial emotion recognition. Current algorithms often focus on visual pattern detection; incorporating conceptual or contextual models of emotion could improve sensitivity and reduce misclassification, particularly across diverse populations with differing emotion concepts.

Funding: The research was supported in part by the National Institutes of Health (R01-MH112640).

Source: NYU

Publisher: NeuroscienceNews.com (organized coverage)

Image credit: Jonathan Freeman

Original research: “Conceptual knowledge predicts the representational structure of facial emotion perception” by Jeffrey A. Brooks & Jonathan B. Freeman, Nature Human Behaviour (published July 23, 2018). doi: 10.1038/s41562-018-0376-6

Conceptual knowledge predicts the representational structure of facial emotion perception

Contemporary theories propose that conceptual knowledge dynamically interacts with visual processing of facial cues, shaping social and emotional perception. This set of studies uses representational similarity analysis together with computer mouse-tracking and reverse-correlation methods to examine individual differences in how conceptual knowledge maps onto facial emotion perception. The findings show that when observers regard two emotions as conceptually similar, their perception of faces from those categories also becomes more similar, even after accounting for physical similarity in the stimuli. Mouse trajectories revealed greater simultaneous attraction to both emotion categories during perception, and reverse-correlated prototypes for conceptually similar emotions were visually more alike. Collectively, the results indicate that differences in conceptual knowledge are reflected in the perceptual processing of facial emotions.