New computational model more than triples the number of facial expressions researchers can use to trace the origins of emotion in the brain.
Researchers at The Ohio State University have developed a computational method that recognizes 21 distinct facial expressions, including complex or mixed emotions such as “happily disgusted” and “sadly angry.” Published in the Proceedings of the National Academy of Sciences, this work expands the set of measurable expressions available to cognitive scientists and offers a more precise way to connect facial behavior with the neural, genetic, and chemical processes that underlie emotion.
Aleix Martinez, a cognitive scientist and associate professor of electrical and computer engineering at Ohio State, emphasizes the significance: “We’ve moved beyond simple labels like ‘happy’ or ‘sad.’ We found strong consistency in how people use facial muscles to express 21 emotion categories. That consistency suggests these expressions are produced similarly across individuals in our study.”
For centuries, from Aristotle to modern scholars, people have sought to understand how facial movements reveal inner feelings. Today’s cognitive scientists aim to translate those facial signals into measurable categories so they can map emotion onto brain circuits. Until now, much of this research focused on six basic emotions—happy, sad, fearful, angry, surprised, and disgusted—because their facial signatures were considered clear and widely shared.
Martinez compares relying on only six categories to painting with just primary colors: it creates a recognizable but oversimplified image. By identifying a broader set of expressions, the new model enriches that palette and enables more nuanced scientific analysis of how the brain encodes emotion.
To build their dataset and algorithm, the team photographed 230 volunteers (130 female, 100 male, mostly college students) producing facial responses to verbal cues such as “you just got some great unexpected news” (to elicit “happily surprised”) or “you smell a bad odor” (to elicit “disgusted”). From roughly 5,000 resulting images, researchers carefully annotated anatomical landmarks—key points on the face such as mouth corners and eyebrow edges—using the Facial Action Coding System (FACS), a standardized method developed for rigorous analysis of facial muscle movements.
Analyzing the FACS annotations, the research team identified 21 distinct emotional expressions. These included the six basic emotions plus a set of compound emotions formed by combinations of those basics. “Compound emotions” describe blended or simultaneous feelings—examples in this study include “happily surprised” (a mix of happiness and surprise) and “sadly angry” (a blend of sadness and anger). The model quantifies how much each basic emotion contributes to a compound expression, revealing systematic patterns in facial behavior.
Some expressions proved highly consistent across participants. Happiness, for instance, was expressed by raised cheeks and a stretched mouth in 99 percent of cases. Surprise was marked by wide eyes and an open mouth 92 percent of the time. The compound “happily surprised” combined the wide-open eyes of surprise with the raised cheeks of happiness in about 93 percent of instances, producing a reliable hybrid face that the model successfully recognized.
The model also clarifies how seemingly contradictory emotions manifest together. “Happily disgusted” combines the scrunched nose and narrowed eyes of disgust with the smiling mouth of happiness—an expression one might make when something is both gross and amusing. By formalizing these combinations, the computational approach helps researchers study more realistic emotional experiences that often contain mixed or layered feelings.
While the model’s primary purpose is basic research in cognition and neuroscience, Martinez and colleagues note possible clinical implications. Conditions such as post-traumatic stress disorder (PTSD), which can heighten sensitivity to fear and anger, or autism spectrum disorders, which often involve difficulty recognizing others’ emotions, could be better understood by examining responses to compound emotional expressions. The expanded set of measurable expressions allows scientists to generate and test hypotheses about neural pathways and chemical systems that give rise to specific emotional patterns, potentially guiding future therapeutic strategies.
Coauthors on the study included doctoral students Shichuan Du and Yong Tao. The research received partial funding from the National Institutes of Health. For inquiries, contact Aleix Martinez, Ohio State University.
Source: Ohio State University press release. Image credit: Ohio State University. Original research: “Compound facial expressions of emotion” by Shichuan Du, Yong Tao, and Aleix M. Martinez, published in Proceedings of the National Academy of Sciences. Published online March 31, 2014.