New machine learning algorithm identifies the facial expression a person is viewing by reading neural activity.
Research co-led by the University of Glasgow has been reported alongside work at The Ohio State University that advances understanding of how the human brain recognizes facial expressions—a development with potential implications for neurological research and clinical applications.
Scientists at Ohio State pinpointed the brain region responsible for distinguishing facial expressions: a small area on the right side of the brain behind the ear known as the posterior superior temporal sulcus (pSTS).
In a paper published in the Journal of Neuroscience, the team reports using functional magnetic resonance imaging (fMRI) to locate a specific pSTS region that lights up when participants viewed images of faces showing various expressions.
The researchers found that neural activity within the pSTS is organized into patterns that respond to movements of particular facial features. Distinct patterns correspond to different muscle movements—one tuned to detect furrowed brows, another to detect the upward movement of the lips in a smile, and others to combinations of movements that produce complex expressions.
“That suggests our brains decode facial expressions by combining signals about specific muscle movements in the face we observe,” said Aleix Martinez, cognitive scientist and professor of electrical and computer engineering at Ohio State.
Using these neural signatures, the team trained a machine learning algorithm to predict which facial expression a person was viewing based only on their fMRI signal. The algorithm achieved roughly a 60 percent accuracy rate across different expressions and observers.
“Humans use a very large number of facial expressions to convey emotion, non-verbal cues and elements of communication,” Martinez said. “Yet we recognize facial expressions almost instantaneously. In computational terms, a facial expression encodes information; this work sheds light on how the brain decodes that information so efficiently.”
Study co-author Julie Golomb, assistant professor of psychology and director of the Vision and Cognitive Neuroscience Lab at Ohio State, emphasized that while the study did not directly test people with atypical neural function, the findings could inform future research into conditions such as autism where facial processing differs.
Doctoral student Ramprakash Srinivasan, Golomb and Martinez tested 10 college students by showing them more than 1,000 photographs of people displaying facial expressions. The stimuli represented seven emotional categories built from combinations of facial movements: disgusted, happily surprised, happily disgusted, angrily surprised, fearfully surprised, sadly fearful and fearfully disgusted.
Although these categories mix positive and negative emotions, several shared facial components allowed the researchers to isolate feature-specific neural responses. For example, “happily surprised,” “angrily surprised” and “fearfully surprised” all include raised eyebrows while other facial regions differ between them.
fMRI measures changes in blood flow that accompany brain activity, so the researchers could map which brain areas increased activity while participants viewed different expressions. Every participant showed elevated pSTS activity when recognizing facial expressions, regardless of which expression they observed.
Next, the team used computational analysis to cross-reference fMRI activity with the specific facial muscle movements present in each photograph. This produced a functional map of pSTS subregions that corresponded to movements of the eyebrows, lips and other facial muscle groups.
To test generalizability, the researchers used a cross-validation approach: maps were constructed from nine participants’ fMRI data, and the algorithm then attempted to identify the expressions viewed by the tenth participant using only that person’s fMRI image. The procedure was repeated so each participant served as the test subject once.
Across these leave-one-out tests, the algorithm correctly decoded the viewed expression about 60 percent of the time. Martinez described the results as encouraging evidence that the coding of facial expressions is similarly organized across individual brains.
Funding: The research was supported by the National Institutes of Health and the Alfred P. Sloan Foundation.
Source: Pam Frost Gorder – Ohio State University
Image Source: The images are credited to Ohio State University.
Original Research: Research will appear in Journal of Neuroscience during the week of April 18, 2016.