Summary: By combining machine learning with brain scans, researchers reveal how people reconstruct the meaning of objects from a few clues.
Source: Aalto University
What is an S-shaped, scaly animal with no legs? What has large ears, a trunk and tusks? What makes a ‘woof’ and chases cats? Researchers at Aalto University used brain imaging and machine learning to show how the brain reconstructs concepts such as “snake,” “elephant” and “dog” from minimal clues. The study was published in Nature Communications.
In the experiment, participants were given three isolated verbal clues describing semantic features of familiar objects. The clues covered a range of everyday categories, including animals, vegetables, fruits, tools and vehicles. Importantly, the actual images or names of the target objects were never presented; instead, subjects had to infer the object from the feature clues alone.
The research team found that brain activation patterns, measured with functional MRI (fMRI), contained information not only about the three presented features but also about a much broader set of semantic features typically associated with the target object. Using large-scale text-derived feature databases and machine-learning-based neural decoding, the researchers trained models to map semantic features to patterns of brain activity. These models were then able to reliably infer which object a participant was thinking of based on their brain activation—for example, distinguishing whether the clues pointed to an elephant or to a dog.
The findings indicate that the brain rapidly combines fragments of information and activates a rich, integrated representation of an object’s meaning. Rather than representing only the explicitly provided clues, neural activity reflects the full network of properties learned over a lifetime: shape, typical behaviors, sounds, uses and other associated attributes. This flexible semantic reconstruction allows fast, accurate identification from sparse input—a useful survival skill when partial information must trigger an appropriate response. “For example, we automatically step away from a wriggling shape on a rocky shore because we understand that it might be a snake and could be dangerous,” says Sasa Kivisaari, postdoctoral researcher at Aalto University.
Machine learning, large-scale text data and neural decoding
To link semantic features to brain activity, the study leveraged extensive internet-based language material to extract meaningful features associated with each concept. Machine learning models learned the relationship between those features and BOLD activation patterns measured with fMRI. The trained decoders demonstrated that neural patterns are best explained by a larger semantic feature set than the few cues provided, revealing how partial information triggers a broadened semantic activation in the brain.
Individual differences, communication and clinical implications
The study’s approach also sheds light on why people sometimes perceive or understand the same concept differently. Semantic organization in the brain varies across individuals, which can influence how easily they interpret limited information or communicate meanings to others. “The organization of meanings in the brain differs from person to person and can affect how easy or hard it is for them to understand one another,” notes Professor Riitta Salmelin.
Beyond basic science, these results have potential clinical relevance. The brain regions involved in assembling and understanding meaningful information overlap with areas affected in early Alzheimer’s disease. Because the decoding method reveals how fragmented information is combined into complete semantic representations, it may prove useful in identifying early changes in semantic processing that accompany memory disorders. “Combining and understanding meaningful information seems to involve the same brain areas that are damaged in early Alzheimer’s disease. Therefore, the method we use may also be applied to the early detection of memory disorders,” says Kivisaari.
Professor Riitta Salmelin’s research group at the Department of Neuroscience and Biomedical Engineering, Aalto University, investigates the neural basis of language processing and semantic information. This research was supported by the Academy of Finland, the Aalto Brain Centre and the Sigrid Jusélius Foundation.
Source: Dr. Sasa Kivisaari, Aalto University
Publisher: NeuroscienceNews.com
Image credit: Sasa L. Kivisaari
Original research: Open access article “Reconstructing meaning from bits of information” by Sasa L. Kivisaari, Marijn van Vliet, Annika Hultén, Tiina Lindh-Knuutila, Ali Faisal & Riitta Salmelin. Published in Nature Communications on February 13, 2019. doi: 10.1038/s41467-019-08848-0
Abstract
Reconstructing meaning from bits of information
Modern semantic theories propose that word meanings can be decomposed into unique combinations of semantic features (for example, “dog” includes “barks”). This study demonstrates, using fMRI, that the brain combines isolated bits of information into coherent object representations. Participants received three isolated semantic features as verbal descriptions. Machine-learning neural decoding was used to learn mappings between individual semantic features and BOLD activation patterns. Brain patterns were best decoded using not only the three presented features but also a far richer set of semantic features typically linked to the target object. These results show that fragmented information is combined into a complete semantic representation and identify brain regions associated with object meaning.