Summary: By combining neuroimaging with advanced artificial intelligence, researchers have mapped a distributed brain network that represents the meaning of spoken sentences.
Source: University of Rochester Medical Center
How do we hear a sentence and instantly grasp its meaning, even though the same words in a different order can mean something completely different?
A recent study using functional MRI and deep artificial neural networks describes how the human brain constructs sentence-level meaning. Rather than locating this ability in a single brain area, the research shows that sentence comprehension depends on a network of regions working together to form unified, contextualized representations.
“It has been unclear whether integration of sentence meaning occurs in one particular region, like the anterior temporal lobes, or across a broader network of areas,” said Andrew Anderson, Ph.D., research assistant professor at the University of Rochester Del Monte Institute for Neuroscience and lead author of the study published in the Journal of Neuroscience.
Anderson highlights a simple illustration: “The sentences ‘the car ran over the cat’ and ‘the cat ran over the car’ use the same words but express entirely different propositions when the word order changes. Sentence meaning is more than just the sum of individual words.”
To investigate where and how the brain encodes these propositional meanings, the researchers collected fMRI data from participants as they read 240 sentences. The resulting brain activity mapped to a distributed set of regions, including anterior and posterior temporal lobes, inferior parietal cortex, and inferior frontal cortex.
The team applied InferSent, a recurrent deep learning model originally developed to produce sentence-level semantic representations. InferSent takes sequences of word vectors and, through recurrent and non-linear processing, produces unified sentence encodings that capture propositional structure and contextual meaning. Using voxel-wise encoding models, the researchers tested whether InferSent could predict fMRI activation patterns tied to sentence meaning.
Their results show that InferSent predicts aspects of neural activity that simple “bag-of-words” models and grammar-based assemblies of word vectors cannot. This indicates that the brain encodes contextualized, sentence-level meaning across a distributed cortical network rather than confining it to a single anatomical site.
“This is the first time this particular model has been used to predict activity across these regions,” Anderson noted. “The results provide compelling evidence that unified semantic representations of sentences are encoded throughout a language network spanning temporal, parietal and frontal cortex.”
The researchers emphasize the broader potential for combining neuroimaging with AI models to study language processing and clinical conditions. They are now adapting similar methods to examine how language comprehension degrades in early Alzheimer’s disease and to predict brain activity during language production rather than reading.
Additional co-authors on the study include Edmund Lalor, Ph.D., Rajeev Raizada, Ph.D., Scott Grimm, Ph.D., Douwe Kiela (Facebook A.I. Research), and Jeffrey Binder, M.D., Leonardo Fernandino, Ph.D., Colin Humphries, Ph.D., and Lisa Conant, Ph.D., from the Medical College of Wisconsin.
Funding: The research was supported by the Del Monte Institute for Neuroscience’s Schmitt Program on Integrative Neuroscience and by the Intelligence Advanced Research Projects Activity.
About this AI research news
Source: University of Rochester Medical Center
Contact: Kelsie Smith Hayduk – University of Rochester Medical Center
Image: The image is in the public domain
Original Research: Closed access. Title: “Deep artificial neural networks reveal a distributed cortical network encoding propositional sentence-level meaning.” Journal: Journal of Neuroscience. Authors include Andrew James Anderson, Douwe Kiela, Jeffrey R. Binder, Leonardo Fernandino, Colin J. Humphries, Lisa L. Conant, Rajeev D. S. Raizada, Scott Grimm, and Edmund C. Lalor.
Abstract
Deep artificial neural networks reveal a distributed cortical network encoding propositional sentence-level meaning
Understanding how and where the brain constructs sentence-level meaning from sequences of words is a major scientific challenge. Prior work has used vector-based word meaning models derived from text corpora to explain portions of sentence-evoked brain activation and to map semantic representations across a network spanning temporal, parietal, and frontal cortices.
However, those models often treat sentences as unordered collections of words and therefore cannot determine whether brain activity reflects unified sentence representations or merely overlapping activations evoked by individual words. To address this, the study used InferSent, a recurrent deep neural network trained on sentence inference, to generate propositional sentence representations that capture word order and context.
Fourteen participants read 240 sentences while undergoing fMRI. Voxel-wise encoding models showed that InferSent predicted elements of neural activation that were not accounted for by bag-of-words approaches or by models that assemble word vectors using grammatical rules. Crucially, this predictive power appeared throughout a distributed cortical network, implying that sentence-level meaning is represented within and across multiple regions rather than residing in a single locus.
Follow-up analyses compared these results with other deep-network approaches (such as ELMo and BERT) and assessed residual neural signal using an experiential semantic model and cross-participant encoding strategies.
Significance Statement
A central challenge in cognitive neuroscience is explaining how the brain transforms sequences of words into coherent sentence meanings. Advances in neuroimaging, large-scale language models, and machine learning now allow researchers to model meaning computationally and test those models against brain activity. This study provides evidence that unified sentence-level information—rather than only word-level features—is encoded broadly across a cortical semantic network, demonstrating the value of deep recurrent models for illuminating how the brain processes complex language.