Summary: Researchers have developed a deep learning system that predicts, with high accuracy, how different regions of the human brain respond to individual words. The model also reveals that representations in auditory cortex depend less on surrounding context than representations in higher-level language regions.
Source: UT Austin, Texas Advanced Computing Center
Can artificial intelligence help us decode how the brain processes language, and can neuroscience explain why neural networks predict human perception so well?
Researchers Alexander Huth and Shailee Jain at The University of Texas at Austin (UT Austin) present evidence that both questions can be explored using modern deep learning techniques combined with brain imaging data.
In a study presented at the 2018 Conference on Neural Information Processing Systems (NeurIPS), the team reports experiments that use artificial neural networks to forecast, more accurately than previous models, how distinct cortical regions respond when people hear particular words.
“When words enter our minds, we rapidly form an understanding of the speaker’s meaning,” said Huth, assistant professor of Neuroscience and Computer Science at UT Austin. “Language processing in the brain likely follows structured principles, but its complexity makes it difficult to express as simple formulas.”
Their approach relies on a recurrent neural network architecture known as long short-term memory (LSTM), which retains information about preceding words so the model can use context to disambiguate meanings and better predict upcoming words.
“Many words are ambiguous on their own,” said Jain, a PhD student in Huth’s lab. “By incorporating prior words, the model infers the intended meaning in that sentence, and we expected that including context would improve predictions of brain activity because the brain itself is sensitive to context.”
Although intuitive, this perspective contrasts with decades of neuroscience studies that examined brain responses to isolated words without accounting for their role within sentences or narratives. Huth has emphasized the value of studying language in natural contexts in a March 2019 paper in the Journal of Cognitive Neuroscience.
To test their ideas, the researchers trained an LSTM language model to predict the next word in a sequence, a task similar to autocomplete systems. They used powerful computing resources at the Texas Advanced Computing Center (TACC) to process large amounts of text and fit the model.
Simultaneously, they analyzed fMRI (functional magnetic resonance imaging) recordings taken while subjects listened to spoken stories (from The Moth Radio Hour). fMRI measures changes in blood oxygenation linked to neural activity, providing a coarse map of where language-related representations occur across the cortex.
Combining the LSTM model’s representations with the fMRI data, the team trained a predictive system that could estimate the brain’s response to each word in a new story that neither the model nor the listeners had encountered before.
Prior work had already localized some language-related responses across the brain, but this study shows that including contextual information—up to twenty preceding words—substantially improves prediction accuracy. Even short context windows produced gains, and prediction quality increased as the model incorporated longer context spans.
“When the LSTM processes more words from the past, it becomes better at predicting the next word,” Jain explained. “That improvement indicates the model is drawing on information distributed across prior words.”
The results also reveal regional differences in how much context matters. Responses in primary auditory areas were largely driven by the acoustic identity of single words and showed little sensitivity to extended context. In contrast, higher-level language regions exhibited stronger dependence on longer contextual windows, aligning with theories that language comprehension is hierarchical and distributed.
“For example, hearing the word ‘dog’ will evoke a consistent response in auditory cortex regardless of the preceding sentence fragments,” Huth said. “Areas involved in more abstract interpretation, however, show clearer context-dependent patterns.”
The study found a notable correspondence between the hierarchical structure of the artificial LSTM network and the brain’s own hierarchy of language processing, suggesting that layered neural network models can capture meaningful aspects of cortical organization.
Natural language processing (NLP) has advanced rapidly, but tasks such as conversational understanding, question answering, and sentiment interpretation remain challenging. The researchers believe that language models built with LSTM architectures, informed by brain data, may help narrow these gaps.
In technical terms, LSTMs and other neural networks represent words as high-dimensional vectors that encode many relationships and features simultaneously. To train the language model, the team used tens of millions of words drawn from Reddit posts, then applied the model to predict responses for thousands of voxels in six participants’ brains as they listened to new stories. To explore how context length and internal network layers affected predictions, they evaluated multiple context lengths and layer choices, generating a large combinatorial set of model fits for each subject.
Running this scale of computation required extensive processing power, memory, and storage. TACC’s systems, including the Maverick supercomputer with GPU and CPU resources and the Corral storage platform, allowed the researchers to parallelize experiments and reduce what would have taken years into weeks.
“Training these models effectively demands large datasets and repeated passes through the data to update model parameters,” Huth said. “Without parallel infrastructure, this work would be prohibitively slow.”
Looking ahead, Huth and Jain are exploring an end-to-end strategy that trains a model to predict brain responses directly from text, rather than first learning a general language model and then mapping it onto brain data. In such an approach, prediction errors in brain activity would drive learning, potentially yielding models that more closely reflect how neural systems encode language.
“If successful, this line of work could produce systems that interpret text and speech in ways closer to human comprehension,” Huth said. “Imagine machine translation that captures meaning rather than relying solely on learned patterns.”
While mind-reading remains a long-term objective, the researchers’ combined use of deep learning and neuroimaging is already yielding valuable insights into both brain function and artificial intelligence.
“The brain is an efficient computational device, and AI aims to replicate many of the tasks a brain performs,” Jain said. “We use AI to probe brain mechanisms, and the resulting knowledge feeds back into better AI systems. The goal is to study biological and artificial cognitive systems together to advance our understanding and engineering of intelligent behavior.”
Source:
UT Austin, Texas Advanced Computing Center
Media Contacts:
Aaron Dubrow – UT Austin, Texas Advanced Computing Center
Image Source:
The image is credited to Huth lab, UT Austin.
Original Research: Closed access
“Are We Ready for Real-world Neuroscience?”
Pawel J. Matusz, Suzanne Dikker, Alexander G. Huth, and Catherine Perrodin. Journal of Cognitive Neuroscience 2019 31:3, 327-338 doi: 10.1162/jocn_e_01276
Abstract
Are We Ready for Real-world Neuroscience?
Real-world environments are dynamic, complex, and multisensory, requiring attention and memory mechanisms to support everyday tasks such as driving, shopping, or making coffee. Traditional laboratory paradigms using simplified stimuli established core principles of perception and brain organization, but recent advances in computation, brain mapping, and signal processing have made it possible to study cognition in more naturalistic settings. This emerging body of work examines how perception, attention, and functional brain organization operate in realistic contexts and highlights both shared themes and methodological differences across approaches. In this Special Focus, early-career researchers and speakers from a Cognitive Neuroscience Society symposium discuss how diverse “real-world neuroscience” methods contribute to building more accurate models of cognitive function and brain organization.