Summary: A new study tested whether large language models (LLMs) can account for human reading dynamics. Researchers compared eye-tracking measurements from 368 adult readers with predictions from more than 400 neural language models, examining how both humans and models process syntactically challenging text, including so-called “garden-path” sentences.
The results reveal a clear split between human cognitive processes and transformer-based AI architectures. LLMs that predict the next word in a sequence capture the earliest stage of reading—how quickly a reader recognizes a word during fluent, forward eye movements—but they fail to predict the additional processing time humans need when integrating words into a broader sentence context. This mismatch is especially pronounced in sentences that lead readers toward an initial, incorrect interpretation.
Key Facts
- Early agreement, later divergence: LLM surprisal values align with human processing during initial visual word recognition in straightforward, linear reading, but do not explain the later integration difficulties readers experience with complex or ambiguous constructions.
- Underestimating integration difficulty: Next-word predictability from AI models dramatically underestimates how long people spend resolving ambiguous or garden-path sentences (for example, “The old man the boat”).
- Regressions are important: Regressive eye movements—backward saccades that revisit earlier words—make up roughly 20% of human reading fixations. Standard autoregressive LLMs lack an analogous mechanism for re-parsing earlier material.
- Large-scale benchmarking: The study compared human eye-tracking metrics against predictions from over 400 LLM variants to identify where modern predictive models succeed and where they fall short of human cognitive behavior.
- Implications for cognitive AI and reading science: While predictive, autoregressive mechanisms help explain routine word recognition, the findings point to the need for multi-stage integration mechanisms in computational models to capture human-like comprehension and reanalysis during reading.
Source: NYU
Why do we sometimes breeze through text and sometimes have to reread it?
A team of linguists and data scientists from New York University and the University of Massachusetts Amherst explored that question by comparing eye-tracking data with model-based predictions. They found that the very earliest moments of reading—when the brain recognizes a printed word—appear to rely on next-word prediction processes similar to those used by LLMs. But as sentences unfold and syntactic complexity increases, human readers often engage in re-evaluation and re-parsing that standard predictive models do not capture.
“Language models acquire strong language capabilities by training to predict the next word, which motivated us to ask whether the same predictive processes could explain how humans comprehend sentences,” said William Timkey, a linguistics doctoral student at NYU and lead author of the paper published in Proceedings of the National Academy of Sciences (PNAS).
Timkey and colleagues found that while LLMs match human reading behavior during smooth forward reading—explaining how long visual word recognition typically takes—they do not account for the processing delays that occur when readers hit a point of structural confusion and must revise their interpretation, a process often accompanied by rereading.
The authors emphasize that these findings do not render LLMs irrelevant to cognitive science. Rather, they show where predictive mechanisms are useful and where additional modeling components are required to explain the full range of human sentence processing.
“We now better understand how humans and models differ,” said Brian Dillon, professor of linguistics at UMass Amherst and senior author. “Recognizing those differences is the first step toward building models that more closely reflect human cognition, which could bring practical benefits for language education and remediation of reading difficulties.”
Tal Linzen, associate professor of linguistics and data science at NYU and a coauthor, added: “AI is a valuable tool for cognitive science, but it is not sufficient. For example, roughly 20% of our eye movements during reading are regressions, and standard LLM architectures offer no account of when and why we perform those backward looks. Developing computational models that incorporate re-parsing or multi-stage integration will be necessary to match human behavior more closely.”
Reading involves at least two core steps: extracting the visual form of a word and then integrating that word into an evolving representation of sentence meaning. The study used the similarity between LLM next-word prediction and theories of human prediction to probe the limits of this account.
To investigate, the researchers recorded eye movements from 368 adult readers while they read specially designed sentences, including many garden-path constructions—grammatically correct sentences that initially lead the reader to an incorrect interpretation. A classic example is “The old man the boat,” where readers often first parse “old man” as a noun phrase and must then revise to interpret “man” as a verb.
The study then compared those eye-tracking patterns with surprisal estimates from 409 different model variants across multiple architectures and training regimes. The comparison showed that model surprisal explains early effects of syntactic disambiguation—essentially routine, expectation-driven processing—but fails to predict the additional time and rereading behavior associated with detecting and repairing structural interpretation errors.
“Word predictability simply doesn’t come close to explaining how much time people spend on difficult words and garden-path sentences,” said Timkey. “The models drastically underpredict the human experience of difficulty in these cases.”
“This second stage—integrating a recognized word into the sentence-level representation and, when needed, revising that representation—is where current predictive models show the largest gaps,” Linzen added.
Funding: This research was supported by grants from the National Science Foundation (BCS-2020914, BCS-2020945, IIS-2504953, and IIS-2504954).
Key Questions Answered
A: Garden-path sentences are grammatically correct sentences designed so a reader’s initial parse is likely to be wrong (for example, “The old man the boat”). They provoke re-evaluation and rereading, making them useful for studying how the brain detects and repairs structural misunderstandings.
Q: Where do LLMs and human readers align?
A: LLMs and human readers align during the first stage of processing—visual word recognition. In straightforward contexts where eyes move smoothly forward, the statistical predictability of a word as measured by model surprisal correlates with how quickly humans recognize that word.
Q: Why can’t current LLMs account for backward eye movements?
A: Most LLMs operate via left-to-right next-word prediction and do not implement explicit mechanisms for revisiting and re-integrating earlier parts of the sentence. Human readers make backward eye movements about 20% of the time to re-examine prior words when structure-building fails; this re-parsing behavior is not reflected in standard autoregressive architectures.
Editorial Notes
- This article was edited by a Neuroscience News editor.
- The journal paper was reviewed in full.
- Additional context was added by staff.
About this AI research news
Author: James Devitt
Source: NYU
Contact: James Devitt – NYU
Image: The image is credited to Neuroscience News
Original Research: Open access. “Eye movements reveal a dissociation between prediction and structural processing difficulty in language comprehension” by William Timkey, Kuan-Jung Huang, Byung-Doh Oh, Grusha Prasad, Suhas Arehalli, Tal Linzen, Brian Dillon. PNAS
DOI: 10.1073/pnas.2532230123
Abstract
Eye movements reveal a dissociation between prediction and structural processing difficulty in language comprehension
When we extract meaning from text, our eyes dwell much longer on some words and often revisit earlier portions of a sentence. These disruptions are especially common in syntactically ambiguous constructions. One hypothesis attributes that difficulty to word predictability (measured as surprisal). Another attributes it to failures in the structure-building process. Prior work has been limited by small samples, coarse measures, and narrow ranges of surprisal estimates.
This large-scale study of 368 readers examined eye movements during reading of syntactically challenging sentences and evaluated 409 surprisal estimates from language models across architectures and training conditions. The results show a sharp dissociation: language model surprisal approximates early structure-building effects, but syntactic disambiguation imposes an additional processing cost—reflected in increased rereading—that surprisal does not capture. We conclude that surprisal accounts for routine structure-building but not for the cost of detecting and repairing structural errors during comprehension.