Summary: Watching a film often feels effortless: we follow speech, read facial expressions, notice musical cues, and piece together quick scene changes into a coherent story. Underneath this smooth experience, the brain constantly and rapidly decides which sensory input—sound or sight—should take priority at each moment.
A recent study shows the frontal cortex functions like a traffic controller for attention, dynamically reallocating processing resources between auditory and visual streams as unpredictable events unfold on screen.
Key Facts
- Frontal cortex as controller: Rather than treating sensory input as a single mixed signal, the frontal cortex actively filters and prioritizes which modality—audio or visual—should guide perception before a scene reaches conscious awareness.
- Distinct anatomical segregation: Direct neural recordings reveal a reproducible spatial map within the frontal lobe: ventral (lower) frontal regions preferentially engage with auditory information, while dorsal (upper) frontal regions are more responsive to visual input.
- Language-driven switching: During scenes in native English, the brain weighted auditory processing more heavily. The instant the film switched to an unfamiliar language, neural activity shifted toward visual centers to capture subtitles, gestures, and facial cues.
- Behavioral validation: Independent online participants rated whether audio or visual information mattered more in specific clips, and their judgments matched the millisecond-by-millisecond neural shifts observed in patients with intracranial electrodes.
- Clinical and technological promise: Mapping how healthy brains reassign sensory attention suggests new directions for therapies targeting attention disorders, autism, language impairments, and hearing loss, and offers design principles for context-adaptive artificial intelligence.
Source: NYU
Most people find movie-watching effortless. We follow dialogue, read expressions, detect music cues, and fuse these streams into a single narrative. Yet the brain is continually making split-second choices about which sensory stream should dominate.
Published in Nature Communications, the new study implicates the frontal cortex—a region tied to planning and higher cognition—as a dynamic allocator of attention between auditory and visual inputs during natural viewing.
Researchers recorded activity directly from 19 patients with epilepsy who had temporary intracranial electrodes implanted for clinical monitoring. While hospitalized, participants watched a 12-minute short film containing scenes in English, Greek, German, and French. Some foreign-language segments included English subtitles, creating natural shifts in audiovisual demands. Intracranial recordings allowed the team to measure neural responses with millisecond precision—temporal resolution beyond what conventional fMRI can provide.
Results showed that the frontal cortex does not uniformly handle all incoming sensory information. Instead, it contains a clear, modality-specific gradient: ventral frontal regions were more responsive to auditory cues, whereas dorsal frontal regions showed stronger responses to visual features.
“This suggests the frontal cortex is organized to handle different types of information during real-world experiences,” said Faxin Zhou, a Ph.D. candidate in the NYU Tandon Biomedical Engineering Department and the study’s first author. “It is not simply a general control hub; it separates sound and sight in a structured way.”
The pattern changed dynamically with language context. During English-language segments, when speech was comprehensible, frontal activity favored auditory processing. When dialogue switched to an unfamiliar language, activity shifted toward visual regions, indicating reliance on facial expressions, gestures, and subtitles to understand the plot.
To test whether those neural shifts reflected perceived importance, the team asked online volunteers to rate which moments in the film were most informative and whether audio or visual cues were more useful. Those behavioral ratings aligned closely with the neural data: comprehensible speech increased auditory weighting, while foreign-language scenes increased visual weighting. In short, the brain reweights sensory priorities on the fly.
“When speech comprehension becomes difficult, the brain reallocates processing toward visual signals,” said Adeen Flinker, Associate Professor of Biomedical Engineering at NYU Tandon and Neurology at NYU Grossman School of Medicine. “That flexibility is likely essential for navigating complex environments filled with competing information.”
This work addresses a major question in neuroscience: how the brain integrates multiple senses in realistic settings. Much prior research used simplified tasks; by contrast, movies provide continuous, unpredictable multisensory input that more closely mirrors everyday life. The findings suggest the frontal cortex may actively decide which sensory stream gets priority before conscious perception occurs, rather than merely merging processed inputs after the fact.
Practical implications include potential new strategies for therapies that help people with attention deficits, autism spectrum conditions, language-processing disorders, or hearing impairment. The study’s insights may also inform the design of adaptive AI systems that allocate computational resources between audio and visual processing depending on context.
Limitations are important to note. Participants were clinical patients with epilepsy, so results may not fully generalize to the broader population. Electrode placement reflected clinical needs rather than experimental design, leaving some frontal regions less sampled. Nonetheless, direct intracranial recordings provide rare temporal detail about how human perception is shaped in real time.
Key Questions Answered:
A: Conventional tools like fMRI capture hemodynamic changes over seconds, which blurs rapid perceptual shifts that happen on millisecond timescales during a movie. Studying patients with surgically implanted intracranial electrodes allowed researchers to record electrical activity directly from the brain with millisecond precision, revealing rapid neural pivots as viewers switched focus from spoken words to subtitles or visual cues.
A: Traditional models often portrayed the frontal cortex as a generalized executive center that operates after sensory information is processed elsewhere. This study indicates a more nuanced picture: the frontal cortex contains an organized, modality-specific map that actively assigns priority to auditory or visual streams, intervening before conscious perception is formed.
A: Current multimodal AI systems frequently process audio, video, and text equally, which can be computationally costly. Emulating the frontal cortex’s dynamic, modality-specific allocation could lead to AI architectures that scale processing resources up or down by modality—focusing on audio when speech is clear and switching to higher-resolution visual processing when audio comprehension drops.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- Journal paper reviewed in full.
- Additional context added by our staff.
About this auditory and visual neuroscience research news
Author: Leah Schmerl
Source: NYU
Contact: Leah Schmerl – NYU
Image: The image is credited to Neuroscience News
Original Research: Open access.
“Frontal cortex organization supporting audiovisual processing during naturalistic viewing” by Faxin Zhou, Amirhossein Khalilian-Gourtani, Patricia Dugan, Andrew Michalak, Orrin Devinsky, Peter Rozman, Werner Doyle, Daniel Friedman & Adeen Flinker. Nature Communications
DOI: 10.1038/s41467-026-73947-8
Abstract
Frontal cortex organization supporting audiovisual processing during naturalistic viewing
The brain flexibly adapts to a multisensory world by coordinating inputs across sensory streams. How audiovisual information is represented and changes over time in natural settings remains unclear.
Using a movie-viewing paradigm, the study recorded intracranial electrocorticography (iEEG) from 19 participants watching a short multilingual film. Through unsupervised clustering and supervised encoding models, researchers identified a robust modality-specific gradient in the frontal cortex: ventral areas predominantly process auditory information, while dorsal areas favor visual inputs.
This cortical organization shifted dynamically with movie context, reflecting flexible allocation of audiovisual resources to construct a coherent percept. Behavioral ratings supported the idea that the frontal cortex plays a primary role in this modality-assignment process. Together, these findings clarify the frontal cortex’s functional architecture for flexible multisensory representation and integration in natural contexts.