Summary: New research shows that visual cues—such as watching a speaker’s lips—help the brain amplify relevant sounds by synchronizing auditory processing with what we see.
Source: UCL
Watching someone’s lips improves listening in noisy places because visual timing cues help the brain boost the sounds that match what the eyes are seeing, according to a new UCL-led study.
Published in Neuron, the study suggests this multisensory interaction could have practical value for people who use hearing aids or cochlear implants and who typically find it difficult to follow conversations in noisy environments like restaurants or public venues.
The researchers demonstrate that visual and auditory information are combined at an earlier and more basic neural level than previously thought, working independently of conscious attention. When the timing of what we see and what we hear is coherent—when lip movements and speech amplitude change in sync—the auditory cortex emphasizes the sound that matches the visual signal. In other words, temporal alignment between sight and sound guides which auditory signals the brain represents most strongly.
“Although the auditory cortex is specialized for processing sound, a notable fraction of its neurons also respond to visual input—something we helped reveal about a decade ago,” said Dr. Jennifer Bizley of the UCL Ear Institute, lead author of the study. “Our latest work clarifies why: visual timing information tells auditory neurons which components of a complex acoustic scene are most relevant, helping listeners pick out one voice from many.”
Earlier work from the group showed that people can separate overlapping sounds more effectively when the target sound is temporally aligned with a visual cue. For this new study, researchers recorded neural activity in ferret auditory cortex while presenting pairs of sounds together with a visual stimulus. When one sound’s amplitude fluctuations were synchronized with changes in the visual stimulus’s luminance, a larger population of auditory cortical neurons responded to that sound, indicating an enhanced neural representation of the temporally matched audio stream.
Dr. Bizley added: “Watching a speaker does more than provide lip-reading cues. Our results show that simple timing alignment between visual motion and sound amplitude changes is enough to bias auditory neurons toward representing the matching sound. That low-level effect can make it easier to follow a single voice amid background noise.”
The authors note potential applications: auditory rehabilitation programs could train listeners to exploit natural audiovisual timing relationships, and assistive device designers might develop smarter signal-processing approaches that take gaze direction into account to enhance target speech. The research also reinforces a practical point for clinicians and patients: people who struggle to hear conversations should consider having their vision checked, because visual input plays an important role in everyday listening.
Funding: The study was led by Dr. Jennifer Bizley and PhD student Huriye Atilgan (UCL Ear Institute) with collaborators from UCL, the University of Rochester, and the University of Washington. Funders included Wellcome, the Royal Society, the Biotechnology and Biological Sciences Research Council, Action on Hearing Loss, the U.S. National Institutes of Health, and the Hearing Health Foundation.
Source: Chris Lane – UCL
Publisher: NeuroscienceNews.com (organized coverage)
Image Source: Image adapted from the UCL news release.
Original research: Open-access research article in the journal Neuron (doi: 10.1016/j.neuron.2017.12.034).
MLA: UCL. “Visual Cues Amplify Sound.” NeuroscienceNews, 15 February 2018.
APA: UCL (2018, February 15). Visual Cues Amplify Sound. NeuroscienceNews.
Chicago: UCL. “Visual Cues Amplify Sound.” NeuroscienceNews (accessed February 15, 2018).
Abstract
Integration of Visual Information in Auditory Cortex Promotes Auditory Scene Analysis through Multisensory Binding
Key findings:
- Visual stimuli can change how auditory cortical neurons respond to mixtures of sounds.
- Temporal coherence between vision and hearing enhances the neural representation of the sound that is bound to the visual signal.
- Visual input produces measurable shifts in the phase of local field potentials in the auditory cortex, indicating a physiological mechanism for cross-sensory influence.
- When visual cortex activity is reversibly silenced, those vision-induced phase effects in auditory cortex disappear, demonstrating the visual cortex as a source of the modulatory signal.
Summary of the study:
Understanding how the brain binds auditory and visual signals to form coherent multisensory objects is essential for explaining everyday perception. One proposed mechanism is temporal coherence: when dynamic signals from different senses change together in time, the brain treats those features as belonging to the same object. The study shows that when a visual stimulus’s luminance fluctuations are temporally coherent with the amplitude fluctuations of a particular sound within an acoustic mixture, the auditory cortex enhances its representation of that sound. This enhancement affects both the features directly involved in binding and other associated sound features. The results indicate that visual information, conveyed via phase changes in local field potentials originating in visual cortex, is combined with auditory input within auditory cortex. These findings support an early, bottom-up multisensory binding mechanism that aids auditory scene analysis by promoting the formation of cross-sensory objects.