Researchers using only fMRI data have reconstructed images of human faces as they were seen by other people, a study led by a Yale undergraduate shows.
Note: See the disclaimer below the article about the research status of this work.
Marvin Chun, professor of psychology, cognitive science and neurobiology and a co-author on the study published in the journal Neuroimage, described the work as “a form of mind reading.” The team used functional magnetic resonance imaging (fMRI) — a noninvasive brain-imaging technique that detects changes in blood flow related to neural activity — to infer visual information about faces viewed by study participants.
Previous advances in fMRI analysis have enabled researchers to discriminate broad categories of viewed scenes — for example, whether a participant was looking at a beach versus a city, or an animal versus a building. Those approaches typically identify the category of what is seen, not the specific object. The Yale team’s report represents a further step: reconstructing the visual appearance of individual faces from brain activity patterns.
The project began as an experiment by Alan S. Cowen, then a Yale junior and the study’s lead, who is now pursuing further training at the University of California, Berkeley. Cowen set out to determine whether the distributed and overlapping brain responses evoked by faces could be translated back into recognizable face images.
Because faces share many common features and because multiple brain regions contribute to face perception, the problem is more complex than reconstructing more distinct object categories. “We perceive faces in much greater detail than other objects,” Cowen explained, and that fine-grained perception is reflected across several cortical areas.
With support from the Yale Provost’s office, Cowen and collaborator Brice Kuhl, then a postdoctoral researcher and now an assistant professor at New York University, scanned six participants while they viewed a set of 300 different “training” faces. The researchers used those scans to build a statistical library linking patterns of fMRI activity to visual face features. Later, the same subjects viewed new faces while undergoing fMRI, and the team applied the library to reconstruct the faces that the subjects had just seen, relying solely on the recorded brain activity.
The results show that, using carefully designed statistical models and neural response patterns, it is possible to approximate the images seen by a viewer. Cowen notes that the accuracy of such reconstructions is likely to improve as imaging methods, analytical techniques and training datasets advance. He envisions potential research applications, such as exploring how individuals with autism spectrum disorder respond to faces and how face-processing differs across populations.
Chun also highlighted the value of supporting ambitious undergraduate research. Projects that are novel and exploratory may not fit conventional external funding routes, yet they can produce important scientific advances when given institutional backing.
Important disclaimer: The abstract page for the paper includes this notice: “Note to users: Uncorrected proofs are Articles in Press that have been copy edited and formatted, but have not been finalized yet. They still need to be proof-read and corrected by the author(s) and the text could still change before final publication.” This means the published version may differ slightly from the uncorrected proof.
Contact: Bill Hathaway – Yale
Source: Yale press release (reported by the university)
Image source: The image is credited to Alan Cowen and is adapted from the Yale press release.
Original research: Abstract for “Neural portraits of perception: Reconstructing face images from evoked brain activity” by Alan S. Cowen, Marvin M. Chun, and Brice A. Kuhl in Neuroimage. Published online March 17, 2014; doi:10.1016/j.neuroimage.2014.03.018 — In Press, Uncorrected Proof.