Summary: Researchers have developed a deep-learning system that interprets viewers’ facial expressions to measure and predict audience reactions to films. With sufficient data, the method can infer how an audience responds and anticipate an individual’s reaction after only a few minutes of observation.
Source: Caltech.
Software learns meaningful patterns in facial expressions automatically
Engineers and computer scientists have created a deep-learning system that interprets complex audience responses to movies by analyzing viewers’ facial expressions. The project was led by researchers at Disney Research in collaboration with Yisong Yue of Caltech and colleagues at Simon Fraser University. The approach centers on a new algorithm called factorized variational autoencoders (FVAEs).
Variational autoencoders are a class of deep neural networks that convert images of complex objects—such as faces—into compact numerical representations, often called latent encodings. The contribution from Yue and collaborators was to train these autoencoders so that they incorporate relevant metadata about the images. By using metadata to structure the encoding space, the model learns a factorized representation in which separate latent dimensions capture distinct aspects of the observed data.
Applied to faces watching a film, the factorized variational autoencoder transforms each frame into a set of numbers that represent interpretable facial features: for example, the degree of smiling, eye openness, or head pose. Metadata lets the algorithm relate those numeric features across time and across individuals—for instance, linking images of the same person at successive moments or different people at the same moment of a movie.
According to Disney research scientist Peter Carr, when provided with sufficient data the system can estimate how an audience reacts to a film so reliably that it can predict an individual viewer’s responses after only a few minutes of observation. Yisong Yue emphasizes that the technique has applications beyond movie screening rooms. Understanding subtle, nonverbal cues is crucial for AI systems that must interpret human behavior in realistic contexts—examples include monitoring or assisting elderly people, where body language or facial cues may indicate discomfort or distress that is not explicitly spoken.
The team presented these results at the IEEE Conference on Computer Vision and Pattern Recognition on July 22 in Honolulu.
“We are surrounded by data, so it is vital to have methods that find meaningful structure automatically,” says Markus Gross, vice president for research at Disney Research. “Our work shows that deep-learning approaches—powered by neural networks—can compress large datasets while preserving their underlying patterns.”
To build and test the method, the researchers analyzed 150 screenings of nine feature films, including titles such as Big Hero 6, The Jungle Book, and Star Wars: The Force Awakens. They instrumented a 400-seat theater with four infrared cameras to capture audience faces in the dark. The resulting dataset includes 68 facial landmarks per face for a total of 3,179 audience members, recorded at roughly two frames per second—amounting to about 16 million face images.
“That is far more data than a person could manually review,” Carr notes. “Computational models are essential to summarize and extract the important trends without discarding the signal.”
The FVAEs are not restricted to facial data. Any time-series dataset gathered from multiple objects or agents can be modeled with the same approach. Yue points out that once a model is learned, it can also generate realistic synthetic sequences. As an illustration, a similar method could model how different tree species respond to varying wind conditions and then generate plausible animated simulations of a forest responding to wind.
The research team included Zhiwei Deng, Rajitha Navarathna, Peter Carr, Stephan Mandt, Yisong Yue, Iain Matthews, and Greg Mori. The study, titled “Factorized Variational Autoencoders for Modeling Audience Reactions to Movies,” was supported by Disney Research. The findings were presented at the IEEE Conference on Computer Vision and Pattern Recognition on July 22 in Honolulu.
Keywords: factorized variational autoencoders, FVAE, audience reactions, facial expressions, deep learning, Disney Research, Caltech, time-series modeling, latent representation.
Abstract
Factorized Variational Autoencoders for Modeling Audience Reactions to Movies
Matrix and tensor factorization techniques are commonly used to recover low-dimensional structure from noisy data. In this work, we explore non-linear tensor factorization based on deep variational autoencoders. Our FVAE approach is well suited to situations where the mapping between latent representations and raw observations is highly non-linear. We apply the method to a large dataset of facial expressions recorded from movie audiences—over 16 million face images. Experiments demonstrate that, compared to classical linear factorization methods, the FVAE gives better reconstructions of the data and discovers latent factors that are more interpretable.