Summary: A new study finds that most people do not recognize racial bias embedded in artificial intelligence (AI) training data, even when the bias is directly visible. When emotion-recognition AI is trained on imbalanced datasets—such as disproportionately happy white faces and sad Black faces—the system learns to associate race with emotion, producing biased outcomes that favor one group over others.
Participants in the study rarely detected these dataset-driven biases unless they belonged to the group portrayed negatively. The findings underscore the need for greater public awareness, improved AI literacy, and clearer transparency about how algorithms are trained and evaluated.
Key Facts:
- Hidden bias: AI trained on racially skewed data misclassified emotions, often labeling white faces as happier than Black faces.
- Human blind spot: Most users failed to notice bias in the training datasets and trusted the AI to be neutral even when it was not.
- Group sensitivity: Members of negatively portrayed groups were more likely to suspect the bias, particularly when their group was overrepresented for negative emotions.
Source: Penn State
Overview: Emotion-recognition AI can reflect and amplify racial imbalances present in its training data. In the reported experiments, datasets containing a preponderance of happy white faces and sad Black faces taught the AI to link race and emotional expression. That association led to biased classification performance: the algorithm performed well for the dominant group but struggled with minority-group expressions.
The study, published in Media Psychology, asked lay users to review the training data and judge whether the AI treated racial groups equally. Researchers found that most people did not notice the confound between race and emotion in the datasets. Only participants from the groups portrayed negatively were consistently more likely to detect the problem.
The research team, which has been investigating algorithmic bias for several years, argues that AI systems should be trained to serve everyone and produce outcomes that are representative across demographic groups. That requires attention not only to the amount of data from each group but also to unintended correlations the model might learn from the examples it sees.
“In the case of this study, AI seems to have learned that race is an important criterion for determining whether a face is happy or sad,” said senior author S. Shyam Sundar, Evan Pugh University Professor and director of the Center for Socially Responsible Artificial Intelligence at Penn State. “Even though we don’t mean for it to learn that.”
The central question the researchers pursued was whether ordinary users can spot bias in training data before or without seeing biased AI behavior. Across three between-subjects online experiments with 769 participants in total, users were shown prototypes of an emotion-recognition system and the images used to train it. The experiments varied how race and emotion were presented in the training sets.
In one scenario, the training data mixed race and emotion so that happy faces were predominantly white and sad faces predominantly Black. In another, some categories contained only white faces, demonstrating under-representation of other racial groups. The final set of conditions combined these contrasts and included a control condition without any racial confound.
Participants were asked whether the AI treated all racial groups equally. Across the three experiments most respondents reported not noticing bias in the training data. When the system’s performance itself displayed racial disparities—accurately classifying emotions for white faces but not for Black faces—participants were more likely to perceive a problem. Notably, Black participants were more sensitive to biased representations that painted their group as predominantly unhappy.
Lead author Cheng “Chris” Chen, an assistant professor of emerging media and technology at Oregon State University, explained: “That is what we mean by biased performance in an AI system where the system favors the dominant group in its classification.” Chen noted that people tend to rely on observed outcomes; when an AI system shows biased performance, observers often base their judgments on that outcome rather than on the composition of the training data.
Sundar emphasized the psychological dimensions of the problem. “People often trust AI to be neutral, even when it isn’t,” he said, highlighting a cognitive gap that allows dataset confounds to go unnoticed unless the bias directly affects a person’s group.
The authors plan follow-up work to develop clearer ways to communicate dataset and algorithmic bias to users, designers, and policymakers. Their goal is to improve media and AI literacy so that developers and the public can better recognize when training data may produce unfair or unrepresentative outcomes.
Key Questions Answered:
A: Most people could not detect racial bias in AI systems trained on skewed datasets, demonstrating how subtle dataset-driven bias can be and how easily it is overlooked.
A: Models learn statistical associations from their training data. If the data over-represents certain emotions for a particular race—such as more happy white faces and more sad Black faces—the model can incorrectly infer that race predicts emotion.
A: It reveals that people often trust AI outputs without scrutinizing training data, which can allow harmful stereotypes and unequal treatment to persist. The result is a clear need for education, transparency, and better bias-detection practices.
About this AI research news
Author: Francisco Tutella
Source: Penn State
Contact: Francisco Tutella – Penn State
Image credit: Neuroscience News
Original Research: Closed access. “Racial Bias in AI Training Data: Do Laypersons Notice?” by S. Shyam Sundar et al., published in Media Psychology. DOI reference: 10.1080/15213269.2025.2558036
Abstract
Racial Bias in AI Training Data: Do Laypersons Notice?
Training data shape algorithmic behavior, yet the public’s ability to detect dataset-driven bias remains unclear. To investigate whether laypersons recognize that systematic misrepresentation and under-representation of racial groups in training data can produce biased AI performance, the authors conducted three online, between-subjects experiments with a prototype emotion-recognition system (N = 769).
Results indicate that the representativeness of training data is not an effective cue for most users to detect algorithmic bias. Instead, observers tend to rely on the AI’s visible performance bias to perceive unfairness. The race of the user also influenced detection: Black participants were more likely to perceive bias when unhappy expressions in the training data were predominantly images of Black individuals. These findings point to a human cognitive limitation that should be considered when designing communications and interventions to expose and correct bias arising from training datasets.