Summary: A new study demonstrates that artificial intelligence systems are more vulnerable to adversarial attacks than previously recognized, exposing them to manipulations that can produce incorrect or dangerous decisions.
Researchers show that adversarial weaknesses are widespread across modern deep neural networks, raising alarm about deploying these systems in safety-critical settings. To evaluate these risks, the team created QuadAttacK, a testing tool that probes neural networks for susceptibility to adversarial manipulation.
The results underscore the urgent need to improve AI robustness, especially in domains where failures can affect human safety, such as autonomous vehicles and medical imaging.
Key Facts:
- Adversarial attacks are deliberate modifications to input data that cause AI systems to misinterpret or misclassify that data, which can produce harmful outcomes.
- QuadAttacK is a software framework designed to learn how a neural network makes decisions and to generate targeted manipulations that reveal vulnerabilities.
- Commonly used deep neural networks, including convolutional networks and vision transformers, were found to be broadly vulnerable in proof-of-concept tests, highlighting the need for improved defenses.
Source: North Carolina State University
Artificial intelligence systems promise major benefits—from self-driving cars to automated medical image interpretation—but a recent study finds these systems are more easily manipulated than many developers expect.
The study focuses on adversarial attacks: intentional, often subtle changes to input data designed to mislead AI models. In practical terms, an attacker could place a small sticker on a stop sign that causes a vision system to ignore or mislabel it, or they could alter medical images so that an AI diagnostic tool provides an incorrect reading.
“In many cases, you can change a stop sign in many ways and a well-trained AI will still recognize it,” says Tianfu Wu, co-author of the study and an associate professor of electrical and computer engineering at North Carolina State University. “But when a model contains a vulnerability and an adversary knows how to exploit it, the consequences can be catastrophic.”
To measure how prevalent these vulnerabilities are, the research team developed QuadAttacK, software that examines a trained neural network’s decision-making patterns and then creates input perturbations designed to mislead the model.
QuadAttacK observes the model’s responses to clean data to infer how internal decisions map to inputs. It then formulates and sends manipulated inputs to the model to test whether it can be driven to produce specific, incorrect outputs. In successful cases, the tool can cause a network to interpret an object as virtually anything the tester chooses—turning a stop sign into a mailbox, a speed-limit sign, or even a green light—by applying carefully designed changes.
For a proof-of-concept evaluation, the team used QuadAttacK against four widely used architectures: two convolutional neural networks (ResNet-50 and DenseNet-121) and two vision transformer models (ViT-B and DeiT-S). All four models showed significant vulnerability, and the researchers were able to fine-tune attacks to make the networks consistently produce targeted misclassifications.
“We were surprised by both the prevalence and the controllability of these attacks,” Wu explains. “That precision matters a great deal—if attackers can reliably force specific misinterpretations, the risk to real-world systems escalates.”
To support wider evaluation and defensive research, the authors have made QuadAttacK available to the research community as a tool for testing neural networks for adversarial weaknesses. Their goal is to encourage robust testing practices and to accelerate development of effective countermeasures.
“Identifying these vulnerabilities is a necessary first step,” Wu says. “The next priority is developing and validating strategies to reduce or eliminate these attack surfaces. We already have candidate defenses, but more work is needed to confirm their effectiveness in practical settings.”
The paper titled “QuadAttacK: A Quadratic Programming Approach to Learning Ordered Top-K Adversarial Attacks” was scheduled for presentation on Dec. 16 at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023) in New Orleans. The lead author is Thomas Paniagua, a Ph.D. student at NC State; co-authors include Ryan Grainger and Tianfu Wu.
Funding: This research received support from the U.S. Army Research Office (grants W911NF1810295 and W911NF2210010) and the National Science Foundation (grants 1909644, 2024688 and 2013451).
About this artificial intelligence research news
Author: Matt Shipman
Source: North Carolina State University
Contact: Matt Shipman – North Carolina State University
Image: The image is credited to Neuroscience News
Original Research: The findings were presented at the Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)