Summary: Capturing the full complexity of human hand movement — which depends on dozens of muscles, many joints, and numerous tendons — has long been a major challenge for robotics, prosthetics, and virtual reality. Engineers at MIT have developed a wearable ultrasound wristband that images the tendons and muscles inside the wrist and uses artificial intelligence to decode those images into real-time, high-fidelity hand motion tracking.
By treating wrist tendons like the strings of a puppet and continuously imaging their motion, the device maps internal anatomical changes to precise positions of the five fingers and the palm. The system enables continuous, fluid control of robotic hands and virtual avatars rather than coarse, discrete gestures. Demonstrations include a user operating a robotic hand to play a short piano tune, shooting a miniature basketball with a robotic controller, and manipulating virtual objects with a pinch gesture.
Key Facts
- The “Puppet String” Principle: Instead of external cameras or sensor-laden gloves, the band images tendons and muscles beneath the skin. Changes in those tissues reveal finger and palm positions.
- AI Decoding: The team trained an AI model to map ultrasound image patterns to all 22 degrees of freedom of human finger and thumb motion, enabling continuous tracking rather than binary or discrete events.
- Real-Time Performance: The integrated system tracks gestures with low latency, allowing wireless control of a robotic hand to play a tune and perform game-like tasks.
- Scalable Dataset: Researchers are assembling a large, diverse dataset of hand movements to improve generalization across hand sizes, shapes, and tasks and to support training dexterous humanoid robots.
Source: MIT
Our hands perform remarkably intricate actions every day, relying on dozens of muscles, many joints, and over a hundred tendons and ligaments. Reproducing that dexterity in machines or accurately capturing it for augmented and virtual reality has been difficult because existing approaches often sacrifice natural motion, sensitivity, or robustness.
The MIT ultrasound wristband produces continuous ultrasound images of muscles, tendons, and ligaments in the wrist while the hand moves. An onboard processing pipeline and an AI model translate those images into the corresponding kinematic poses of the fingers and palm in real time, enabling intuitive control of external devices or virtual interfaces.
The wristband can be trained to recognize an individual wearer’s motion patterns and to communicate those motions wirelessly to robots or virtual environments. In tests, a user controlled a robotic hand to press piano keys and to tap a desktop basketball hoop, and used the same gestures to manipulate 3D objects on a computer screen with smooth zoom and grasp interactions.
Beyond consumer applications, the researchers plan to expand their database of hand motions across diverse users to enable more generalized AI models. Such models could help train humanoid robots for tasks that require high dexterity, including delicate assembly or certain surgical manipulations, and could replace bulky controllers in AR/VR systems for more natural, hands-on interaction.
“We think this wearable ultrasound approach can immediately improve hand tracking for virtual and augmented reality, and it can also provide valuable training data for dexterous humanoid robots,” says Xuanhe Zhao, Uncas and Helen Whitaker Professor of Mechanical Engineering at MIT.
Zhao, Gengxi Lu, and colleagues describe the wristband and algorithms in a paper published in Nature Electronics. The project team includes former postdoctoral researchers, graduate students, and collaborators from the University of Southern California.
Seeing strings
Existing hand-tracking approaches have trade-offs. Camera-based tracking requires clear sightlines and complex setups. Sensor gloves can be bulky and reduce tactile feedback. Electromyography (EMG) reads electrical signals from muscles but is susceptible to noise and often cannot resolve continuous, fine-grained motion paths between two poses.
Ultrasound imaging captures the internal mechanics of tendons and muscles beneath the skin and is therefore less affected by visual occlusion and more sensitive to subtle changes. The research team adapted compact ultrasound transducers—similar in size to a smartwatch face—and integrated them into a wearable band with compact electronics to create a practical, continuous imaging device for the wrist.
Mapping manipulation
The wristband produces grayscale ultrasound frames that show distinct regions corresponding to different tendons and muscle groups. The research showed that specific regions in those images consistently correlate with particular degrees of freedom of the hand. For example, changes in one localized region indicate thumb extension while changes in another correspond to index finger movement.
To create labeled training data, volunteers wore the wristband while moving their hands through many poses; external cameras recorded the ground-truth hand positions. The team then annotated ultrasound image regions with corresponding hand degrees of freedom. Training a deep learning model on these labeled pairs allowed the AI to predict continuous hand poses directly from ultrasound frames without camera input.
In tests with eight volunteers of varying hand and wrist sizes, the wristband accurately tracked gestures, grasps, and a full set of American Sign Language letters. The system also handled everyday objects such as a tennis ball, a water bottle, scissors, and a pencil, reliably predicting hand configuration during manipulation.
The researchers are working to miniaturize hardware further and to train models on a broader range of users and gestures so that the wristband can become a general-purpose, wearable hand tracker for controlling robots, interacting with virtual environments, and advancing dexterous robotic learning.
Key Questions Answered:
A: Cameras require line of sight and complex setups; gloves can limit natural tactile feedback. Ultrasound images internal tendon and muscle motion through the skin, so it avoids visual obstructions and captures subtle, continuous finger motions that EMG often cannot resolve.
A: Yes. The wristband can support intuitive mid-air gestures—like pinching to zoom or grasping virtual tools—and its small form factor makes it a promising replacement for bulky AR/VR controllers in gaming, design, and other interactive applications.
A: The team trained the AI using ground-truth labels from external cameras. By correlating camera-recorded hand poses with ultrasound image patterns, the model learned to predict finger and palm configurations from internal wrist images alone.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- The journal paper was reviewed in full by the editorial team.
- Additional context was added by staff to clarify applications and limitations.
About this robotics and neurotech research news
Author: Abby Abazorius
Source: MIT
Contact: Abby Abazorius – MIT
Image credit: Melanie Gonick, MIT
Original Research: Closed access. Paper: “Dexterous hand tracking via wearable wrist imaging” by Gengxi Lu, SeongHyeon Kim, Xiaoyu Chen, Yushun Zeng, Dian Li, Shu Wang, Baoqiang Liu, Shucong Li, Runze Li, Bolei Deng, Junhang Zhang, Chen Gong, Anantha P. Chandrakasan, Qifa Zhou & Xuanhe Zhao. Nature Electronics. DOI: 10.1038/s41928-026-01594-4
Abstract
Dexterous hand tracking via wearable wrist imaging
The human hand is highly dexterous, enabling rich interaction with both physical and virtual environments. Accurate, continuous tracking of hand motion can accelerate advances in spatial computing, virtual and augmented reality, robotics, and prosthetics. Existing methods—camera tracking, strain and inertial sensors, and electromyography—each have limitations in coverage, natural interaction, or sensitivity to continuous motion.
This work reports a fully integrated, wireless, wearable ultrasound imaging wristband combined with an artificial intelligence algorithm. The wristband continuously tracks arbitrary configurations of five fingers and the palm in real time with low latency and demonstrates intuitive control for virtual-reality tasks and robotic-hand applications.