Summary: Researchers have developed an AI system that reconstructs detailed hand muscle activity from standard video footage. Where previous methods required electrodes attached to the skin, this new approach extracts internal muscle signals non-invasively using only recorded video and related performance cues.
Trained on a high-precision dataset of expert pianists, the model infers fine-grained muscle activation patterns with strong accuracy. The advance promises affordable, remote assessment of fine motor control for healthcare, rehabilitation, sports science, and human–machine interaction.
Key Facts:
- Sensor-free muscle tracking: The system estimates hidden hand muscle activity from video alone, eliminating the need for EMG electrodes.
- High accuracy across tasks: The model predicts both the timing and amplitude of muscle activation reliably, even for performers and pieces it did not see during training.
- Wide applicability: Potential uses include rehabilitation monitoring, sports performance analysis, robotics, and gesture-based interfaces.
Source: Institute of Science Tokyo
Fine hand control and the limits of traditional measurement
Precise hand movements, such as those required for piano performance, rely on coordinated activation of small muscles beneath the skin. Capturing those muscle signals has typically required electromyography (EMG) sensors, which are costly, intrusive, and technically demanding to deploy outside specialized labs.
A team led by Professor Hideki Koike, Department of Computer Science, School of Computing, Institute of Science Tokyo, together with Dr. Shinichi Furuya of Sony Computer Science Laboratories, developed a novel AI framework that overcomes these constraints.
Their method, Piano Keystroke-Pose-Muscle Network (PianoKPM Net), infers miniature hand muscle activity using only video recordings combined with keystroke data when available. The work was published online on September 19, 2025, and is scheduled for presentation at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025) in San Diego on December 2, 2025.
Central to this advance is a new dataset, PianoKPM, which documents how professional pianists move and control their hands with exceptional precision. The dataset comprises 12.6 hours of synchronized recordings from 20 expert pianists performing seven distinct musical tasks. Each performance includes multi-view video at 60 frames per second, reconstructed 3D hand poses, 1 kHz keystroke timing, audio, and high-sampling-rate EMG signals (2 kHz) from six small hand muscles.
Altogether, the dataset contains over five million pose frames and 28 million EMG samples, forming the first detailed paired collection linking visible motion and internal muscle activity at this scale. This rich mapping enables the AI to learn associations between external motion cues and the underlying physiological signals.
PianoKPM Net uses pose estimates and keystroke timing to reconstruct the timing and strength of muscle activations. In head-to-head comparisons with strong baseline models such as NeuroPose and CodeTalker, PianoKPM Net showed superior performance in predicting both muscle activation amplitude and temporal patterns. The model retained strong predictive quality even when evaluated on different pianists and unseen musical pieces, demonstrating robustness and generalization.
By turning a simple camera into a non-invasive sensor for muscle coordination, this approach enables detailed physiological study without attaching electrodes. That reduction in cost and discomfort makes continuous or remote monitoring of fine motor behavior practically feasible outside specialized clinics.
Practical implications and future uses
The implications extend well beyond piano research. In sports science, video-based muscle estimates could guide training and prevent overuse injuries by revealing patterns of muscle exertion. In rehabilitation, clinicians could track recovery progress remotely and continuously, using video feeds instead of repeated, invasive measurements. For robotics and human–computer interaction, understanding a user’s muscle effort can refine assistive control strategies and make gesture interfaces more responsive and personalized.
Professor Koike notes that the PianoKPM Net and the PianoKPM dataset together provide an accessible foundation for studying internal physiological signals and muscle activity, supporting advances in human augmentation and refined human–machine interfaces.
The research team plans to release both the dataset and the trained model publicly, enabling other researchers and developers to build on the work. Open release will support standardized benchmarks for motion-to-muscle estimation and accelerate progress in motor learning research, embodied intelligence, and assistive robotics.
Linking visual observation and physiological measurement, PianoKPM Net replaces complex EMG setups with a scalable, camera-based approach. This paves the way for new studies in performance science, clinical evaluation, and design of human-centered technologies, and it could enable remote skill training over low-latency networks even where specialized biological measurement equipment is unavailable.
Key Questions Answered:
A: It estimated subtle hand muscle activity accurately from ordinary video recordings, without requiring physical EMG sensors.
A: The model outperformed existing deep-learning methods in predicting both the timing and strength of muscle activations in comparative tests.
A: It provides a low-cost, non-invasive alternative to traditional EMG sensors, enabling broader and more comfortable measurement of fine motor activity.
Editorial Notes:
- This article was edited by a Neuroscience News editor.
- The underlying journal paper was reviewed in full by editorial staff.
- Additional context was provided by editorial contributors.
About this AI and neurotech research news
Author: Miki Yamaoka (Miki Yamaoka, Institute of Science Tokyo)
Source: Institute of Science Tokyo
Contact: Miki Yamaoka – Institute of Science Tokyo
Image: Image credited to Neuroscience News
Original Research: The findings will be presented at the 39th Conference on Neural Information Processing Systems (NeurIPS 2025).