MULTIMODAL DEEP LEARNING FOR CONTINUOUS HAND MOTION ESTIMATION USING ELECTROMYOGRAPHY AND ULTRASOUND TOWARD DEXTEROUS PROSTHETICS AND ADAPTIVE REHABILITATION

Author

• Sulav Bastola, Kennesaw State UniversityFollow
• Sulav Bastola, Kennesaw State University

Semester of Graduation

Spring 2026

Degree Type

Dissertation

Degree Name

INTERDISCIPLINARY ENGINEERING

Department

Electrical and Computer Engineering

Committee Chair/First Advisor

Dr. Coskun Tekes

Second Advisor

Dr. Geza Kogler

Third Advisor

Dr. Paul Lee

Fourth Advisor

Dr. Razvan Cristian Voicu

Abstract

Substantial challenges in performing daily activities are experienced by stroke survivors and individuals with limb loss due to impaired upper-limb motor function. Accurate decoding of multi-degree-of-freedom (multi-DoF) hand movements is considered critical for prosthetic control, rehabilitation, and human–robot interaction. In this study, advanced sensing, modeling, and data management strategies are investigated to enhance continuous hand motion estimation. The integration of surface electromyography (sEMG) and ultrasound radiofrequency (US RF) signals is explored, with a hybrid convolutional neural network–long short-term memory model with attention mechanisms employed to improve multi-DoF hand gesture prediction. Using an extended dataset, an expanded subject cohort, and novel rotational hand motions, robustness, accuracy, and generalizability are demonstrated to be improved relative to single-modality systems. The challenge of limited multimodal datasets is addressed through the development of a spectrally constrained Variational Autoencoder–Wasserstein GAN with Gradient Penalty (VAE–WGAN–GP) for the synthesis of physiologically realistic sEMG and US RF signals. When real datasets are augmented with generated signals, cross-subject generalization and predictive performance for continuous hand motion estimation are enhanced, providing a scalable solution for labor-intensive data collection. Practical deployment is further addressed by introducing a vector-quantized variational autoencoder (VQ-VAE) for the compression of pre-beamformed US RF data, enabling high-throughput wireless transmission while preserving signal fidelity. Collectively, it is shown that multimodal sensing, deep generative modeling, and efficient data compression can synergistically enhance accurate, robust, and scalable hand motion decoding, thereby advancing the development of intuitive prosthetic systems, rehabilitation robotics, and human–computer interaction applications.