A CNN-Transformer Hybrid Recognition Approach for sEMG-based Dynamic Gesture Prediction

Liu, Yanhong; Li, Xingyu; Yang, Lei; Bian, Guibin; Yu, Hongnian

Abstract

As a unique physiological electrical signal in the human body, surface electromyography (sEMG) signals always include human movement intention and muscle state. Through the collection of sEMG signals, different gestures can be effectively recognized. At present, the convolutional neural network (CNN) has been widely applied to different gesture recognition systems. However, due to its inherent limitations in global context feature extraction, it exists a certain shortcoming on high-precision prediction tasks. To solve this issue, a CNN-transformer hybrid recognition approach is proposed for high-precision dynamic gesture prediction. In addition, the continuous wavelet transform (CWT) is proposed for to acquire the time-frequency maps. To realize effective feature representation of local features from the time-frequency maps, an attention fusion block (AFB) is proposed to build the deep CNN network branch to effectively extract key channel information and spatial information from local features. Faced with the inherent limitations in global context feature extraction of CNNs, a transformer network branch is proposed to model the global relationship between pixels, called convolution and transformer (CAT) network branch. In addition, a multi-scale feature attention block (MFA) is proposed for effective feature aggregation of local features and global contexts by learning adaptive multi-scale features and suppressing irrelevant scale information. The experimental results on the established multi-channel sEMG signal time-frequency map dataset show that the proposed CNN transformer hybrid recognition network has competitive recognition performance compared with other state-of-the-art recognition networks, and the average recognition speed of each spectrogram on the test set is only 14.7ms. The proposed network can effectively improve network performance and identification efficiency.

Citation

Liu, Y., Li, X., Yang, L., Bian, G., & Yu, H. (2023). A CNN-Transformer Hybrid Recognition Approach for sEMG-based Dynamic Gesture Prediction. IEEE Transactions on Instrumentation and Measurement, 72, Article 2514816. https://doi.org/10.1109/tim.2023.3273651

• Journal Article Type: Article
• Acceptance Date: Apr 12, 2023
• Online Publication Date: May 8, 2023
• Publication Date: 2023
• Deposit Date: Jul 6, 2023
• Journal: IEEE Transactions on Instrumentation and Measurement
• Print ISSN: 0018-9456
• Publisher: Institute of Electrical and Electronics Engineers (IEEE)
• Peer Reviewed: Peer Reviewed
• Volume: 72
• Article Number: 2514816
• DOI: https://doi.org/10.1109/tim.2023.3273651
• Keywords: Hand Gesture Recognition, sEMG Sensor, CNN, Transformer, Feature Fusion