Dynamic Hand Gesture Recognition Using 3D-CNN and LSTM Networks

Ur Rehman, Muneeb; Ahmed, Fawad; Attique Khan, Muhammad; Tariq, Usman; Abdulaziz Alfouzan, Faisal; M. Alzahrani, Nouf; Ahmad, Jawad

Abstract

Recognition of dynamic hand gestures in real-time is a difficult task because the system can never know when or from where the gesture starts and ends in a video stream. Many researchers have been working on vision-based gesture recognition due to its various applications. This paper proposes a deep learning architecture based on the combination of a 3D Convolutional Neural Network (3D-CNN) and a Long Short-Term Memory (LSTM) network. The proposed architecture extracts spatial-temporal information from video sequences input while avoiding extensive computation. The 3D-CNN is used for the extraction of spectral and spatial features which are then given to the LSTM network through which classification is carried out. The proposed model is a light-weight architecture with only 3.7 million training parameters. The model has been evaluated on 15 classes from the 20BN-jester dataset available publicly. The model was trained on 2000 video-clips per class which were separated into 80% training and 20% validation sets. An accuracy of 99% and 97% was achieved on training and testing data, respectively. We further show that the combination of 3D-CNN with LSTM gives superior results as compared to MobileNetv2 + LSTM.

• Journal Article Type: Article
• Acceptance Date: Jul 27, 2021
• Publication Date: Oct 11, 2021
• Deposit Date: Oct 28, 2021
• Publicly Available Date: Oct 28, 2021
• Journal: Computers, Materials & Continua
• Print ISSN: 1546-2218
• Publisher: Tech Science Press
• Peer Reviewed: Peer Reviewed
• Volume: 70
• Issue: 3
• Pages: 4675-4690
• DOI: https://doi.org/10.32604/cmc.2022.019586
• Keywords: Convolutional neural networks; 3D-CNN; LSTM; spatio-temporal; jester; real-time hand gesture recognition
• Public URL: http://researchrepository.napier.ac.uk/Output/2816883