Extracting Visual Micro-Doppler Signatures from Human Lips Motion Using UoG Radar Sensing Data for Hearing Aid Applications

Saeed, Umer; Shah, Syed Aziz; Ghadi, Yazeed Yasin; Khan, Muhammad Zakir; Ahmad, Jawad; Shah, Syed Ikram; Hameed, Hira; Abbasi, Qammer H.

Abstract

This study proposes a secure and effective lips-reading system that can accurately detect lips movements, even when face masks are worn. The system utilizes radio frequency (RF) sensing and ultra-wideband (UWB) radar technology, which overcomes the challenges posed by traditional vision-based systems. By leveraging deep learning models, the system interprets lips and mouth movements and achieves an overall accuracy of 90% for both mask-on and mask-off scenarios. The study utilized a trusted dataset from the University of Glasgow (UoG), consisting of spectrograms of lips motions stating five vowels and a voiceless class from distinct participants. The cutting-edge deep learning algorithm, Residual Neural Network (ResNet50), was used for the evaluation of the dataset and achieved an 87% accurate detection rate with a mask-on scenario, which is a 14% improvement compared to prior published work. The findings of this study contribute to the development of a robust lips-reading framework that can enhance communication accessibility in applications such as hearing aids, voice-controlled systems, biometrics, and more.

• Journal Article Type: Article
• Acceptance Date: Aug 22, 2023
• Online Publication Date: Aug 31, 2023
• Publication Date: Oct 1, 2023
• Deposit Date: Oct 17, 2023
• Journal: IEEE Sensors Journal
• Print ISSN: 1530-437X
• Electronic ISSN: 1558-1748
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 23
• Issue: 19
• Pages: 22111-22118
• DOI: https://doi.org/10.1109/jsen.2023.3308972
• Keywords: ResNet50; InceptionV3; VGG16; RF sensing; UWB radar; lips-reading; speech recognition