Skip to main content

Research Repository

Advanced Search

Extracting Visual Micro-Doppler Signatures from Human Lips Motion Using UoG Radar Sensing Data for Hearing Aid Applications

Saeed, Umer; Shah, Syed Aziz; Ghadi, Yazeed Yasin; Khan, Muhammad Zakir; Ahmad, Jawad; Shah, Syed Ikram; Hameed, Hira; Abbasi, Qammer H.

Authors

Umer Saeed

Syed Aziz Shah

Yazeed Yasin Ghadi

Muhammad Zakir Khan

Syed Ikram Shah

Hira Hameed

Qammer H. Abbasi



Abstract

This study proposes a secure and effective lips-reading system that can accurately detect lips movements, even when face masks are worn. The system utilizes radio frequency (RF) sensing and ultra-wideband (UWB) radar technology, which overcomes the challenges posed by traditional vision-based systems. By leveraging deep learning models, the system interprets lips and mouth movements and achieves an overall accuracy of 90% for both mask-on and mask-off scenarios. The study utilized a trusted dataset from the University of Glasgow (UoG), consisting of spectrograms of lips motions stating five vowels and a voiceless class from distinct participants. The cutting-edge deep learning algorithm, Residual Neural Network (ResNet50), was used for the evaluation of the dataset and achieved an 87% accurate detection rate with a mask-on scenario, which is a 14% improvement compared to prior published work. The findings of this study contribute to the development of a robust lips-reading framework that can enhance communication accessibility in applications such as hearing aids, voice-controlled systems, biometrics, and more.

Citation

Saeed, U., Shah, S. A., Ghadi, Y. Y., Khan, M. Z., Ahmad, J., Shah, S. I., Hameed, H., & Abbasi, Q. H. (2023). Extracting Visual Micro-Doppler Signatures from Human Lips Motion Using UoG Radar Sensing Data for Hearing Aid Applications. IEEE Sensors Journal, 23(19), 22111-22118. https://doi.org/10.1109/jsen.2023.3308972

Journal Article Type Article
Acceptance Date Aug 22, 2023
Online Publication Date Aug 31, 2023
Publication Date Oct 1, 2023
Deposit Date Oct 17, 2023
Journal IEEE Sensors Journal
Print ISSN 1530-437X
Electronic ISSN 1558-1748
Publisher Institute of Electrical and Electronics Engineers
Peer Reviewed Peer Reviewed
Volume 23
Issue 19
Pages 22111-22118
DOI https://doi.org/10.1109/jsen.2023.3308972
Keywords ResNet50; InceptionV3; VGG16; RF sensing; UWB radar; lips-reading; speech recognition