Privacy-preserving Facial Emotion Classification with Visual Micro-Doppler Signatures for Hearing Aid Applications

Anwar, Usman; Dong, Yinhuan; Arslan, Tughrul; Dashtipour, Kia; Gogate, Mandar; Hussain, Amir; Abbasi, Qammer H.; Imran, Muhammad Ali; Russ, Tom C.; Lomax, Peter

Abstract

Facial expressions are a crucial aspect of non-verbal communication and often reflect underlying emotional states. Researchers often use facial emotion detection as a tool to gain insights into cognitive processes, emotional states and cognitive load. The conventional camera-based methods to sense human emotions are privacy intrusive, lack adaptability, and are sensitive to variability. These technologies have limited generalization and may not adapt well to variations in ambient lighting, facial landmark localization, facial occlusions and emotion intensity. Radio Frequency (RF) sensing offers promising avenues for improvement with contactless, non-invasive, privacy-preserving and reliable radar-based measurements. The proposed framework utilizes deep-learning techniques to classify facial micro-doppler signatures, generated from an ultra-wideband (UWB) radar. The method relies on continuous multi-level feature learning from radar time-frequency Doppler measurements. The spatiotemporal facial features are extracted from the radar data to train deep learning models. The proposed system achieves a high multiclass classification accuracy of 77% on the continuous streamed data covering basic emotions of anger, disgust, fear, happy, neutral and sadness. The system can transform next-generation multi-modal hearing aids with emotion-aware listening effort and cognitive load detection. This can be particularly useful in translating the emotion-assisted cognitive effort for real-time speech enhancement and personalized auditory experience.

Citation

Anwar, U., Dong, Y., Arslan, T., Dashtipour, K., Gogate, M., Hussain, A., Abbasi, Q. H., Imran, M. A., Russ, T. C., & Lomax, P. (2025). Privacy-preserving Facial Emotion Classification with Visual Micro-Doppler Signatures for Hearing Aid Applications. IEEE Transactions on Instrumentation and Measurement, 74, Article 8002310. https://doi.org/10.1109/tim.2025.3548782

• Journal Article Type: Article
• Online Publication Date: Mar 6, 2025
• Publication Date: 2025
• Deposit Date: Jun 23, 2025
• Journal: IEEE Transactions on Instrumentation and Measurement
• Print ISSN: 0018-9456
• Publisher: Institute of Electrical and Electronics Engineers
• Peer Reviewed: Peer Reviewed
• Volume: 74
• Article Number: 8002310
• DOI: https://doi.org/10.1109/tim.2025.3548782