Towards intelligibility-oriented audio-visual speech enhancement

Hussain, Tassadaq; Gogate, Mandar; Dashtipour, Kia; Hussain, Amir

Abstract

Existing deep learning (DL) based approaches are generally optimised to minimise the distance between clean and enhanced speech features. These often result in improved speech quality however they suffer from a lack of generalisation and may not deliver the required speech intelligibility in real noisy situations. In an attempt to address these challenges, researchers have explored intelligibility-oriented (I-O) loss functions and integration of audio-visual (AV) information for more robust speech enhancement (SE). In this paper, we introduce DL based I-O SE algorithms exploiting AV information, which is a novel and previously unexplored research direction. Specifically, we present a fully convolutional AV SE model that uses a modified short-time objective intelligibility (STOI) metric as a training cost function. To the best of our knowledge, this is the first work that exploits the integration of AV modalities with an I-O based loss function for SE. Comparative experimental results demonstrate that our proposed I-O AV SE framework outperforms audio-only (AO) and AV models trained with conventional distance-based loss functions, in terms of standard objective evaluation measures when dealing with unseen speakers and noises.

Citation

Hussain, T., Gogate, M., Dashtipour, K., & Hussain, A. (2021, September). Towards intelligibility-oriented audio-visual speech enhancement. Presented at The Clarity Workshop on Machine Learning Challenges for Hearing Aids (Clarity-2021), Online

• Presentation Conference Type: Conference Paper (published)
• Conference Name: The Clarity Workshop on Machine Learning Challenges for Hearing Aids (Clarity-2021)
• Start Date: Sep 16, 2021
• End Date: Sep 17, 2021
• Publication Date: 2021
• Deposit Date: May 28, 2024
• Peer Reviewed: Peer Reviewed
• Publisher URL: https://claritychallenge.org/clarity2021-workshop/papers/Clarity_2021_CEC1_paper_final_hussain.pdf