An investigation into audiovisual speech correlation in reverberant noisy environments

Cifani, Simone; Abel, Andrew; Hussain, Amir; Squartini, Stefano; Piazza, Francesco

Abstract

As evidence of a link between the various human communication production domains has become more prominent in the last decade, the field of multimodal speech processing has undergone significant expansion. Many different specialised processing methods have been developed to attempt to analyze and utilize the complex relationship between multimodal data streams. This work uses information extracted from an audiovisual corpus to investigate and assess the correlation between audio and visual features in speech. A number of different feature extraction techniques are assessed, with the intention of identifying the visual technique that maximizes the audiovisual correlation. Additionally, this paper aims to demonstrate that a noisy and reverberant audio environment reduces the degree of audiovisual correlation, and that the application of a beamformer remedies this. Experimental results, carried out in a synthetic scenario, confirm the positive impact of beamforming not only for improving the audio-visual correlation but also in a complete audio-visual speech enhancement scheme. Thus, this work inevitably highlights an important aspect for the development of future promising bimodal speech enhancement systems.

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: COST Action 2102 International Conference
• Start Date: Oct 15, 2008
• End Date: Oct 18, 2008
• Publication Date: 2009
• Deposit Date: Oct 11, 2019
• Publisher: Springer
• Pages: 331-343
• Series Title: Lecture Notes in Computer Science
• Series Number: 5641
• Series ISSN: 0302-9743
• Book Title: Cross-Modal Analysis of Speech, Gestures, Gaze and Facial Expressions: COST Action 2102 International Conference Prague, Czech Republic, October 15-18, 2008 Revised Selected and Invited Papers
• ISBN: 978-3-642-03319-3
• DOI: https://doi.org/10.1007/978-3-642-03320-9_31
• Keywords: Discrete Cosine Transform; Automatic Speech Recognition; Speech Enhancement; Noisy Speech; Active Appearance Model
• Public URL: http://researchrepository.napier.ac.uk/Output/1793489