Skip to main content

Research Repository

Advanced Search

Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement


Leandro A. Passos


Javier Del Ser

Ahsan Adeel


This paper proposes a novel multimodal self-supervised architecture for energy-efficient audio-visual (AV) speech enhancement that integrates Graph Neural Networks with canonical correlation analysis (CCA-GNN). The proposed approach lays its foundations on a state-of-the-art CCA-GNN that learns representative embeddings by maximizing the correlation between pairs of augmented views of the same input while decorrelating disconnected features. The key idea of the conventional CCA-GNN involves discarding augmentation-variant information and preserving augmentation-invariant information while preventing capturing of redundant information. Our proposed AV CCA-GNN model deals with multimodal representation learning context. Specifically, our model improves contextual AV speech processing by maximizing canonical correlation from augmented views of the same channel and canonical correlation from audio and visual embeddings. In addition, it proposes a positional node encoding that considers a prior-frame sequence distance instead of a feature-space representation when computing the node’s nearest neighbors, introducing temporal information in the embeddings through the neighborhood’s connectivity. Experiments conducted on the benchmark ChiME3 dataset show that our proposed prior frame-based AV CCA-GNN ensures a better feature learning in the temporal context, leading to more energy-efficient speech reconstruction than state-of-the-art CCA-GNN and multilayer perceptron.


Passos, L. A., Papa, J. P., Del Ser, J., Hussain, A., & Adeel, A. (2023). Multimodal audio-visual information fusion using canonical-correlated Graph Neural Network for energy-efficient speech enhancement. Information Fusion, 90, 1-11.

Journal Article Type Article
Acceptance Date Sep 6, 2022
Online Publication Date Sep 11, 2022
Publication Date 2023-02
Deposit Date Sep 28, 2022
Publicly Available Date Mar 12, 2024
Journal Information Fusion
Print ISSN 1566-2535
Publisher Elsevier
Peer Reviewed Peer Reviewed
Volume 90
Pages 1-11
Keywords Canonical correlation analysis, Graph Neural Networks, Multimodal learning, Positional encoding, Prior frames neighborhood
Public URL


This file is under embargo until Mar 12, 2024 due to copyright reasons.

Contact to request a copy for personal use.

Downloadable Citations