Skip to main content

Research Repository

Advanced Search

A novel brain-inspired compression-based optimised multimodal fusion for emotion recognition

Gogate, M.; Adeel, A.; Hussain, A.


A. Adeel


The curse of dimensionality is a well-established phenomenon. However, the properties of high dimensional data are often poorly understood and overlooked during the process of data modelling and analysis. Similarly, how to optimally fuse different modalities is still a big research question. In this paper, we addressed these challenges by proposing a novel two level brain-inspired compression based optimised multimodal fusion framework for emotion recognition. In the first level, the framework extracts the compressed and optimised multimodal features by applying a deep convolutional neural network (CNN) based compression on each modality (i.e. audio, text, and visuals). The second level simply concatenates the extracted optimised and compressed features for classification. The performance of the proposed approach with two different compression levels (i.e. 78% and 98%) is compared with late fusion (class level-1 dimension, class probabilities level-4 dimension) and early fusion (feature level-72000 dimension). The simulation results and critical analysis have demonstrated up to 10% and 5% performance improvement as compared to the state-of-the-art support vector machine (SVM) and long-short-term memory (LSTM) based multimodal emotion recognition systems respectively. We hypothesise that there exist an optimal level of compression at which optimised multimodal features could be extracted from each modality, which could lead to a significant performance improvement.

Presentation Conference Type Conference Paper (Published)
Conference Name 2017 IEEE Symposium Series on Computational Intelligence (SSCI)
Start Date Nov 27, 2017
End Date Dec 1, 2017
Online Publication Date Feb 8, 2018
Publication Date Feb 8, 2018
Deposit Date Sep 2, 2019
Publisher Institute of Electrical and Electronics Engineers
Pages 1-7
Public URL

You might also like

Downloadable Citations