Multimodal Emotion Recognition from Art Using Sequential Co-Attention

Tashu, Tsegaye Misikir; Hajiyeva, Sakina; Horvath, Tomas

Abstract

In this study, we present a multimodal emotion recognition architecture that uses both feature-level attention (sequential co-attention) and modality attention (weighted modality fusion) to classify emotion in art. The proposed architecture helps the model to focus on learning informative and refined representations for both feature extraction and modality fusion. The resulting system can be used to categorize artworks according to the emotions they evoke; recommend paintings that accentuate or balance a particular mood; search for paintings of a particular style or genre that represents custom content in a custom state of impact. Experimental results on the WikiArt emotion dataset showed the efficiency of the approach proposed and the usefulness of three modalities in emotion recognition.

• Journal Article Type: Article
• Acceptance Date: Aug 17, 2021
• Online Publication Date: Aug 21, 2021
• Publication Date: 2021
• Deposit Date: Mar 27, 2024
• Publicly Available Date: Mar 27, 2024
• Journal: Journal of Imaging
• Print ISSN: 2313-433X
• Publisher: MDPI
• Peer Reviewed: Peer Reviewed
• Volume: 7
• Issue: 8
• Article Number: 157
• DOI: https://doi.org/10.3390/jimaging7080157
• Keywords: multimodal; emotions; attention; art; modality fusion; emotion analysis
• Public URL: http://researchrepository.napier.ac.uk/Output/3577489