Sentic blending: Scalable multimodal fusion for the continuous interpretation of semantics and sentics

Cambria, Erik; Howard, Newton; Hsu, Jane; Hussain, Amir

Abstract

The capability of interpreting the conceptual and affective information associated with natural language through different modalities is a key issue for the enhancement of human-agent interaction. The proposed methodology, termed sentic blending, enables the continuous interpretation of semantics and sentics (i.e., the conceptual and affective information associated with natural language) based on the integration of an affective common-sense knowledge base with any multimodal signal-processing module. In this work, in particular, sentic blending is interfaced with a facial emotional classifier and an opinion mining engine. One of the main distinguishing features of the proposed technique is that it does not simply perform cognitive and affective classification in terms of discrete labels, but it operates in a multidimensional space that enables the generation of a continuous stream characterising user's semantic and sentic progress over time, despite the outputs of the unimodal categorical modules have very different time-scales and output labels.

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI)
• Start Date: Apr 16, 2013
• End Date: Apr 19, 2013
• Online Publication Date: Sep 30, 2013
• Publication Date: 2013
• Deposit Date: Oct 11, 2019
• Publisher: Institute of Electrical and Electronics Engineers
• Pages: 108-117
• Book Title: 2013 IEEE Symposium on Computational Intelligence for Human-like Intelligence (CIHLI)
• DOI: https://doi.org/10.1109/CIHLI.2013.6613272
• Keywords: Multimodal fusion, SenticNet, Facial expression analysis, Affective common-sense, Emotion recognition
• Public URL: http://researchrepository.napier.ac.uk/Output/1793170