An Incremental Dialogue System for Learning Visually Grounded Language (demonstration system)

Yu, Yanchao; Eshghi, Arash; Lemon, Oliver

Abstract

We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic, and bi-directional grammar framework – Dynamic Syntax and Type Theory with Records (DS-TTR1, (Eshghi et al., 2012; Kempson et al., 2001)) – with a set of visual classifiers that are learned throughout the interaction and which ground the semantic/contextual representations that it produces (c.f. Kennington & Schlangen (2015)) Our approach extends Dobnik et al. (2012) in integrating perception (vision in this case) and language within a single formal system: Type Theory with Records (TTR (Cooper, 2005)). The combination of deep semantic representations in TTR with an incremental grammar (Dynamic Syntax) allows for complex multi-turn dialogues to be parsed and generated (Eshghi et al., 2015). These include clarification interaction, corrections, ellipsis, and utterance continuations (see e.g. the dialogue in Fig. 1)

Citation

Yu, Y., Eshghi, A., & Lemon, O. (2016, July). An Incremental Dialogue System for Learning Visually Grounded Language (demonstration system). Presented at 20th Workshop Series on the Semantics and Pragmatics of Dialogue 2016, New Brunswick, US

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 20th Workshop Series on the Semantics and Pragmatics of Dialogue 2016
• Start Date: Jul 16, 2016
• End Date: Jul 18, 2016
• Publication Date: 2016
• Deposit Date: Jun 29, 2023
• Pages: 120-121
• Series ISSN: 2308-2275
• Book Title: Proceedings of the 20th Workshop on the Semantics and Pragmatics of Dialogue
• Public URL: http://researchrepository.napier.ac.uk/Output/3125803