Combining Visual and Social Dialogue for Human-Robot Interaction

Gunson, Nancie; Hernandez Garcia, Daniel; Part, Jose L.; Yu, Yanchao; Sieińska, Weronika; Dondrup, Christian; Lemon, Oliver

Abstract

We will demonstrate a prototype multimodal conversational AI system that will act as a receptionist in a hospital waiting room, combining visually-grounded dialogue with social conversation. The system supports visual object conversation in the waiting room (e.g. looking for available seats or personal belongings), task-based dialogues regarding navigation and check-in procedures in the hospital, as well as access to the latest news, and a quiz game about coronavirus. The prototype system therefore demonstrates how to weave together a wide range of natural, daily conversations with end users that vary in complexity; from complex visual dialogue to chitchat and quiz games, to task-oriented domain-specific conversations. We are currently able to demonstrate the system via a web-based interface. It will soon be deployed on the ARI robot in a hospital waiting room.

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: 2021 International Conference on Multimodal Interaction
• Start Date: Oct 18, 2021
• End Date: Oct 22, 2022
• Online Publication Date: Oct 18, 2021
• Publication Date: 2021-10
• Deposit Date: Jun 27, 2023
• Publisher: Association for Computing Machinery (ACM)
• Pages: 841-842
• Book Title: ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction
• ISBN: 978-1-4503-8481-0
• DOI: https://doi.org/10.1145/3462244.3481303