Dr Yanchao Yu Y.Yu@napier.ac.uk
Lecturer
Interactively learning visually grounded word meanings from a human tutor
Yu, Yanchao; Eshghi, Arash; Lemon, Oliver
Authors
Arash Eshghi
Oliver Lemon
Abstract
We present a multi-modal dialogue system for interactive learning of perceptually grounded word meanings from a human tutor. The system integrates an incremental, semantic parsing/generation framework - Dynamic Syntax and Type Theory with Records (DS-TTR) - with a set of visual classifiers that are learned throughout the interaction and which ground the meaning representations that it produces. We use this system in interaction with a simulated human tutor to study the effect of different dialogue policies and capabilities on accuracy of learned meanings, learning rates, and efforts/costs to the tutor. We show that the overall performance of the learning agent is affected by (1) who takes initiative in the dialogues; (2) the ability to express/use their confidence level about visual attributes; and (3) the ability to process elliptical as well as incrementally constructed dialogue turns.
Presentation Conference Type | Conference Paper (Published) |
---|---|
Conference Name | 5th Workshop on Vision and Language |
Start Date | Aug 12, 2016 |
Publication Date | 2016 |
Deposit Date | Jun 28, 2023 |
Publicly Available Date | Jun 28, 2023 |
Publisher | Association for Computational Linguistics (ACL) |
Pages | 48-53 |
Book Title | Proceedings of the 5th Workshop on Vision and Language |
Publisher URL | https://aclanthology.org/W16-3206/ |
Files
Interactively learning visually grounded word meanings from a human tutor
(3.2 Mb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
You might also like
The PARLANCE mobile application for interactive search in English and Mandarin
(2014)
Presentation / Conference Contribution
Comparing attribute classifiers for interactive language grounding
(2015)
Presentation / Conference Contribution
Towards visual dialogue for human-robot interaction
(2021)
Presentation / Conference Contribution
A comprehensive evaluation of incremental speech recognition and diarization for conversational AI
(2020)
Presentation / Conference Contribution
Incremental Generation of Visually Grounded Language in Situated Dialogue (demonstration system)
(2016)
Presentation / Conference Contribution
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search