An Open Intent Discovery Evaluation Framework

Anderson, Grant; Hart, Emma; Gkatzia, Dimitra; Beaver, Ian

Abstract

In the development of dialog systems the discovery of the set of target intents to identify is a crucial first step that is often overlooked. Most intent detection works assume that a labelled dataset already exists, however creating these datasets is no trivial task and usually requires humans to manually analyse, decide on intent labels and tag accordingly. The field of Open Intent Discovery addresses this problem by automating the process of grouping utterances and providing the user with the discovered intents. Our Open Intent Discovery framework allows for the user to choose from a range of different techniques for each step in the discovery process, including the ability to extend previous works with a human-readable label generation stage. We also provide an analysis of the relationship between dataset features and optimal combination of techniques for each step to help others choose without having to explore every possible combination for their unlabelled data.

Citation

Anderson, G., Hart, E., Gkatzia, D., & Beaver, I. (2024, September). An Open Intent Discovery Evaluation Framework. Presented at SIGDIAL 2024, Kyoto, Japan

• Presentation Conference Type: Conference Paper (published)
• Conference Name: SIGDIAL 2024
• Start Date: Sep 18, 2024
• End Date: Sep 20, 2024
• Acceptance Date: Jul 15, 2024
• Online Publication Date: Oct 1, 2024
• Publication Date: 2024
• Deposit Date: Jul 16, 2024
• Publicly Available Date: Oct 3, 2024
• Publisher: Association for Computational Linguistics (ACL)
• Peer Reviewed: Peer Reviewed
• Pages: 760–769
• Book Title: Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue
• ISBN: 9798891761612
• DOI: https://doi.org/10.18653/v1/2024.sigdial-1.64
• External URL: https://2024.sigdial.org/