Skip to main content

Research Repository

Advanced Search

All Outputs (56)

Evaluating Language Model Vulnerability to Poisoning Attacks in Low-Resource Settings (2024)
Journal Article
Plant, R., Giuffrida, M. V., Pitropakis, N., & Gkatzia, D. (2024). Evaluating Language Model Vulnerability to Poisoning Attacks in Low-Resource Settings. IEEE/ACM Transactions on Audio, Speech and Language Processing, 33, 54-67. https://doi.org/10.1109/taslp.2024.3507565

Pre-trained language models are a highly effective source of knowledge transfer for natural language processing tasks, as their development represents an investment of resources beyond the reach of most researchers and end users. The widespread avail... Read More about Evaluating Language Model Vulnerability to Poisoning Attacks in Low-Resource Settings.

You Are What You Write: Author re-identification privacy attacks in the era of pre-trained language models (2024)
Journal Article
Plant, R., Giuffrida, V., & Gkatzia, D. (2025). You Are What You Write: Author re-identification privacy attacks in the era of pre-trained language models. Computer Speech and Language, 90, Article 101746. https://doi.org/10.1016/j.csl.2024.101746

The widespread use of pre-trained language models has revolutionised knowledge transfer in natural language processing tasks. However, there is a concern regarding potential breaches of user trust due to the risk of re-identification attacks, where m... Read More about You Are What You Write: Author re-identification privacy attacks in the era of pre-trained language models.

Exploring the impact of data representation on neural data-to-text generation (2024)
Presentation / Conference Contribution
Howcroft, D. M., Watson, L. N., Nedopas, O., & Gkatzia, D. (2024, September). Exploring the impact of data representation on neural data-to-text generation. Presented at INLG 2024, Tokyo, Japan

A relatively under-explored area in research on neural natural language generation is the impact of the data representation on text quality. Here we report experiments on two leading input representations for data-to-text generation: attribute-value... Read More about Exploring the impact of data representation on neural data-to-text generation.

Automatic Metrics in Natural Language Generation: A survey of Current Evaluation Practices (2024)
Presentation / Conference Contribution
Schmidtova, P., Mahamood, S., Balloccu, S., Dusek, O., Gatt, A., Gkatzia, D., Howcroft, D. M., Platek, O., & Sivaprasad, A. (2024, September). Automatic Metrics in Natural Language Generation: A survey of Current Evaluation Practices. Presented at INLG 2024, Tokyo, Japan

Automatic metrics are extensively used to evaluate Natural Language Processing systems. However, there has been increasing focus on how the are used and reported by practitioners within the field. In this paper, we have conducted a survey on the use... Read More about Automatic Metrics in Natural Language Generation: A survey of Current Evaluation Practices.

Automated Human-Readable Label Generation in Open Intent Discovery (2024)
Presentation / Conference Contribution
Anderson, G., Hart, E., Gkatzia, D., & Beaver, I. (2024, September). Automated Human-Readable Label Generation in Open Intent Discovery. Presented at Interspeech 2024, Kos, Greece

The correct determination of user intent is key in dialog systems. However, an intent classifier often requires a large, labelled training dataset to identify a set of known intents. The creation of such a dataset is a complex and time-consuming task... Read More about Automated Human-Readable Label Generation in Open Intent Discovery.

An Open Intent Discovery Evaluation Framework (2024)
Presentation / Conference Contribution
Anderson, G., Hart, E., Gkatzia, D., & Beaver, I. (2024, September). An Open Intent Discovery Evaluation Framework. Presented at SIGDIAL 2024, Kyoto, Japan

In the development of dialog systems the discovery of the set of target intents to identify is a crucial first step that is often overlooked. Most intent detection works assume that a labelled dataset already exists, however creating these datasets i... Read More about An Open Intent Discovery Evaluation Framework.

Participatory Design with Domain Experts: A Delphi Study for a Career Support Chatbot (2024)
Presentation / Conference Contribution
Wilson, M., Brazier, D., Gkatzia, D., & Robertson, P. (2024, July). Participatory Design with Domain Experts: A Delphi Study for a Career Support Chatbot. Presented at ACM Conversational User Interfaces 2024 (CUI ’24), Luxembourg, Luxembourg

We present a study of collaboration with expert participants for the purpose of the responsible design of a conversational agent. The Delphi study was used to identify and develop design and evaluation criteria for an automated career support interve... Read More about Participatory Design with Domain Experts: A Delphi Study for a Career Support Chatbot.

Reproducing Human Evaluation of Meaning Preservation in Paraphrase Generation (2024)
Presentation / Conference Contribution
Watson, L. N., & Gkatzia, D. (2024, May). Reproducing Human Evaluation of Meaning Preservation in Paraphrase Generation. Presented at HumEval2024 at LREC-COLING 2024, Turin, Italy

Reproducibility is a cornerstone of scientific research, ensuring the reliability and generalisability of findings. The ReproNLP Shared Task on Reproducibility of Evaluations in NLP aims to assess the reproducibility of human evaluation studies. This... Read More about Reproducing Human Evaluation of Meaning Preservation in Paraphrase Generation.

Unveiling NLG Human-Evaluation Reproducibility: Lessons Learned and Key Insights from Participating in the ReproNLP Challenge (2023)
Presentation / Conference Contribution
Watson, L., & Gkatzia, D. (2023, September). Unveiling NLG Human-Evaluation Reproducibility: Lessons Learned and Key Insights from Participating in the ReproNLP Challenge. Presented at 3rd Workshop on Human Evaluation of NLP Systems (HumEval), Varna, Bulgaria

Human evaluation is crucial for NLG systems as it provides a reliable assessment of the quality, effectiveness, and utility of generated language outputs. However, concerns about the reproducibility of such evaluations have emerged, casting doubt on... Read More about Unveiling NLG Human-Evaluation Reproducibility: Lessons Learned and Key Insights from Participating in the ReproNLP Challenge.

Edge NLP for Efficient Machine Translation in Low Connectivity Areas (2023)
Presentation / Conference Contribution
Watt, T., Chrysoulas, C., & Gkatzia, D. (2023, October). Edge NLP for Efficient Machine Translation in Low Connectivity Areas. Presented at IEEE 9th World Forum on Internet of Things: 2nd Workshop on Convergence of Edge Intelligence in IoT (EdgeAI-IoT 2023), Aveiro, Portugal

Machine translation (MT) usually requires connectivity and access to the cloud which is often limited in many parts of the world, including hard to reach rural areas. Edge natural language processing (NLP) aims to solve this problem by processing lan... Read More about Edge NLP for Efficient Machine Translation in Low Connectivity Areas.

TaskMaster: A Novel Cross-platform Task-based Spoken Dialogue System for Human-Robot Interaction (2023)
Presentation / Conference Contribution
Strathearn, C., Yu, Y., & Gkatzia, D. (2023, March). TaskMaster: A Novel Cross-platform Task-based Spoken Dialogue System for Human-Robot Interaction. Presented at 'HRCI23, Stockholm, Sweden

The most effective way of communication between humans and robots is through natural language communication. However, there are many challenges to overcome before robots can effectively converse in order to collaborate and work together with humans.... Read More about TaskMaster: A Novel Cross-platform Task-based Spoken Dialogue System for Human-Robot Interaction.

Responsible Design & Evaluation of a Conversational Agent for a National Careers Service (2023)
Presentation / Conference Contribution
Wilson, M., Cruickshank, P., Gkatzia, D., & Robertson, P. (2023, September). Responsible Design & Evaluation of a Conversational Agent for a National Careers Service. Presented at Symposium on Future Directions in Information Access (FDIA) 2023, Vienna, Austria

This PhD project applies a research-through-design approach to the development of a conversational agent for a national career service for young people. This includes addressing practical, interactional and ethical aspects of the system. For each asp... Read More about Responsible Design & Evaluation of a Conversational Agent for a National Careers Service.

Working with troubles and failures in conversation between humans and robots: workshop report (2023)
Journal Article
Förster, F., Romeo, M., Holthaus, P., Wood, L. J., Dondrup, C., Fischer, J. E., Liza, F. F., Kaszuba, S., Hough, J., Nesset, B., Hernández García, D., Kontogiorgos, D., Williams, J., Özkan, E. E., Barnard, P., Berumen, G., Price, D., Cobb, S., Wiltschko, M., Tisserand, L., …Kapetanios, E. (2023). Working with troubles and failures in conversation between humans and robots: workshop report. Frontiers in Robotics and AI, 10, Article 1202306. https://doi.org/10.3389/frobt.2023.1202306

This paper summarizes the structure and findings from the first Workshop on Troubles and Failures in Conversations between Humans and Robots. The workshop was organized to bring together a small, interdisciplinary group of researchers working on misc... Read More about Working with troubles and failures in conversation between humans and robots: workshop report.

enunlg: a Python library for reproducible neural data-to-text experimentation (2023)
Presentation / Conference Contribution
Howcroft, D. M., & Gkatzia, D. (2023, September). enunlg: a Python library for reproducible neural data-to-text experimentation. Presented at 16th International Natural Language Generation Conference, Prague, Czechia

Over the past decade, a variety of neural ar-chitectures for data-to-text generation (NLG) have been proposed. However, each system typically has its own approach to pre-and post-processing and other implementation details. Diversity in implementatio... Read More about enunlg: a Python library for reproducible neural data-to-text experimentation.

LOWRECORP: the Low-Resource NLG Corpus Building Challenge (2023)
Presentation / Conference Contribution
Chandu, K. R., Howcroft, D., Gkatzia, D., Chung, Y.-L., Hou, Y., Emezue, C., Rajpoot, P., & Adewumi, T. (2023, September). LOWRECORP: the Low-Resource NLG Corpus Building Challenge. Presented at 16th International Natural Language Generation Conference, Prague, Czechia

Most languages in the world do not have sufficient data available to develop neural-network-based natural language generation (NLG) systems. To alleviate this resource scarcity, we propose a novel challenge for the NLG community: low-resource languag... Read More about LOWRECORP: the Low-Resource NLG Corpus Building Challenge.

Building a dual dataset of text-and image-grounded conversations and summarisation in Gàidhlig (Scottish Gaelic) (2023)
Presentation / Conference Contribution
Howcroft, D. M., Lamb, W., Groundwater, A., & Gkatzia, D. (2023, September). Building a dual dataset of text-and image-grounded conversations and summarisation in Gàidhlig (Scottish Gaelic). Presented at The 16th International Natural Language Generation Conference

Gàidhlig (Scottish Gaelic; gd) is spoken by about 57k people in Scotland, but remains an under-resourced language with respect to natural language processing in general and natural language generation (NLG) in particular. To address this gap, we deve... Read More about Building a dual dataset of text-and image-grounded conversations and summarisation in Gàidhlig (Scottish Gaelic).

Barriers and enabling factors for error analysis in NLG research (2023)
Journal Article
Van Miltenburg, E., Clinciu, M., Dušek, O., Gkatzia, D., Inglis, S., Leppänen, L., Mahamood, S., Schoch, S., Thomson, C., & Wen, L. (2023). Barriers and enabling factors for error analysis in NLG research. Northern European Journal of Language Technology, 9(1), https://doi.org/10.3384/nejlt.2000-1533.2023.4529

Earlier research has shown that few studies in Natural Language Generation (NLG) evaluate their system outputs using an error analysis, despite known limitations of automatic evaluation metrics and human ratings. This position paper takes the stance... Read More about Barriers and enabling factors for error analysis in NLG research.

Most NLG is Low-Resource: here's what we can do about it (2022)
Presentation / Conference Contribution
Howcroft, D. M., & Gkatzia, D. (2022, December). Most NLG is Low-Resource: here's what we can do about it. Presented at Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), Abu Dhabi, UAE

Many domains and tasks in natural language generation (NLG) are inherently 'low-resource', where training data, tools and linguistic analyses are scarce. This poses a particular challenge to researchers and system developers in the era of machine-lea... Read More about Most NLG is Low-Resource: here's what we can do about it.

A Commonsense-Enhanced Document-Grounded Conversational Agent: A Case Study on Task-Based Dialogue (2022)
Book Chapter
Strathearn, C., & Gkatzia, D. (2023). A Commonsense-Enhanced Document-Grounded Conversational Agent: A Case Study on Task-Based Dialogue. In M. Abbas (Ed.), Analysis and Application of Natural Language and Speech Processing (123-144). Springer. https://doi.org/10.1007/978-3-031-11035-1_6

This paper argues that future dialogue systems must retrieve relevant information from multiple structured and unstructured data sources in order to generate natural and informative responses as well as exhibit commonsense capabilities and flexibilit... Read More about A Commonsense-Enhanced Document-Grounded Conversational Agent: A Case Study on Task-Based Dialogue.