Synonym-Based Essay Generation and Augmentation for Robust Automatic Essay Scoring

Tashu, Tsegaye Misikir; Horváth, Tomáš

Abstract

Automatic essay scoring (AES) models based on neural networks (NN) have had a lot of success. However, research has shown that NN-based AES models have robustness issues, such that the output of a model changes easily with small changes in the input. We proposed to use keyword-based lexical substitution using BERT that generates new essays (adversarial samples) which are lexically similar to the original essay to evaluate the robustness of AES models trained on the original set. In order to evaluate the proposed approach, we implemented three NN-based scoring approaches and trained the scoring models using two stages. First, we trained each model using the original data and evaluate the performance using the original test and newly generated test set to see the impact of the adversarial sample of the model. Secondly, we trained the models by augmenting the generated adversarial essay with the original data to train a robust model against synonym-based adversarial attacks. The results of our experiments showed that extracting the most important words from the essay and replacing them with lexically similar words, as well as generating adversarial samples for augmentation, can significantly improve the generalization of NN-based AES models. Our experiments also demonstrated that the proposed defense is capable of not only defending against adversarial attacks, but also of improving the performance of NN-based AES models.

Citation

Tashu, T. M., & Horváth, T. (2022, November). Synonym-Based Essay Generation and Augmentation for Robust Automatic Essay Scoring. Presented at 23rd International Conference on Intelligent Data Engineering and Automated Learning (IDEAL), Manchester

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: 23rd International Conference on Intelligent Data Engineering and Automated Learning (IDEAL)
• Start Date: Nov 24, 2022
• End Date: Nov 26, 2022
• Online Publication Date: Nov 21, 2022
• Publication Date: 2022
• Deposit Date: Apr 8, 2024
• Publisher: Springer
• Pages: 12-21
• Series Title: Lecture Notes in Computer Science
• Series Number: 13756
• Series ISSN: 0302-9743
• Book Title: Intelligent Data Engineering and Automated Learning – IDEAL 2022: 23rd International Conference, IDEAL 2022, Manchester, UK, November 24–26, 2022, Proceedings
• ISBN: 9783031217524
• DOI: https://doi.org/10.1007/978-3-031-21753-1_2
• Keywords: Adversarial attack, Data augmentation, Automatic essay scoring
• Public URL: http://researchrepository.napier.ac.uk/Output/3587396