Leveraging label hierarchy using transfer and multi-task learning: A case study on patent classification

Aroyehun, Segun Taofeek; Angel, Jason; Majumder, Navonil; Gelbukh, Alexander; Hussain, Amir

Abstract

When labels are organized into a meaningful taxonomy, the parent-child relationship between labels at different levels can give the classifier additional information not deducible from the data alone, especially with limited training data. As a case study, we illustrate this effect on the task of patent classification—the task of categorizing patent documents based on their technical content. Existing approaches do not take into consideration this additional information. Experiments on two patent classification datasets, WIPO-alpha and USPTO-2M, show that our regularized Gated Recurrent Unit (GRU) architecture already gives a performance improvement with a micro-averaged precision score using the top prediction of 0.5191 and 0.5740 on the two datasets, respectively. However, knowledge transfer along the label hierarchy gives further significant improvement on WIPO-alpha, raising the score to 0.5376, and a small improvement on USPTO-2M to 0.5743. Our analyses reveal that incorporating label information improves performance on classes with fewer examples and makes model robust to errors that result from predicting closely related labels.

Citation

Aroyehun, S. T., Angel, J., Majumder, N., Gelbukh, A., & Hussain, A. (2021). Leveraging label hierarchy using transfer and multi-task learning: A case study on patent classification. Neurocomputing, 464, 421-431. https://doi.org/10.1016/j.neucom.2021.07.057

• Journal Article Type: Article
• Acceptance Date: Jul 25, 2021
• Online Publication Date: Jul 30, 2021
• Publication Date: 2021-11
• Deposit Date: Oct 14, 2021
• Journal: Neurocomputing
• Print ISSN: 0925-2312
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 464
• Pages: 421-431
• DOI: https://doi.org/10.1016/j.neucom.2021.07.057
• Keywords: Transfer learning, Multi-task learning, Patent classification, Natural language processing, Neural networks, Machine learning
• Public URL: http://researchrepository.napier.ac.uk/Output/2811581