Unsupervised Anomaly Thresholding from Reconstruction Errors

Ndubuaku, Maryleen U.; Anjum, Ashiq; Liotta, Antonio

Abstract

Internet of Things (IoT) sensors generate massive streaming data which needs to be processed in real-time for many applications. Anomaly detection is one popular way to process such data and discover nuggets of information. Various machine learning techniques for anomaly detection rely on pre-labelled data which is very expensive and not feasible for streaming scenarios. Autoencoders have been found effective for unsupervised outlier removal because of their inherent ability to better reconstruct data with higher density. Our work aims to leverage this principle to investigate approaches through which the optimal threshold for anomaly detection can be obtained in an automated and adaptive fashion for streaming scenarios. Rather than experimentally setting an optimal threshold through trial and error, we obtain the threshold from the reconstruction errors of the training data. Inspired by image processing, we investigate how thresholds set by various statistical approaches can perform in an image dataset.

Citation

Ndubuaku, M. U., Anjum, A., & Liotta, A. (2019, October). Unsupervised Anomaly Thresholding from Reconstruction Errors. Presented at IDCS: International Conference on Internet and Distributed Computing Systems, Naples, Italy

• Presentation Conference Type: Conference Paper (published)
• Conference Name: IDCS: International Conference on Internet and Distributed Computing Systems
• Start Date: Oct 10, 2019
• End Date: Oct 12, 2019
• Online Publication Date: Nov 10, 2019
• Publication Date: 2019
• Deposit Date: Feb 24, 2020
• Publisher: Springer
• Pages: 123-129
• Series Title: Lecture Notes in Computer Science
• Series Number: 11874
• Series ISSN: 0302-9743
• Book Title: Unsupervised Anomaly Thresholding from Reconstruction Errors
• ISBN: 9783030349134
• DOI: https://doi.org/10.1007/978-3-030-34914-1_12
• Keywords: Anomaly detection, Anomaly thresholding, Unsupervised learning
• Public URL: http://researchrepository.napier.ac.uk/Output/2321868