Novel single and multi-layer echo-state recurrent autoencoders for representation learning

Chouikhi, Naima; Ammar, Boudour; Hussain, Amir; Alimi, Adel M.

Abstract

Representation learning impacts the performance of Machine Learning (ML) models. Feature extraction-based methods such as Auto-Encoders (AEs) are used to find new, more accurate data representations from original ones. They perform efficiently on a specific task, in terms of: (1) high accuracy, (2) large short-term memory and (3) low execution time. The Echo-State Network (ESN) is a recent specific kind of a Recurrent Neural Networks (RNN), that presents very rich dynamics on account of its reservoir-based hidden layer. It is widely used in dealing with complex non-linear problems and has been shown to outperform classical approaches in a number of benchmark tasks. In this paper, the powerful dynamism and large memory provided by the ESN and complementary strengths of AEs in feature extraction are integrated, to develop a novel Echo-State Recurrent Autoencoder (ES-RA). In order to devise more robust alternatives to conventional reservoir-based networks, both single- (SL-ES-RA) and multi-layer (ML-ES-RA) models are formulated. The new features, once extracted from ESN’s hidden layer, are applied to various benchmark ML tasks including classification, time series prediction and regression. A range of evaluation metrics are shown to improve considerably compared to those obtained when applying original data features. An accuracy-based comparison is performed between our proposed recurrent AEs and two variants of ELM feed-forward AEs (Single and ML), for both noise free and noisy data. In summary, a comparative empirical study reveals the key contribution of exploiting recurrent connections in improving benchmark performance results.

• Journal Article Type: Article
• Acceptance Date: Jun 6, 2022
• Online Publication Date: Jul 11, 2022
• Publication Date: 2022-09
• Deposit Date: Oct 5, 2022
• Journal: Engineering Applications of Artificial Intelligence
• Print ISSN: 0952-1976
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 114
• Article Number: 105051
• DOI: https://doi.org/10.1016/j.engappai.2022.105051
• Keywords: Echo-State Network, Autoencoder, Multi-layer ESN, Representation learning, Classification, Time series prediction
• Public URL: http://researchrepository.napier.ac.uk/Output/2899288