Skip to main content

Research Repository

Advanced Search

Novel single and multi-layer echo-state recurrent autoencoders for representation learning

Chouikhi, Naima; Ammar, Boudour; Hussain, Amir; Alimi, Adel M.


Naima Chouikhi

Boudour Ammar

Adel M. Alimi


Representation learning impacts the performance of Machine Learning (ML) models. Feature extraction-based methods such as Auto-Encoders (AEs) are used to find new, more accurate data representations from original ones. They perform efficiently on a specific task, in terms of: (1) high accuracy, (2) large short-term memory and (3) low execution time. The Echo-State Network (ESN) is a recent specific kind of a Recurrent Neural Networks (RNN), that presents very rich dynamics on account of its reservoir-based hidden layer. It is widely used in dealing with complex non-linear problems and has been shown to outperform classical approaches in a number of benchmark tasks. In this paper, the powerful dynamism and large memory provided by the ESN and complementary strengths of AEs in feature extraction are integrated, to develop a novel Echo-State Recurrent Autoencoder (ES-RA). In order to devise more robust alternatives to conventional reservoir-based networks, both single- (SL-ES-RA) and multi-layer (ML-ES-RA) models are formulated. The new features, once extracted from ESN’s hidden layer, are applied to various benchmark ML tasks including classification, time series prediction and regression. A range of evaluation metrics are shown to improve considerably compared to those obtained when applying original data features. An accuracy-based comparison is performed between our proposed recurrent AEs and two variants of ELM feed-forward AEs (Single and ML), for both noise free and noisy data. In summary, a comparative empirical study reveals the key contribution of exploiting recurrent connections in improving benchmark performance results.

Journal Article Type Article
Acceptance Date Jun 6, 2022
Online Publication Date Jul 11, 2022
Publication Date 2022-09
Deposit Date Oct 5, 2022
Journal Engineering Applications of Artificial Intelligence
Print ISSN 0952-1976
Publisher Elsevier
Peer Reviewed Peer Reviewed
Volume 114
Article Number 105051
Keywords Echo-State Network, Autoencoder, Multi-layer ESN, Representation learning, Classification, Time series prediction
Public URL