Bi-level multi-objective evolution of a Multi-Layered Echo-State Network Autoencoder for data representations

Chouikhi, Naima; Ammar, Boudour; Hussain, Amir; Alimi, Adel M.

Abstract

The Multi-Layered Echo-State Network (ML-ESN) is a recently developed, highly powerful type of recurrent neural network. It has succeeded in dealing with several non-linear benchmark problems. On account of its rich dynamics, ML-ESN is exploited in this paper, for the first time, as a recurrent Autoencoder (ML-ESNAE) to extract new features from original data representations. Further, the challenging and crucial task of optimally determining the ML-ESNAE architecture and training parameters is addressed, in order to extract more efficient features from the data. Traditionally, in a ML-ESN, the number of parameters (hidden neurons, sparsity rates, weights) are randomly chosen and manually altered to achieve a minimum learning error. On one hand, this random setting may not guarantee best generalization results. On the other, it can increase the network’s complexity. In this paper, a novel bi-level evolutionary optimization approach is thus proposed for the ML-ESNAE, to deal with these challenges. The first level offers Pareto multi-objective architecture optimization, providing maximum learning accuracy while maintaining a reduced complexity target. Next, every Pareto optimal solution obtained from the first level undergoes a mono-objective weights optimization at the second level. Particle Swarm Optimization (PSO) is used as an evolutionary tool for both levels 1 and 2. An empirical study shows that the evolved ML-ESNAE produces a noticeable improvement in extracting new, more expressive data features from original ones. A number of application case studies, using a range of benchmark datasets, show that the extracted features produce excellent results in terms of classification accuracy. The effectiveness of the evolved ML-ESNAE is demonstrated for both noisy and noise-free data. In conclusion, the evolutionary ML-ESNAE is proposed as a new benchmark for the evolutionary AI and machine learning research community.

• Journal Article Type: Article
• Acceptance Date: Mar 6, 2019
• Online Publication Date: Mar 13, 2019
• Publication Date: 2019-05
• Deposit Date: Dec 8, 2020
• Journal: Neurocomputing
• Print ISSN: 0925-2312
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 341
• Pages: 195-211
• DOI: https://doi.org/10.1016/j.neucom.2019.03.012
• Keywords: Multi-Layered Echo State Network, Autoencoder, Data representation, PSO, Multi-objective optimization, Architecture optimization, Weights optimization
• Public URL: http://researchrepository.napier.ac.uk/Output/2709414