S. Squartini
Preprocessing based solution for the vanishing gradient problem in recurrent neural networks
Squartini, S.; Hussain, A.; Piazza, F.
Abstract
In this paper, a possible solution to the vanishing gradient problem in recurrent neural networks (RNN) is proposed. The main idea consists of pre-processing the signal (a time series typically) through a wavelet decomposition, in order to separate the short term information from the long term one, and treating each scale by different RNNs. The partial results concerning all the different scales of time and frequencies are combined by another 'expert' (a nonlinear structure typically) in order to achieve the final goal. This new approach is distinct from the other ones reported in the literature to-date, as it tends to simplify the RNN's learning, working directly at the signal level and avoiding relevant changes in network architecture and learning techniques. The overall system (called the recurrent multiscale network, RMN) is described and its performance tested through typical tasks, namely the latching problem and time series prediction.
Presentation Conference Type | Conference Paper (Published) |
---|---|
Conference Name | 2003 IEEE International Symposium on Circuits and Systems (ISCAS) |
Start Date | May 25, 2003 |
End Date | May 28, 2003 |
Online Publication Date | Jun 20, 2003 |
Publication Date | 2003 |
Deposit Date | Oct 15, 2019 |
Publisher | Institute of Electrical and Electronics Engineers |
ISBN | 0-7803-7761-3 |
DOI | https://doi.org/10.1109/ISCAS.2003.1206412 |
Public URL | http://researchrepository.napier.ac.uk/Output/1793740 |
You might also like
Applications of Deep Learning and Reinforcement Learning to Biological Data
(2018)
Journal Article
Guided Policy Search for Sequential Multitask Learning
(2018)
Journal Article
Learning Latent Features With Infinite Nonnegative Binary Matrix Trifactorization
(2018)
Journal Article
Cross-modality interactive attention network for multispectral pedestrian detection
(2018)
Journal Article
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search