Decebal Constantin Mocanu
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
Mocanu, Decebal Constantin; Mocanu, Elena; Stone, Peter; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio
Authors
Elena Mocanu
Peter Stone
Phuong H. Nguyen
Madeleine Gibescu
Antonio Liotta
Abstract
Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.
Citation
Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., & Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1), Article 2383. https://doi.org/10.1038/s41467-018-04316-3
Journal Article Type | Article |
---|---|
Acceptance Date | Apr 20, 2018 |
Online Publication Date | Jun 19, 2018 |
Publication Date | Jun 19, 2018 |
Deposit Date | Jul 29, 2019 |
Publicly Available Date | Jul 30, 2019 |
Journal | Nature Communications |
Electronic ISSN | 2041-1723 |
Publisher | Nature Publishing Group |
Peer Reviewed | Peer Reviewed |
Volume | 9 |
Issue | 1 |
Article Number | 2383 |
DOI | https://doi.org/10.1038/s41467-018-04316-3 |
Keywords | General Biochemistry, Genetics and Molecular Biology; General Physics and Astronomy; General Chemistry |
Public URL | http://researchrepository.napier.ac.uk/Output/1995597 |
Publisher URL | https://doi.org/10.1038%2Fs41467-018-04316-3 |
Contract Date | Jul 29, 2019 |
Files
Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science
(1.6 Mb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
© The Author(s) 2018
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2025
Advanced Search