Skip to main content

Research Repository

Advanced Search

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

Mocanu, Decebal Constantin; Mocanu, Elena; Stone, Peter; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio

Authors

Decebal Constantin Mocanu

Elena Mocanu

Peter Stone

Phuong H. Nguyen

Madeleine Gibescu

Antonio Liotta



Abstract

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

Journal Article Type Article
Acceptance Date Apr 20, 2018
Online Publication Date Jun 19, 2018
Publication Date Jun 19, 2018
Deposit Date Jul 29, 2019
Publicly Available Date Jul 30, 2019
Journal Nature Communications
Electronic ISSN 2041-1723
Publisher Nature Publishing Group
Peer Reviewed Peer Reviewed
Volume 9
Issue 1
Article Number 2383
DOI https://doi.org/10.1038/s41467-018-04316-3
Keywords General Biochemistry, Genetics and Molecular Biology; General Physics and Astronomy; General Chemistry
Public URL http://researchrepository.napier.ac.uk/Output/1995597
Publisher URL https://doi.org/10.1038%2Fs41467-018-04316-3
Contract Date Jul 29, 2019

Files

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science (1.6 Mb)
PDF

Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/

Copyright Statement
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
© The Author(s) 2018









Downloadable Citations