Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

Mocanu, Decebal Constantin; Mocanu, Elena; Stone, Peter; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio

Abstract

Through the success of deep learning in various domains, artificial neural networks are currently among the most used artificial intelligence methods. Taking inspiration from the network properties of biological neural networks (e.g. sparsity, scale-freeness), we argue that (contrary to general practice) artificial neural networks, too, should not have fully-connected layers. Here we propose sparse evolutionary training of artificial neural networks, an algorithm which evolves an initial sparse topology (Erdős–Rényi random graph) of two consecutive layers of neurons into a scale-free topology, during learning. Our method replaces artificial neural networks fully-connected layers with sparse ones before training, reducing quadratically the number of parameters, with no decrease in accuracy. We demonstrate our claims on restricted Boltzmann machines, multi-layer perceptrons, and convolutional neural networks for unsupervised and supervised learning on 15 datasets. Our approach has the potential to enable artificial neural networks to scale up beyond what is currently possible.

Citation

Mocanu, D. C., Mocanu, E., Stone, P., Nguyen, P. H., Gibescu, M., & Liotta, A. (2018). Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature Communications, 9(1), Article 2383. https://doi.org/10.1038/s41467-018-04316-3

• Journal Article Type: Article
• Acceptance Date: Apr 20, 2018
• Online Publication Date: Jun 19, 2018
• Publication Date: Jun 19, 2018
• Deposit Date: Jul 29, 2019
• Publicly Available Date: Jul 30, 2019
• Journal: Nature Communications
• Electronic ISSN: 2041-1723
• Publisher: Nature Publishing Group
• Peer Reviewed: Peer Reviewed
• Volume: 9
• Issue: 1
• Article Number: 2383
• DOI: https://doi.org/10.1038/s41467-018-04316-3
• Keywords: General Biochemistry, Genetics and Molecular Biology; General Physics and Astronomy; General Chemistry
• Public URL: http://researchrepository.napier.ac.uk/Output/1995597
• Publisher URL: https://doi.org/10.1038%2Fs41467-018-04316-3