Skip to main content

Research Repository

Advanced Search

A topological insight into restricted Boltzmann machines

Mocanu, Decebal Constantin; Mocanu, Elena; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio

Authors

Decebal Constantin Mocanu

Elena Mocanu

Phuong H. Nguyen

Madeleine Gibescu

Antonio Liotta



Abstract

Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.

Citation

Mocanu, D. C., Mocanu, E., Nguyen, P. H., Gibescu, M., & Liotta, A. (2016). A topological insight into restricted Boltzmann machines. Machine Learning, 104(2-3), 243-270. https://doi.org/10.1007/s10994-016-5570-z

Journal Article Type Article
Acceptance Date Jun 16, 2016
Online Publication Date Jul 15, 2016
Publication Date 2016-09
Deposit Date Jul 29, 2019
Publicly Available Date Jul 30, 2019
Journal Machine Learning
Print ISSN 0885-6125
Electronic ISSN 1573-0565
Publisher Springer
Peer Reviewed Peer Reviewed
Volume 104
Issue 2-3
Pages 243-270
DOI https://doi.org/10.1007/s10994-016-5570-z
Keywords Deep learning; Sparse restricted Boltzmann machines; Complex networks; Scale-free networks; Small-world networks
Public URL http://researchrepository.napier.ac.uk/Output/2006063
Contract Date Jul 29, 2019

Files

A topological insight into restricted Boltzmann machines (1.3 Mb)
PDF

Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/

Copyright Statement
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.









Downloadable Citations