Decebal Constantin Mocanu
A topological insight into restricted Boltzmann machines
Mocanu, Decebal Constantin; Mocanu, Elena; Nguyen, Phuong H.; Gibescu, Madeleine; Liotta, Antonio
Authors
Elena Mocanu
Phuong H. Nguyen
Madeleine Gibescu
Antonio Liotta
Abstract
Restricted Boltzmann Machines (RBMs) and models derived from them have been successfully used as basic building blocks in deep artificial neural networks for automatic features extraction, unsupervised weights initialization, but also as density estimators. Thus, their generative and discriminative capabilities, but also their computational time are instrumental to a wide range of applications. Our main contribution is to look at RBMs from a topological perspective, bringing insights from network science. Firstly, here we show that RBMs and Gaussian RBMs (GRBMs) are bipartite graphs which naturally have a small-world topology. Secondly, we demonstrate both on synthetic and real-world datasets that by constraining RBMs and GRBMs to a scale-free topology (while still considering local neighborhoods and data distribution), we reduce the number of weights that need to be computed by a few orders of magnitude, at virtually no loss in generative performance. Thirdly, we show that, for a fixed number of weights, our proposed sparse models (which by design have a higher number of hidden neurons) achieve better generative capabilities than standard fully connected RBMs and GRBMs (which by design have a smaller number of hidden neurons), at no additional computational costs.
Citation
Mocanu, D. C., Mocanu, E., Nguyen, P. H., Gibescu, M., & Liotta, A. (2016). A topological insight into restricted Boltzmann machines. Machine Learning, 104(2-3), 243-270. https://doi.org/10.1007/s10994-016-5570-z
Journal Article Type | Article |
---|---|
Acceptance Date | Jun 16, 2016 |
Online Publication Date | Jul 15, 2016 |
Publication Date | 2016-09 |
Deposit Date | Jul 29, 2019 |
Publicly Available Date | Jul 30, 2019 |
Journal | Machine Learning |
Print ISSN | 0885-6125 |
Electronic ISSN | 1573-0565 |
Publisher | Springer |
Peer Reviewed | Peer Reviewed |
Volume | 104 |
Issue | 2-3 |
Pages | 243-270 |
DOI | https://doi.org/10.1007/s10994-016-5570-z |
Keywords | Deep learning; Sparse restricted Boltzmann machines; Complex networks; Scale-free networks; Small-world networks |
Public URL | http://researchrepository.napier.ac.uk/Output/2006063 |
Contract Date | Jul 29, 2019 |
Files
A topological insight into restricted Boltzmann machines
(1.3 Mb)
PDF
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Downloadable Citations
About Edinburgh Napier Research Repository
Administrator e-mail: repository@napier.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2025
Advanced Search