Next Article in Journal
Machine Learning Photovoltaic String Analyzer
Previous Article in Journal
Smart Contracts: A Review of Security Threats Alongside an Analysis of Existing Solutions
Open AccessArticle

Emergence of Network Motifs in Deep Neural Networks

1
Department of Civil, Environmental and Architectural Engineering, University of Padova, Via Marzolo 9, 35131 Padova, Italy
2
Department of Physics and Astronomy, University of Padova; Istituto Nazionale di Fisica Nucleare—Sezione di Padova, Via Marzolo 8, 35131 Padova, Italy
3
Department of General Psychology, University of Padova, Via Venezia 8, 35131 Padova, Italy
4
Department of Information Engineering, University of Padova, Via Gradenigo 6/b, 35131 Padova, Italy
*
Authors to whom correspondence should be addressed.
Entropy 2020, 22(2), 204; https://doi.org/10.3390/e22020204
Received: 27 December 2019 / Revised: 3 February 2020 / Accepted: 7 February 2020 / Published: 11 February 2020
(This article belongs to the Special Issue Information Transfer in Multilayer/Deep Architectures)
Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.
Keywords: deep learning; artificial neural networks; network motifs; complex systems deep learning; artificial neural networks; network motifs; complex systems
  • Externally hosted supplementary file 1
    Link: https://github.com/MatteoZambra/SM_ML__MScThesis
    Description: The above link refers to the GItHub repo where the code written to produce the results depicted in the main text. A Readme.md file provides a minimal description of the program behavior and run settings.
MDPI and ACS Style

Zambra, M.; Maritan, A.; Testolin, A. Emergence of Network Motifs in Deep Neural Networks. Entropy 2020, 22, 204.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

1
Back to TopTop