# Rosenblatt’s First Theorem and Frugality of Deep Learning

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

**Theorem**

**1.**

## 2. Formal Problem Statement

## 3. Shallow Neural Network Solution

**Theorem**

**2.**

**Example**

**1.**

**Example**

**2.**

· | ${P}_{0}$ | ${P}_{1}$ | ${P}_{2}$ | ${P}_{3}$ | ${P}_{4}$ | ${P}_{5}$ | |

${P}_{0}$ | ${P}_{0}$ | ${P}_{1}$ | ${P}_{2}$ | ${P}_{3}$ | ${P}_{4}$ | ${P}_{5}$ | |

${P}_{1}$ | ${P}_{1}$ | ${P}_{0}$ | ${P}_{3}$ | ${P}_{2}$ | ${P}_{5}$ | ${P}_{4}$ | |

${P}_{2}$ | ${P}_{2}$ | ${P}_{4}$ | ${P}_{0}$ | ${P}_{5}$ | ${P}_{1}$ | ${P}_{3}$ | |

${P}_{3}$ | ${P}_{3}$ | ${P}_{5}$ | ${P}_{1}$ | ${P}_{4}$ | ${P}_{0}$ | ${P}_{2}$ | |

${P}_{4}$ | ${P}_{4}$ | ${P}_{2}$ | ${P}_{5}$ | ${P}_{0}$ | ${P}_{3}$ | ${P}_{1}$ | |

${P}_{5}$ | ${P}_{5}$ | ${P}_{3}$ | ${P}_{4}$ | ${P}_{1}$ | ${P}_{2}$ | ${P}_{0}$ |

## 4. Deep Neural Network Solution

**Theorem**

**3.**

## 5. Neural Network for $\mathit{r}$-Bounded Problem

**Theorem**

**4.**

**Theorem**

**5.**

## 6. Conclusions and Outlook

- A shallow neural network combined from elementary Rosenblatt’s perceptrons can solve the travel maze problem in accordance with Rosenblatt’s first theorem.
- The complexity of the constructed solution of the travel maze problem for a deep network is much smaller than for the solution provided by the shallow network (the main terms are $2L{n}^{2}$ versus ${(n!)}^{L}$ for the numbers of neurons and $2{L}^{3}$ versus $L{n}^{2}{(n!)}^{L}$ for the numbers of connections).

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Rosenblatt, F. Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms; Spartan Books: Washington, DC, USA, 1962. [Google Scholar]
- Venkatesh, B.; Anuradha, J. A review of feature selection and its methods. Cybern. Inf. Technol.
**2019**, 19, 3–26. [Google Scholar] [CrossRef] [Green Version] - Al-Tashi, Q.; Abdulkadir, S.J.; Rais, H.M.; Mirjalili, S.; Alhussian, H. Approaches to multi-objective feature selection: A systematic literature review. IEEE Access
**2020**, 8, 125076–125096. [Google Scholar] [CrossRef] - Rong, M.; Gong, D.; Gao, X. Feature selection and its use in big data: Challenges, methods, and trends. IEEE Access
**2019**, 7, 19709–19725. [Google Scholar] [CrossRef] - Minsky, M.; Papert, S. Perceptrons; MIT Press: Cambridge, MA, USA, 1988. [Google Scholar]
- Seifert, B.G. Binary classifiers, pcrceptrons and connectedness in metric spaces and graphs. J. Pure Appl. Algebra
**1992**, 83, 197–203. [Google Scholar] [CrossRef] [Green Version] - Albergante, L.; Mirkes, E.; Bac, J.; Chen, H.; Martin, A.; Faure, L.; Barillot, E.; Pinello, L.; Gorban, A.; Zinovyev, A. Robust and Scalable Learning of Complex Intrinsic Dataset Geometry via ElPiGraph. Entropy
**2020**, 22, 296. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Bishop, J.M. History and Philosophy of Neural Networks. In Encyclopedia of Life Support Systems (EOLSS): Computational Intelligence; Ishibuchi, H., Ed.; UNESCO Eolss Publishers: Paris, France, 2015. [Google Scholar]
- Ito, Y. Finite mapping by neural networks and truth functions. Math. Sci.
**1992**, 17, 69–77. [Google Scholar] - Kůrková, V.; Savický, P.; Hlavávcková, K. Representations and rates of approximation of real-valued Boolean functions by neural networks. Neural Netw.
**1998**, 11, 651–659. [Google Scholar] [CrossRef] - Kainen, P.C.; Kůrková, V. An integral upper bound for neural network approximation. Neural Comput.
**2009**, 21, 2970–2989. [Google Scholar] [CrossRef] [PubMed] - Kůrková, V. Limitations of Shallow Networks. In Recent Trends in Learning From Data. Studies in Computational Intelligence; Oneto, L., Navarin, N., Sperduti, A., Anguita, D., Eds.; Springer: Cham, Switzerland, 2020; Volume 896, pp. 129–154. [Google Scholar] [CrossRef]
- Berner, J.; Grohs, P.; Kutyniok, G.; Petersen, P. The Modern Mathematics of Deep Learning. arXiv
**2021**, arXiv:2105.04026. [Google Scholar] - Gorban, A.N.; Mirkes, E.M.; Tyukin, I.Y. How deep should be the depth of convolutional neural networks: A backyard dog case study. Cogn. Comput.
**2020**, 12, 388–397. [Google Scholar] [CrossRef] [Green Version] - Weiss, E.; Schwartz, O. Computation of Matrix Chain Products on Parallel Machines. In Proceedings of the 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), Rio de Janeiro, Brazil, 20–24 May 2019; pp. 491–500. [Google Scholar] [CrossRef]
- Alman, J.; Vassilevska Williams, V. A Refined Laser Method and Faster Matrix Multiplication. In Proceedings of the 2021 ACM-SIAM Symposium on Discrete Algorithms (SODA), Virtual, 10–13 January 2021; pp. 522–539. [Google Scholar] [CrossRef]
- Gupta, B.; Sehgal, S. Survey on techniques used in autonomous maze solving robot. In Proceedings of the 2014 5th International Conference-Confluence The Next Generation Information Technology Summit (Confluence), Noida, India, 25–26 September 2014; pp. 323–328. [Google Scholar] [CrossRef]
- Wu, C.M.; Liaw, D.C.; Lee, H.T. A method for finding the routes of mazes. In Proceedings of the 2018 International Automatic Control Conference (CACS), Taoyuan, Taiwan, 4–7 November 2018; pp. 1–4. [Google Scholar] [CrossRef]
- Golovenkin, S.E.; Bac, J.; Chervov, A.; Mirkes, E.M.; Orlova, Y.V.; Barillot, E.; Gorban, A.N.; Zinovyev, A. Trajectories, bifurcations, and pseudo-time in large clinical datasets: Applications to myocardial infarction and diabetes data. GigaScience
**2020**, 9, giaa128. [Google Scholar] [CrossRef] [PubMed] - Chen, H.; Albergante, L.; Hsu, J.Y.; Lareau, C.A.; Lo Bosco, G.; Guan, J.; Zhou, S.; Gorban, A.N.; Bauer, D.E.; Aryee, M.J.; et al. Single-cell trajectories reconstruction, exploration and mapping of omics data with STREAM. Nat. Commun.
**2019**, 10, 1–14. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Bianchini, M.; Scarselli, F. On the complexity of neural network classifiers: A comparison between shallow and deep architectures. IEEE Trans. Neural Netw. Learn. Syst.
**2014**, 25, 1553–1565. [Google Scholar] [CrossRef] [PubMed] - Gorban, A.N. Neuroinformatics: What are us, where are we going, how to measure our way? A lecture given at the USA-NIS Neurocomputing opportunities workshop, Washington DC, July 1999 (Associated with IJCNN’99). arXiv
**2003**, arXiv:cond-mat/0307346. [Google Scholar] [CrossRef]

**Figure 1.**Rosenblatt’s elementary perceptron (re-drawn from the Rosenblatt book [1]).

**Figure 2.**Have we chosen the right delicacies (

**right**) for our guests (

**left**)? (

**a**) A prototype travel maze problem. (

**b**) A simplified form of the problem with piece-wise linear paths for further formal description (Section 2). Complexity depends on the number of guests and the number of links in a path.

**Figure 3.**Game diagram with L stages (a formalized and simplified version of the travel maze problem).

**Figure 4.**A shallow (fully connected) neural network for the travel maze problem (Figure 3). It differs from the classical elementary perceptron (Figure 1) by ${n}^{2}$ output neurons instead of one and can be considered as a union of ${n}^{2}$ elementary perceptrons with joint retina and hidden layer of A-elements.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kirdin, A.; Sidorov, S.; Zolotykh, N.
Rosenblatt’s First Theorem and Frugality of Deep Learning. *Entropy* **2022**, *24*, 1635.
https://doi.org/10.3390/e24111635

**AMA Style**

Kirdin A, Sidorov S, Zolotykh N.
Rosenblatt’s First Theorem and Frugality of Deep Learning. *Entropy*. 2022; 24(11):1635.
https://doi.org/10.3390/e24111635

**Chicago/Turabian Style**

Kirdin, Alexander, Sergey Sidorov, and Nikolai Zolotykh.
2022. "Rosenblatt’s First Theorem and Frugality of Deep Learning" *Entropy* 24, no. 11: 1635.
https://doi.org/10.3390/e24111635