Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map

The study presents a neural network, which uses filters based on logistic mapping (LogNNet). LogNNet has a feedforward network structure, but possesses the properties of reservoir neural networks. The input weight matrix, set by a recurrent logistic mapping, forms the kernels that transform the input space to the higher-dimensional feature space. The most effective MNIST handwritten digit recognition occurs under chaotic behavior of logistic map. An advantage of LogNNet implementation on IoT Devices is the significant savings in used memory (more than 10 times) compared to other neural networks. The presented network architecture uses an array of weights with a total memory size from 1 kB to 29 kB and achieves a classification accuracy of 80.3-96.3 %.


I. INTRODUCTION
N the age of neural networks and Internet of Things (IoT) developments, the search for new neural network architectures capable of operating on devices with a small amount of memory (tens of kB RAM) becomes an urgent agenda [1], [2].The ability to perform processing of incoming information on the spot, without sending it to the cloud, is an important feature of intelligent IoT devices.Currently, effective memory redistribution algorithms in convolutional neural networks (CNN) are available and do not exceed 2kB of consumed RAM [3].However, the complexity of the algorithms leads to the large size of the program itself.An alternative approach to reduce the number of trained weights is based on physical reservoir computing (RC) [4].RC uses complex physical dynamic systems (coupled oscillators [5]- [7], memristor crossbar arrays [8], opto-electronic feedback loop [9]), or recurrent neural networks (echo state networks (ESNs) [10] and liquid state machines (LSMs) [11]), as reservoirs with rich dynamics and powerful computing capabilities.The couplings in the reservoir are not trained, but are specified in a special way.The reservoir translates the input data into a higher dimensionality of space (kernel trick [12]), and the output neural network, after training, can classify the result more correctly.The search for simple algorithms for Manuscript received ********.This work was supported by the Russian Science Foundation (grant no.16-19-00135).
simulating complex reservoir dynamics is an important research task.The current paper presents a new architecture for a neural network, where complex dynamics is simulated by the application of a logistic mapping to the weights of a multilayer feedforward network.Previously, logistic mapping was used in neural networks as activation function [13], [14] and as a model object [15], [16].

II. DATA AND METHODS
The database of handwritten numbers MNIST was used for the study, consisting of 70,000 images of handwritten digits from "0" to "9".Each image is 28 × 28 pixels in size (see Fig. 1a).The database is divided into two sets.The first set consists of 60,000 images intended for training the network, and the second set of 10,000 images is used to test the network and to calculate classification accuracy.The image is presented in grayscale with the intensity of each pixel in the range from 0 to 255.Before inputting the network, a two-dimensional image was converted into a one-dimensional array Y[i], where i = 1..784, using special transformation T-patterns.The main types of T-patterns used are presented in Fig. 1b, c, d.In addition, Tpatterns can be used for reverse transformation from a onedimensional array to a two-dimensional array.
where W1, W2 are the weight matrices, Sh is the hidden layer with the number of active elements P and the bias element Sh[0]=1, Sout is the output layer, fh is identity activation function, which is normalized in the range from -0.5 to 0.5, and fout is logistic activation function.
The weights W1 in the reservoir are set recursively, based on the logistic mapping where r is a positive parameter.Applying (2) to the elements of the weight matrix W1, the following equation is obtained The initial values of the elements of the first row ) where A and B are adjustable parameters, A = 0.3, B = 5.9, i = 0..784.
The classification of handwritten digits in the range "0" -"9" takes place according to the largest value of the element of the output layer Sout [n], where n = 0..9.The training of the weights W2 is performed by the error back-propagation method [17] with the learning rate of 0.3.The initial values of the weights W2 are set randomly between −0.5 and 0.5.
Weights W1 are not trained, and only the weights W2 of the output classifier are trained.This approach resembles the operation of a recurrent reservoir network, except the recurrence transformation is applied not to the input data, but to the elements of the matrix W1.The logistic mapping is widely known as the simplest model that demonstrates the transition to chaotic behavior through a sequence of period doubling bifurcations (Feigenbaum's script [18]).The cascade of period doubling bifurcations in the logistic mapping can be visualized using a phase-parameter diagram, which is also called a one-parameter bifurcation diagram (Fig. 3).In the text, brief information on the number of network layers and neurons is indicated in the format LogNNet-784:P:10.

III. RESULT AND DISCUSSION
The dependencies of classification accuracy on the number of training epochs for different T-patterns are shown in Fig. 4.
As T-pattern-3 provides the best results, all subsequent calculations are given with this transformation.The application of transformations to the initial images should be considered as a heuristic technique, which allows to overcome the disadvantages of back-propagation of errors learning method.Configuring a neural network is truly considered an art [19], and many authors use affine transformations [20], [21] and elastic deformations [22].The dependence of the classification accuracy on the parameter r is presented in Fig. 5.The accuracy grows with increasing r and increasing number of neurons P. At values of r = 1.65, 1.805, 1.885, 1.967, 1.992 local maximums of classification accuracy is observed.The shape of the graph resembles the dependence of the Lyapunov exponent on r [18], and indicates the importance of the chaotic behavior of the logistic mapping in the recognition process.The important role of the chaotic dynamics of the reservoir was highlighted in the studies on ESNs (setting the spectral radius) [10], [23], and LSMs (setting separation property) [11].
The shape of distribution of weights W1 for different reservoir neurons, at r = 1.885, is presented in Fig. 6 (for convenience of visualization, the inverse transformation Tpattern-1 was applied, and the element W1[0][p] was placed in the upper left corner).Due to the formation of matrix elements through the logistic relation ( 2), the kernels shape changes with each step p. Kernels transform the input space to the higherdimensional feature space.The result of the kernels operation is released by the reservoir neurons and transmitted to the output classifier.The advantage of LogNNet is that the weights W1 do not have to be stored in a large two-dimensional array with the number of elements N = 785 × P. The weights can be sequentially calculated during the algorithm operation, at the training and testing stages.For calculation, only three parameters r, A and B should be known (Algorithm-1), and only one memory cell N = 1 should be reserved with the capacity MeW1 = 4 B (a single type occupies 4 bytes).In practice, it is convenient to reserve a one-dimensional array W1' [i], where i = 0..784 and recalculate its values with a subsequent increase in p, W1'[i] = 1-r•(W1'[i]) 2 (Algorithm-2).In this case, the maximum reserved memory is estimated as MeW1 ~ 3 kB.If each element W1 would be stored separately in memory, then, for P = 100, N = 78500, the array would occupy the memory MeW1 ~ 306 kB (Algorithm-3).Therefore, memory savings to use Alg.-1 and Alg.-2 are obvious.To evaluate the total memory, it is necessary to sum over all layers of the network MeW = MeW1+ MeW2+.. MeWx.
Table 1 gives comparative estimates of the classification accuracy and the amount of allocated memory for the weight arrays for various network configurations, including for LogNNet with 1-2 layer output classifier (Alg.-2,r = 1.885), and for 1-2 layer feedforward networks Fig.With an increase in the number of neurons P, the use of a large array W1 becomes less beneficial in comparison with the calculations of W1'.This result is related to an increase in the time for accessing memory cells.

IV. CONCLUSION
The study presents the neural network LogNNet, which uses filters based on logistic mapping to achieve high accuracy indicators for the classification of handwritten digits from the MNIST database.The network uses less memory for weight arrays than many well-known network configurations.The future research directions may lie in the search for other chaotic mappings and initial conditions (3) for more efficient operation of LogNNet.

Fig. 1 .
An example of a 28 × 28 pixel image of the handwritten digit "8" from the MNIST database (a).Patterns for transforming the image array: T-pattern-1 (b), T-pattern-2 (c), T-pattern-3 (d).T-pattern-1 performs a column-by-column input of the image, T-pattern-2 converts the image in a spiral (clockwise) form, and T-pattern-3 is a combination of line-by-line input of the central part of the image, with the addition of borders in a spiral (clockwise) form.After the T-pattern transformation, the values of the array Y are normalized by dividing each element of the array by 255, and a bias element Y [0] = 1 is added.A. Velichko is with the Institute of Physics and Technology, Petrozavodsk State University, 31 Lenina str., Petrozavodsk 185910, Russia (e-mail: velichko@petrsu.ru).Neural Network for Low-Memory IoT Devices and MNIST Image Recognition Using Kernels Based on Logistic Map Andrei Velichko I A. Network architecture The network architecture used in the current study is presented in Fig. 2a.Data processing is performed similarly to the feedforward network (Fig 2b), according to the equations h

Fig. 4 .
Fig. 4. The dependence of the classification accuracy on the number of epochs, for different T-patterns.The number of reservoir neurons is P = 25 (a), P = 100 (b) and r = 1.885.

Table 2
summarises a comparison of the processing times of a single image by the LogNNet network on the IoT device Raspberry Pi 4 for a different number of neurons in reservoir P. The execution times (talg2 and talg3) reflect the application of the one-dimensional array W1' (Alg.-2) and the two-dimensional array W1 (Alg.-3),respectively.

TABLE II COMPARISON
OF PROCESSING TIMES OF ONE IMAGE ON RASPBERRY PI 4.