An Overview of Machine Learning within Embedded and Mobile Devices–Optimizations and Applications

Embedded systems technology is undergoing a phase of transformation owing to the novel advancements in computer architecture and the breakthroughs in machine learning applications. The areas of applications of embedded machine learning (EML) include accurate computer vision schemes, reliable speech recognition, innovative healthcare, robotics, and more. However, there exists a critical drawback in the efficient implementation of ML algorithms targeting embedded applications. Machine learning algorithms are generally computationally and memory intensive, making them unsuitable for resource-constrained environments such as embedded and mobile devices. In order to efficiently implement these compute and memory-intensive algorithms within the embedded and mobile computing space, innovative optimization techniques are required at the algorithm and hardware levels. To this end, this survey aims at exploring current research trends within this circumference. First, we present a brief overview of compute intensive machine learning algorithms such as hidden Markov models (HMM), k-nearest neighbors (k-NNs), support vector machines (SVMs), Gaussian mixture models (GMMs), and deep neural networks (DNNs). Furthermore, we consider different optimization techniques currently adopted to squeeze these computational and memory-intensive algorithms within resource-limited embedded and mobile environments. Additionally, we discuss the implementation of these algorithms in microcontroller units, mobile devices, and hardware accelerators. Conclusively, we give a comprehensive overview of key application areas of EML technology, point out key research directions and highlight key take-away lessons for future research exploration in the embedded machine learning domain.


Introduction
Embedded computing systems are fast proliferating every aspect of human endeavor today, finding useful application in areas such as wearable systems for health monitoring, wireless systems for military surveillance, networked systems as found in the internet of things (IoT), smart appliances for home automation, antilock braking systems in automobiles, amongst others [1].Recent research trends in computing technology have seen a merger of machine learning methods and embedded computing for diverse applications.For example, to target the hostility and dynamism of mobile ad hoc networks (MANETs), Haigh et al. [2] explored enhancing the self-configuration of a MANET using machine learning techniques.Besides, the recent breakthroughs of deep learning models in application areas such as computer vision [3][4][5][6][7], speech recognition [8,9], language translation, and processing [10,11], robotics, and healthcare [12] make this overlap a key research direction for the development of next-generation embedded devices.Thus, this has opened a research thrust between embedded devices and machine learning models termed "Embedded Machine Learning" where machine learning models are executed within resource-constrained environments [13].This research surveys key issues within this convergence of embedded systems and machine learning.
Machine learning methods such as SVMs for feature classification [14], CNNs for intrusion detection [15], and other deep learning techniques, require high computational and memory resources for effective training and inferencing [16][17][18][19].General-purpose CPUs, even with their architectural modification over the years, including pipelining, deep cache memory hierarchies, multicore enhancements, etc., cannot meet the high computational demand of deep learning models.However, graphic processing units (GPUs), due to their high floating-point performance and thread-level parallelism, are more suitable for training deep learning models [13].Extensive research is actively being carried out to develop suitable hardware acceleration units using FPGAs [20][21][22][23][24][25][26], GPUs, ASICs, and TPUs to create heterogeneous and sometimes distributed systems to meet up the high computational demand of deep learning models.At both the algorithm and hardware levels, optimization techniques for classical machine learning and deep learning algorithms are being investigated such as pruning, quantization, reduced precision, hardware acceleration, etc. to enable the efficient execution of machine learning models in mobile devices and other embedded systems [27][28][29].
The convergence of machine learning methods and embedded systems in which computationally intensive machine learning models target the resource-constrained embedded environment has opened a plethora of opportunities for research in computing technology.Although EML is just in its cradle, quite some work has been done to: (1) optimize different machine learning models to fit into resource-limited environments, (2) develop efficient hardware architectures (acceleration units) using custom chipsets to accelerate the implementation of these algorithms, and (3) create novel and innovative specialized hardware architectures to meet the high-performance requirements of these models.Thus, there is a need to bring these perspectives together to provide the interested researcher with the fundamental concepts of EML and further provide the computer architect with insights and possibilities within this space.
Interestingly, several surveys have been carried out to achieve this.For example references [30,31] survey deep learning concepts, models, and optimizations.In these surveys, little consideration is given to the hardware architectural design, which is a key concern in developing efficient machine learning systems.Pooja [32] surveys recent trends in the hardware architectural design for machine learning applications, using the tensor processing unit as a case study.However, the research did not explore the different DNN architectures and just skimmed through some deep learning optimization techniques.Jiasi and Xukan [33], in their review, explored deep learning concepts, narrowing down on inference at end devices but do not compare different embedded chipset architectures to inform which architecture or optimization is appropriate for the different DNN models.They also present applications of deep learning in end devices.They, however, only explored a type of deep learning model (DNNs) and did not discuss other deep learning models (CNN, RNN), which have gained attention in recent times.Sergio et al. [24] have carried out a comprehensive survey on ML in embedded and mobile devices, presenting ML concepts and techniques for optimization and also investigated different application areas.They, however, also do not explore other models of DNNs or make appropriate tradeoffs.To address these drawbacks, this survey presents key compute and memory intensive machine learning algorithms, which are the HMM, k-NN, SVM, GMM, and the different shades of DNNs (CNN and RNN), and present hardware-based and algorithm-based optimization techniques required to compress these algorithms within resource-constrained environments.To sum up, the authors decided to consider diverse application areas where machine learning has been utilized in proffering solutions to stringent problems in this big data era.A comprehensive layout of this survey is presented in Figure 1.
Sensors 2021, 21, 4412 3 of 44 scribes machine learning in resource-constrained environments (MCUs, mobile devices, acceleration units, and TinyML).Section 4 presents challenges and possible optimization opportunities in embedded machine learning.Section 5 provides diverse areas of applications of embedded machine learning technology, while Section 6 presents plausible research directions, open issues, and lessons learned.In Section 7, a concise conclusion is presented.Machine    The key contributions of this survey are as follows: i.
We present a survey of machine learning models commonly used in embedded systems applications.ii.
We describe an overview of compute-intensive machine learning models such as HMMs, k-NNs, SVMs, GMMs, and DNNs.iii.
We provide an overview of different optimization schemes adopted for these algorithms.iv.
We present an overview of the implementation of these algorithms within resourcelimited environments such as MCUs, mobile devices, hardware accelerators, and TinyML.v.
We survey the challenges faced in embedded machine learning and review different optimization techniques to enhance the execution of deep learning models within resource-constrained environments.vi.
We present diverse application areas of embedded machine learning, identify open issues and highlight key lessons learned for future research exploration.
The remainder of this paper is organized as follows.Section 2 presents embedded machine learning algorithms and specific optimization techniques, while Section 3 describes machine learning in resource-constrained environments (MCUs, mobile devices, acceleration units, and TinyML).Section 4 presents challenges and possible optimization opportunities in embedded machine learning.Section 5 provides diverse areas of applica-tions of embedded machine learning technology, while Section 6 presents plausible research directions, open issues, and lessons learned.In Section 7, a concise conclusion is presented.

Embedded Machine Learning Techniques
Machine learning is a branch of artificial intelligence that describes techniques through which systems learn and make intelligent decisions from available data.Machine learning techniques can be classified under three major groups, which are supervised learning, unsupervised learning, and reinforcement learning as described in Table 1.In supervised learning, labeled data can be learned while in unsupervised learning, hidden patterns can be discovered from unlabeled data, and in reinforcement learning, a system may learn from its immediate environment through the trial and error method [34][35][36].The process of learning is referred to as the training phase of the model and is often carried out using computer architectures with high computational resources such as multiple GPUs.After learning, the trained model is then used to make intelligent decisions on new data.This process is referred to as the inference phase of the implementation.The inference is often intended to be carried out within user devices with low computational resources such as IoT and mobile devices.

Scope of ML Techniques Overview
In recent times, machine learning techniques have been finding useful applications in various research areas and particularly in embedded computing systems.In this research, we surveyed recent works of literature concerning machine learning techniques implemented within resource-scarce environments such as mobile devices and other IoT devices between 2014 and 2020.We present the results of this survey in a tabular form given in Table 2. Our survey revealed that of all available machine learning techniques, SVMs, GMMs, DNNs, k-NNs, HMMs, decision trees, logistic regression, k-means, and naïve Bayes are common techniques adopted for embedded and mobile applications.Naïve Bayes and decision trees have low complexity in terms of computation and memory costs and thus do not require innovative optimizations as pointed out by Sayali and Channe [37].Logistic regression algorithms are computationally cheaper than naïve Bayes and decision trees, meaning they have even lower complexity [38].HMMs, k-NNs, SVMs, GMMs, and DNNs are however computationally and memory intensive and hence, require novel optimization techniques to be carried out to be efficiently squeezed within resource-limited environments.We have thus limited our focus to these compute intensive ML models and discuss state-of-the-art optimization techniques through which these algorithms may be efficiently implemented within resource-constrained environments.

Hidden Markov Models
Hidden Markov Model is an unsupervised machine learning technique based on augmenting the Markov chain [85].The Markov chain is a technique that describes the probability of a sequence of events from a set of random variables.HMMs have been successfully adopted for speech recognition, activity recognition, and gesture tracking applications [86].In [56], Patil and Thorat adopt HMM within an embedded device for detecting diseases in grapes.Charissa and Song-bae in [61] implemented HMM using a smartphone for recognizing human activities.HMMs are however compute and memory intensive and thus require some optimization techniques for effective execution in resourcelimited environments [87].

The HMM Algorithm
The HMM is an algorithm that extracts meaningful information from available data through observing a sequence of "hidden states" or "hidden classes" in the data and can subsequently make accurate predictions of future states based on the current state [85].Five important components that make up the hidden Markov model are the number of states, number of distinct observations, the state transition model, observation model, and initial state distribution [85].To determine the probability of observations, a forward algorithm is adopted, while to predict the sequence of hidden states in the available data, the Viterbi algorithm is used.The learning phase of the HMM is carried out using the Baum-Welch Algorithm or the forward-backward algorithm [86].An overview of these problems and algorithms is given by Equations ( 1)-( 3), as defined in Table 3.

Some HMM Optimization Schemes
Although HMMs are suitable for different applications, they require a large amount of computational and memory resources for efficient implementation.Embedded and mobile devices are however resource-scarce environments and thus require novel optimization schemes to be carried out for the efficient execution of HMMs.In [87], Toth and Nemeth presented an optimized HMM to target smartphone environments for speech synthesis.They optimize the HMM by selecting optimal parameters and implemented the model using fixed-point arithmetic instead of computation-intensive floating-point arithmetic.Fu et al. [88] proposed a series of optimization techniques including parameter reduction using decision tree-based clustering, model compression, feature size reduction, and fixedpoint arithmetic implementation.Their optimized HMM target resource-scarce embedded platforms for speech synthesis.A list of optimizations is presented in Table 4.

Reference
Optimization Scheme Application Comments [87] Optimal HMM parameter selection reduced precision.

Speech Synthesis
This technique reduces general computation time and memory footprint however, fixed-point representation introduces accuracy errors to synthesizing. [88] HMM parameter reduction, model compression, feature vector size reduction, reduced precision

Speech Synthesis
This research successfully compressed the HMM from 293 MB to 3.61 MB.However, the accuracy of the speech synthesis is just fair owing to huge parameter reduction.

k-Nearest Neighbours
k-NN is a non-parametric and lazy supervised machine technique often adopted for classification problems e.g., text categorization [89].k-NN algorithms have been adopted in several embedded applications.For example, Hristo et al. [41] develop a mobile device fingerprinting system based on the k-NN algorithm.Additionally, Sudip et al. [76] develop a smartphone-based health monitoring system using the k-NN model.A smartphone-based data mining system is presented in [60], for fall detection using a k-NN algorithm.k-NN algorithms are also memory and compute intensive and require appropriate optimizations for resource-scarce environments.

The k-NN Algorithm
The k-NN algorithm unlike other ML approaches is a lazy learner because it does not use specialized training data to generalize, rather it uses all available data to classify.It is also non-parametric because it does not make assumptions from available data [40].The Algorithm is such that computes the distance between the input data and other data points and using a predefined "k value", classification is done by estimating proximity.The important Equations ( 4) and ( 5) that describe the k-NN model are defined in Table 5.
Table 5. Important equations involved in k-NNs.

Function Definitions Equation
k-NN Prediction Function ŷ is the predicted output, x is input and ρ is Top 1 and σ is the identity function.
Euclidean distance D(p, q) is the Euclidean distance, p and q are subjects to be compared with n characteristics. (5)

Some k-NN Optimization Schemes
k-NN models require the entire input data to be stored in memory for prediction to be done hence they are memory and compute intensive [70].To improve the efficiency of the k-NN model, Li et al. [89] proposed an improved k-NN that uses different k values for different categories instead of a predefined fixed k value.Norouzi et al. [90] investigated the optimization of k-NN algorithms by mapping input features to binary codes which are very memory efficient.There have also been some hardware-oriented optimization schemes to accelerate k-NN models.Saikia et al. [91].Mohsin and Perera [55] developed an acceleration unit to increase the execution speed of k-NN models in resource-scarce environments.Also, Gupta et al. [70] proposed a modified k-NN model termed ProtoNN which is a highly compressed k-NN model suitable for resource-limited IoT devices.Table 6 describes some k-NN optimization schemes.[70]

Binary and Multi-class classification
The k-NN model is highly compressed with good accuracy

Support Vector Machines
Support vector machine is a supervised machine learning technique based on Vapnik's statistical learning theory often adopted for object classification problems.SVMs have been successfully applied to regression, ranking, and clustering problems [92].Also, SVMs have been useful in the prediction of the power and performance, auto-tuning, and runtime scheduling of high-performance applications [93].In [2] for example, SVM is adopted in maintaining a near-optimal configuration of a MANET.Also, SVMs are used in the design and development of a low-cost and energy-efficient intelligent sensor [94].SVMs are, however, computationally and memory intensive and thus require hardware acceleration units to be effectively executed in resource-limited situations.In [95], the FPGA hardware implementation of an SVM is surveyed with optimization techniques.

The SVM Algorithm
An SVM is a linear or non-linear classifier that can identify two distinct objects by separating them into two unique classes with high accuracy.The SVM is then trained, during which a hyperplane is developed to separate the data belonging to each unique class.These hyperplane samples are referred to as "support vectors," which are then used to classify new data.The problem equation for training an SVM is given in Equation ( 6): where ∝ i ∝ j are Lagrange Multipliers, k x i , x j are the kernel functions, x and y are positions, W is the quadratic function.
The algorithm flow is given in the pseudo-code, as shown in Algorithm 1.

Algorithm 1. Pseudocode for training a support vector machine
Require: X and y loaded with training labeled data, α ← 0 or α ← partially trained SVM 1: C ← some value (10 for example) 2: repeat 3: for all {x i y i }, {x i y i } do 4: Optimize ∝ i and ∝ j 5: end for 6: until no changes in α or other resource constraint criteria met Ensure: Retain only the support vectors (∝ i > 0) The implementation of training, testing, and predicting phases of an SVM involves kernel functions that are the dot products and the Gaussian kernel functions.These computations make SVM computationally intensive.Additionally, having to train much data to inform accurate prediction makes SVM models memory intensive.Thus, efficient optimizations are required both at the hardware architecture and algorithm levels.To efficiently compute kernel functions, hardware acceleration units have been developed using FPGAs so that these computationally intensive operations can be moved to the hardware [92].Also, at the algorithm level, a sequential minimal optimization method may be used to reduce the memory usage [96].

Some SVM Optimizations Schemes
SVM techniques are computing and memory intensive and thus require appropriate optimization methods to be successfully executed in resource-constrained environments.Some works of literature have investigated the optimization of SVM models.The training of the SVM involves solving a quadratic programming (QP) problem, and thus to solve this problem optimally, optimization techniques involving chunking, decomposition or sequential minimal optimizations, etc. may be carried out to reduce the memory footprint required for training the model.For inference, bit precision techniques, Logarithm number representations, quantization, etc., are some optimization techniques that may be applied to fit SVM models within resource-constrained environments.In [94], Boni et al. develop a model selection technique targeted at reducing SVMs for resource-constrained environments.Table 7 presents a comprehensive list of SVM optimization schemes for both the training and classification phases.Interestingly, the Kernel selection also informs the computational requirement of SVM models.Some kernel types are the Laplacian kernel, the Gaussian kernel, the sigmoid kernel, the linear kernel, etc.Of these kernel types, the most suitable for resource-constrained environments is the Laplacian kernel because it can be implemented using shifters [97].This method solves the quadratic programming problem (QP) through the removal of rows and columns of zero Lagrange multipliers, thus reducing the entire memory footprint of the SVM model.
Although chunking reduces the size of the model, for large data sets, even the reduced non-zero multipliers are larger than the available memory of the system.
[99] Decomposition This method targets replacing the original QP problem with a sequence of smaller problems and, for every sub-problem, invokes an iterative numerical routine, which is expected to converge optimally, thus reducing memory footprint and computation requirement.
At every step, a numerical QP solver which may require numerical precision issues to be addressed is adopted.This process requires invoking an entire iterative QP library routine, thus introducing more computation time. [96] Sequential Minimal Optimization This is the most widely used optimization technique.It involves breaking down the QP problem into sub-problems and subsequently solve the smallest possible optimization problem at every step.
By solving the smallest possible optimization problem at every step, SMO avoids extra matrix memory storage and invoking an entire library at every step.This technique thus saves computation time and memory footprint.
[93] Accelerated SMO In this research work, the SMO algorithm is accelerated by introducing an efficient adaptive method for processing data input, incorporating a two-level parallelization scheme, avoiding caching, reducing shrinking, and integrating data reuse.
This technique is not suitable for real-time applications and requires high computational resources to implement successfully.
[100] Reduced SVM In this work, a Smoothing Algorithm is proposed to solve the QP problem.The QP problem is first broken down by reducing the variables and constraints of the model.These samples can adequately represent the original variables and constraints of the entire dataset, and subsequently, the proposed algorithm is used to solve the problem.
This technique reduces the computational requirement of SVM but targets only non-linear kernel types.

INFERENCE PHASE OPTIMIZATIONS [101] Fixed Point Optimization
In this research, the SVM algorithm is implemented using fixed-point arithmetic.Fixed-point number representation, when compared with contemporary floating-point, is less compute-intensive.Thus, this reduced precision technique is adopted to execute the SVM model within a smartphone for a multiclass human activity recognition application.
When reducing precision, accuracy becomes a concern because the lower the bit precision adopted, the more likely it is to introduce classification errors.
[102] Quantization In this work, a probabilistic technique is used to observe the effect of introducing quantization to the SVM model.The parameters of the model are quantized and using two datasets (Iris and Sonar) which are embedded system-oriented models, the effect of quantization is measured Although quantization reduces computation time and memory usage, the process introduces noise which could introduce errors except when fine-tuned.Although LNS saves computation time, it requires more hardware resources to implement because it replaces multiplication computations with addition and subtraction. [104]

Multiplication free SVM
In this research, To address the limitation of hardware resources in resource-constrained environments, The SVM parameters are converted to fixed-point representation, and a hardware-friendly kernel and CORDIC-like iterative algorithm are proposed to avoid multiplication computations using only adders and shifters.
This technique reduces the computational requirements of the system; however, reduced bit precision (fixed-point representation) introduces accuracy errors.[94] Order Model Selection In this research, a model selection algorithm is proposed to select the hyper-parameters with the best Pareto-optimal state using the classification error and the number of support vectors.
This research adopted quantization techniques that introduce classification errors due to the influence of noise.

Gaussian Mixture Model
GMMs are density models capable of representing a large class of sample distributions.They are used in finding the traffic density patterns in a large set of data.This characteristic makes them suitable for analyzing large sensor data in IoT devices and biometric systems, particularly for speaker recognition systems [105].In [50], GMM is adopted within an embedded board for analyzing the volume of sensor data at run time to monitor certain conditions of a system.Although GMMs are efficient, deep learning models pose a more effective method of analyzing raw sensor data.

The GMM Algorithm
A GMM is a weighted sum of M component Gaussian densities as shown in Equations (7a) and (7b).The parameters of a GMM are retrieved during the training phase using an expectation-maximization (EM) algorithm or maximum a posteriori (MAP) estimation technique.The accuracy of a GMM hugely depends on the amount of computational power and memory bandwidth required to implement the model: where x is a D-dimensional continuous-valued data vector (features), i = 1, . . ., M, are the mixture weights, p(x|λ) is the probability density function, and g(x|µ i , Σ i ) are the component Gaussian densities, µ i is the mean vector, Σ i is the covariance matrix.

Some GMM Optimization Schemes
GMMs are used in representing and modeling large volumes of data as in sensor data systems and in background modeling tasks [105].This characteristic makes GMMs highly computationally and memory intensive, and unsuitable for real-time oriented applications.Some literature has explored optimization techniques for reducing the computational requirement and memory footprint of GMMs.Pushkar and Bharadwaj [106], propose an enhanced GMM algorithm by minimizing the floating-point computations, modifying the switching schedule, and automatically selecting the number of modes to target resource-constrained embedded environments.Additionally, Shen et al. in [107] propose an optimized GMM model based on compressive sensing to reduce the dimensionality of the data while still retaining the relevant information.This technique is computationally and memory efficient.In another publication, Salvadori et al. [39] proposed a GMM optimization technique based on integers to target processors with no floating point unit (FPU).This work showed low computation and a highly reduced memory footprint.A list of these optimization schemes is presented in Table 8.

Minimization of Floating-Point Computations Background Subtraction
The results of this research were impressive, showing no degradation in accuracy except for lower recall rates.

Background Subtraction for real-time tracking in Embedded Vision
The results of this research reveal good performance for computational speed and reduce the memory footprint by 50% [39] Integer-based technique

Background/foreground Segmentation
This work shows good performance for processors without FPU, thus reducing computation cost and reducing the memory footprint to 1/12 of the original GMM; however, it cannot be adopted for models with more than two Gaussians

Deep Learning Models
Deep learning models are machine-learning techniques that model the human brain [30].They use a hierarchical array of layers to learn from available data and make new predictions based on the information they extract from the set of raw data [13,30,31].The primary layer types that make up a deep learning model are pooling layers, convolutional layers, classifier layers, and local response normalization layers.The high accuracy of deep learning algorithms in various areas of applications has made them very attractive for use in recent times.However, hardware architectural computational power is pushing hard to meet up the computational demand of these models to inform their optimal implementation.In this survey, we explore the three main classes of deep learning models, which are DNNs, CNNs, and RNNs.Table 9 describes popular DNN models with their parameters.Furthermore, like other machine learning models, deep learning models go through three phases; train, test, and predict.The training phase of deep learning models is carried out using a feedforward technique that entails sequentially passing data through the entire network for a prediction to be made and back-propagating the error through the network.The technique for backpropagation is called stochastic gradient descent (SGD), which adjusts the weights or synapses of each layer in the network using a non-linear activation function (tanh, sigmoid, rectified linear unit (ReLU)) [108][109][110].Lin and Juan [111] carry out research where they explore the possibility of developing an efficient hardware architecture to accelerate the activation function of the network.The training process is often carried out many times for the model to efficiently learn, and then using the trained model, the prediction is made on new data.The training is very computationally and memory intensive and is often carried out offline using very high-performance computing resources mostly found in large data centers, while the inference targets low cost and resourceconstrained environments.

Convolution Layers
The convolution layers are the input layers of most DNNs, and they are used to extract characteristic features from a given input data using a set of filters.These filters are vector products (kernels), and their coefficients form the synaptic layer weights [112].Convolution layers thus perform the major number of multiply and accumulate operations in the entire network, which means they are computationally intensive and are the major drawback to the real-time performance of deep learning models.Hardware acceleration units can be developed to accelerate these layers to reduce the latency of implementing deep learning models.The output neuron of a convolution is described in Equation (8): where out(x, y) f 0 represent the output neuron and in(x, y) f 0 represents the input neuron, in x and y directions, w f i , f 0 k x k y represent the synaptic weight and k x k y represent the kernel position, N i f are the input features.In addition, f i and f 0 represent the input and output feature maps, respectively.

Pooling Layers
The pooling layers are used in subsampling the feature maps obtained for convolutions and computing the maximum/average of neighboring neurons in the same feature map.In summary, this layer helps reduce the input layer dimensionality, thereby reducing the total number of inputs into subsequent layers and we traverse down the neural network [112].The pooling layer has no synaptic weight attached.Some research prunes away this layer from the entire network to reduce computation time.The equation to evaluate a pooling layer is given in Equation ( 9): where out( f 0 , x, y) is the output neurons at positions x and y, K is the number of feature maps in x and y directions, k x , k y are kernel positions in x and y directions respectively and f 0 represents the output feature maps.

Normalization Layers
These layers inform competition between neurons at the same location but in different feature maps.They perform a process very similar to lateral inhibition in biological neurons.Their values may be evaluated from Equation (10).Some research works skip this layer in DNN implementation to accelerate the entire network.
where out(x, y) f are output neurons, in(x, y) f are input neurons, N f are the input features, f and g are input and output feature maps respectively, k is the number of adjacent feature maps, and c, α, and β are constants.

Fully-Connected Layers
These are layers where all the output features are fully connected to all the input features using synaptic weights.Each output neuron is a weighted sum of all the input neurons.Equation ( 11) describes the value of each output neuron.These layers are often used for classification and output.Interestingly, because this layer is fed with the processed output features of the previous layers, i.e., the Convolution, Pooling, and Normalization layers, the input features are always lower than those for other layers i.e., they perform a reduced amount of matrix multiplications.However, the full connection using synaptic weights makes them very memory hungry as they make up for the largest share of all synaptic weights: where "t" is the non-linear activation function, N i are the input features, w ij are the synaptic weights, i and j are the input and output feature maps respectively.

Fully-Connected Deep Neural Networks
DNNs are a class of neural networks that are used for speech recognition applications, extracting high-level human behaviors, etc.The DNN architecture is such that all the layers in the network are fully connected, and the output layer with the highest activation value gives the required prediction.This characteristic feature makes them suitable for learning from unstructured data.Owing to the full connectivity of all the layers in the network, DNNs are computationally intensive than CNNs but highly memory intensive.More explicitly, because they have to perform routine multiply-accumulate computations, their computational logic is not complex but they require large enough memory to store the synapses and activation functions.Therefore, optimizations for FC DNNs concern more memory-centric thrusts like model compression, sparsity, pruning, and quantization.In [113], DNN models are developed to be implemented in mobile devices by distributing the computation across different processors within the mobile device.A fully connected DNN is also referred to as a multilayer perceptron (MLP).

Convolutional Neural Networks
CNNs are a class of neural networks that are suitable for computer vision applications, object recognition applications, etc. [114][115][116].In ConvNets, key features in an image are extracted then converted into complex representations using the pooling layer, and subsequently, the fully-connected layers classify the image and identify the image appro-priately.The CNN architecture is primarily made up of a large portion of convolution layers, followed by few fully connected layers.Some CNNs sometimes add pooling and normalization layers in-between the convolution and fully connected layers.The presence of convolution layers that perform kernel functions (vector-matrix multiplications) makes CNNs very computation-intensive and less memory-hungry due to the few fully connected layers.Therefore, optimizations for CNN models concern more computer-centric directions such as innovative hardware acceleration developments, processor technology, tiling, and data reuse, reduced precision, quantization, etc. Popular CNNs are ResNets, LeNets, AlexNets, GoogLeNets, VGGNets, etc.The computation and memory requirements of some CNN models are given in Figure 2 and summarized in Table 10.
connected DNN is also referred to as a multilayer perceptron (MLP).

Convolutional Neural Networks
CNNs are a class of neural networks that are suitable for computer vision applications, object recognition applications, etc. [114][115][116].In ConvNets, key features in an image are extracted then converted into complex representations using the pooling layer, and subsequently, the fully-connected layers classify the image and identify the image appropriately.The CNN architecture is primarily made up of a large portion of convolution layers, followed by few fully connected layers.Some CNNs sometimes add pooling and normalization layers in-between the convolution and fully connected layers.The presence of convolution layers that perform kernel functions (vector-matrix multiplications) makes CNNs very computation-intensive and less memory-hungry due to the few fully connected layers.Therefore, optimizations for CNN models concern more computercentric directions such as innovative hardware acceleration developments, processor technology, tiling, and data reuse, reduced precision, quantization, etc. Popular CNNs are ResNets, LeNets, AlexNets, GoogLeNets, VGGNets, etc.The computation and memory requirements of some CNN models are given in Figure 2 and summarized in Table 10.RNNs are a class of Neural Networks used for natural language translation applications, speech recognition, etc. RNNs, unlike CNNs, process input data sequentially and store the previous element.They thus retain and use past information which makes them suitable for text prediction applications and also for suggesting words in a sentence.The architecture of an RNN model is primarily made up of fully connected layers and normalization layers.This organization makes RNNs memory-centric in operation since they have to store weight synapses in available memory.A type of RNN referred to as Long Short Term Memory (LSTM) is gaining interest in recent times and process to be more effective than conventional RNNs [117][118][119][120]. Khan et al. [121] developed a data analytic framework with the real-world application using Spark Machine Learning and LSTM techniques.There are great hardware acceleration opportunities in implementing RNNs [122][123][124], and [23] explores the FPGA acceleration of an RNN.Table 11 describes some of the DNN models and their architecture.

Machine Learning in Resource-Constrained Environments
Machine learning techniques are currently targeting resource-scarce environments such as mobile devices, embedded devices, and other internet of things devices.In this section, we present an overview of different resource-limited environments like microcontroller units (MCUs) and mobile devices.We also discuss the option of hardware acceleration units used to speed up the execution of these algorithms in resourcescarce environments.

Machine Learning Using Microcontrollers
Microcontrollers are at the front end of provisional hardware to implement diverse embedded systems and other IoT applications [126].An MCU consists of a microprocessor, memory, I/O ports, and other peripherals all integrated into one chip.At the processor core of MCUs are general-purpose CPUs for adequate computation.Table 12 describes a list of popular MCUs using their compute and memory resources.From this list, we can observe the resource limitation of most MCUs in terms of available power and on-chip memory (flash + SRAM).This resource limitation is a critical drawback in the implementation of machine learning models.For example, some typical CNN model sizes are AlexNet (240 MB) [27], VGG 16 (528 MB) [127], VGG 19 (549 MB) [127], etc. Model size describes the number of bytes required to store all the parameters of the model.In this study, we survey techniques for compressing these models to fit into the available memory of MCUs for efficient computation to be appropriately done.BeagleV TM RISC-V U74 1.0 GHz SD Card 8 GB 3000 mA Table 12 presents a list of different microcontrollers categorized using their clock frequency, available flash memory, SRAM, and their current consumption.As can be observed in this table, microcontrollers have very limited hardware resources.This scarcity of resources makes them unsuitable for high-end machine learning applications, except the machine learning models are heavily optimized to fit within this space [24].

Machine Learning Using Hardware Accelerators
General-purpose CPU chipsets, although ubiquitous, do not possess enough computational ability to process compute-intensive ML algorithms.To address this drawback, hardware acceleration units may be developed using GPUs, FPGAs, and even ASICs.However, the most popular accelerators are FPGA-based accelerators owing to the programmability of FPGAs.Hardware acceleration is developed such that compute-intensive segments of the ML algorithm, e.g., kernel operations, are offloaded to the specialized accelerator, thereby relieving the CPU to process much simpler operations, improving the overall computation speed of the system.Some machine learning accelerators are Arm Ethos NPUs, Intel's Movidus NCS, Nvidia's Jetson nano, etc. Table 13 presents some of these accelerators.A critical drawback in adopting these accelerators is cost.Also, some works of literature have explored the development of FPGA-based accelerators for machine learning algorithms.Wang et al. [22] proposed a scalable deep learning accelerator to accelerate the kernel computations of deep learning.The authors introduce some optimization schemes such as tiling techniques, pipelining, FIFO buffers, and data reuse to further improved their proposed architecture.

Machine Learning in Mobile Devices
Machine learning techniques are gradually permeating mobile devices for applications such as speech recognition, computer vision, etc. Mobile devices may be categorized under resource-constrained systems owing to their limited computational and memory resources.Hence, for machine learning algorithms to be successfully implemented within these devices, appropriate optimizations must be carried out.Lane et al. [137] developed a software accelerator for accelerating the execution of deep learning models within mobile devices.We present a survey of some mobile machine learning applications in the literature as tabulated in Table 14.

TinyML
Machine learning inference at the edge particularly within very low power MCUs is gaining increased interest amongst the ML community.This interest pivots on creating a suitable platform where ML models may be efficiently executed within IoT devices.This has thus opened a growing research area in embedded machine learning termed TinyML.TinyML is a machine learning technique that integrates compressed and optimized machine learning to suit very low-power MCUs [141].TinyML primarily differs from cloud machine learning (where compute intensive models are implemented using high-end computers in large datacenters like Facebook [142]), Mobile machine learning in terms of their very low power consumption (averagely 0.1 W) as shown in Table 15.TinyML creates a platform whereby machine learning models are pushed to user devices to inform good user experience for diverse applications and it has advantages such as energy efficiency, reduced costs, data security, low latency, etc., which are major concerns in contemporary cloud computing technology [141].Colby et al. [143] presented a survey where neural network architectures (MicroNets) target commodity microcontroller units.The authors efficiently ported MicroNets to MCUs using the TensorFlow Lite Micro platform.There are different platforms developed to easily port ML algorithms to resource-constrained environments.Table 16 presents a list of available TinyML frameworks commonly adopted to push ML models into different compatible resource-limited devices.

Challenges and Optimization Opportunities in Embedded Machine Learning
Embedded computing systems are generally limited in terms of available computational power and memory requirements.Furthermore, they are required to consume very low power and to meet real-time constraints.Thus, for these computationally intensive machine learning models to be executed efficiently in the embedded systems space, appropriate optimizations are required both at the hardware architecture and algorithm levels [148,149].In this section, we survey optimization methods to tackle bottlenecks in terms of power consumption, memory footprint, latency concerns, and throughput and accuracy loss.

Power Consumption
The total energy consumed by an embedded computing application is the sum of the energy required to fetch data from the available memory storage and the energy required to perform the necessary computation in the processor.Table 17 shows the energy required to perform different operations in an ASIC.It can be observed from Table 17 that the amount of energy required to fetch data from the SRAM is much less, than when fetching data from the off-chip DRAM and very minimal if the computation is done at the register files.From this insight, we can conclude that computation should be done as close to the processor as possible to save energy.However, this is a bottleneck because the standard size of available on-chip memory in embedded architectures is very low compared to the size of deep learning models [124].Algorithmic-based optimization techniques for model compression such as parameter pruning, sparsity, and quantization may be applied to address this challenge [150].Also, hardware design-based optimizations such as Tiling and data reuse may be utilized [25].The next section expatiates some of these optimization methods in further detail.Furthermore, most machine-learning models, especially deep learning models, require huge amounts of multiply and accumulate (MAC) operations for effective training and inference.Figure 3 describes the power consumed by the MAC unit as a function of the bit precision adopted by the system.We may observe that the higher the number of bits, the higher the power consumed.Thus, to reduce the power consumed during computation, reduced bit precision arithmetic and data quantization may be utilized [151].to the processor as possible to save energy.However, this is a bottleneck because the standard size of available on-chip memory in embedded architectures is very low compared to the size of deep learning models [124].Algorithmic-based optimization techniques for model compression such as parameter pruning, sparsity, and quantization may be applied to address this challenge [150].Also, hardware design-based optimizations such as Tiling and data reuse may be utilized [25].The next section expatiates some of these optimization methods in further detail.Furthermore, most machine-learning models, especially deep learning models, require huge amounts of multiply and accumulate (MAC) operations for effective training and inference.Figure 3 describes the power consumed by the MAC unit as a function of the bit precision adopted by the system.We may observe that the higher the number of bits, the higher the power consumed.Thus, to reduce the power consumed during computation, reduced bit precision arithmetic and data quantization may be utilized [151].It may be deduced from the graph that lower number precisions consume less power than high precisions with no loss in prediction accuracy.However, we can observe that when precision is reduced below a particular threshold (16 bit fp), the accuracy of the model is greatly affected.Thus, quantization It may be deduced from the graph that lower number precisions consume less power than high precisions with no loss in prediction accuracy.However, we can observe that when precision is reduced below a particular threshold (16 bit fp), the accuracy of the model is greatly affected.Thus, quantization may be performed successfully to conserve energy but quantizing below 16-bit fp may require retraining and fine-tuning to restore the accuracy of the model.

Memory Footprint
The available on-chip and off-chip memory in embedded systems are very limited compared to the size of ML parameters (synapses and activations) [27].Thus, there is a bottleneck for storing model parameters and activations within this constrained memory.Network pruning (removing redundant parameters) [150] and data quantization [151] (reducing the number of bits used to represent model parameters) are the primary optimization techniques adopted to significantly compress the overall model size such that they can fit into the standard memory sizes of embedded computers.

Latency and Throughput Concerns
Embedded systems are required to meet real-time deadlines.Thus, latency and overall throughput can be a major concern as an inability to meet these tight constraints could sometimes result in devastating consequences.The parameters of deep learning models are very large and are often stored off-chip or in external SDCARDs, which introduces latency concerns.Latency results from the time required to fetch model parameters from off-chip DRAM or external SDCARDs before appropriate computation can be performed on these parameters [150].Thus, storing the parameters as close as possible to the computation unit using Tiling and data reuse, hardware-oriented direct memory access (DMA) optimization techniques would reduce the latency and thus, inform high computation speed [152].In addition, because ML models require a high level of parallelism for efficient performance, throughput is a major issue.Memory throughput can be optimized by introducing pipelining [20].

Prediction Accuracy
Although deep learning models are tolerant of low bit precision [153], reducing the bit precision below a certain threshold could significantly affect the prediction accuracy of these models and introduce no little errors, which could be costly for the embedded application.To address the errors which model compression techniques such as reduced precision or quantization introduce, the compressed model can be retrained or fine-tuned to improve precision accuracy [124,150,154,155].

Some Hardware-Oriented and Algorithm-Based Optimization Techniques
Hardware acceleration units may be designed using custom FPGAs or ASICs to inform low latency and high throughput.These designs are such that they may optimize the data access from external memory and/or introduce an efficient pipeline structure using buffers to increase the throughput of the architecture.In sum, some hardware-based optimization techniques are presented in this section to guide computer architects in designing and developing highly efficient acceleration units to inform high performance 4.5.1.Tiling and Data Reuse Tiling is a technique that involves decomposing a large volume of data into small tiles that can be cached on-chip [25,156].This technique targets the bottleneck of memory footprint in resource-constrained environments.This technique also introduces scalability to the entire architecture as different sizes of volume data can be easily broken down into bits that may be stored on-chip.Much more, this technique reduces the latency of the system as tiled inputs may easily be reused for computation without having to re-fetch parameters from off-chip.Furthermore, since tiled data can be stored on-chip, energy consumption is reduced.Hardware accelerators may be designed and developed to integrate a tile unit to carry out the tiling process [22].The pseudocode of the tiling process is given in Algorithm 2.

Require:
Ni: the number of the input neurons No: the number of the output neurons Tile_Size: the tile size of the input data batchsize: the batch size of the input data for n = 0; n < batchsize; n ++ do for k = 0; k < Ni; k+ = Tile_Size do for j = 0; j < No; j ++ do y end if end for end for end for end for

Direct Memory Access and On-Chip Buffers
More recent FPGA architectures owing to the limitation in computation and memory of custom FPGAs, provide general-purpose processors and external memory to offload computation from the FPGA processing logic to the CPU [157].This architectural organization if not properly utilized, can result in latency concerns.DMAs are units that transfer data between the external memory and the on-chip buffers in the processing logic of the FPGA [152].Thus, optimizing this process would lead to an efficient performance in execution speed.

Layer Acceleration
A deep learning network is made up of different kinds of layers (pooling, normalization, fully connected, convolution, etc.).A technique for speeding up the rate of execution of a network and saving memory storage is to design and develop specialized architectures to accelerate particular layers in the network layer [26].In [22], an accelerator is designed using FPGA technology to accelerate certain parts of a deep neural network.Also, in [51], an accelerator is designed to accelerate convolution layers in a CNN.Network layer acceleration is a hardware-oriented optimization scheme and could pose challenges such as hardware design and high time-to-market since specialized architectures are often considered [148].

Network Pruning
Network pruning is concerned with the removal of certain parts of the network, ranging from weights to layers that do not contribute to the overall efficiency of the network [158].It entails rounding off certain unused weights and activations to zero to reduce the total memory and computation resource required for efficient computation.These weights and activations are such that they would not alter the accuracy of the execution of the model if avoided.Pruning can either be structured or unstructured [158].In [124], the pruning technique is employed to compress a neural network within the resource-constrained environments of embedded and mobile devices.The pruning entailed removing i.e., set to zero, weights that are lower than a certain threshold value, and the pruned network is then retrained to improve accuracy.

Reduced Precision
Machine learning algorithms, particularly deep learning models, were originally implemented using a high floating-point number representation format [159,160].The floating-point number system is made up of a sign, an exponent, and a mantissa [160].A single floating-point value can be computed using the formula presented in Equation ( 12).The floating-point number system is, however, power-hungry and thus unsuitable for embedded machine learning applications owing to the resource constraints [161,162].More so, floating-point arithmetic currently faces drawbacks like manipulating overflow, underflow, and exceptions [163].This has thus made the fixed-point number system a better alternative owing to the reduced complexity and power consumption of its implementation with the combined range given in Equation ( 13), [164,165].Hwang and Sung [166] Investigate the ternary quantization of a Feedforward deep neural network using fixed-point arithmetic.Additionally, [167] considers training a deep network using 16-bit fixed-point arithmetic with stochastic rounding to minimize accuracy loss.Although fixed point arithmetic is power efficient, it is not suitable for representing deep learning parameters that are nonlinear.Gustafson and Yonemoto [168] present a new number system called Posit given in Equation ( 14).Langroudi et al. [163] adopt the posit number system in training a deep convolutional neural network.
where value is the floating-point value, sign is the sign bit, mantissa is the mantissa bit.
where ∝ represents the input integer, QI = # of integer bits and QF = # of fractional bits and ε is the resolution of the fixed-point number.
where X represents the Posit value, r value represents the number regime and useed represents the scale factor.
4.5.6.Quantization SVM and DNN model parameters are often represented using 32-bit floating-point values [92,158], which are highly computationally and memory intensive.However, research shows that these models can be implemented efficiently using low precision parameters (8-bit or 16-bit) with minimal accuracy loss [113,169].Quantization describes techniques aimed at reducing the bit width of the weights and activations of a machinelearning model to reduce the memory storage and communication overhead required for computation.This process thereby reduces the bandwidth required for communication, overall power consumption, area, and circuitry required to implement the design.Many research works have considered different quantization techniques for deep learning models.Courbariaux et al. [170] consider training a deep model using binary representation (+1 and −1), of the model parameters using a binarization function given in Equation (15).Also, [166] proposes a quantization scheme using ternary values (+1, 0, −1).The proposed equation is given in Equations ( 16) and (17).Other quantization techniques involving Bayesian quantization, weighted entropy-based quantization, vector quantization, and two-bit networks are adopted in [50,51,158], and [171], respectively.Although quantization techniques increase execution speed, the algorithm requires fine-tuning to avoid accuracy loss [172,173]: where x b is the binarized variable (weights and activations) and x is the real-valued variable.
where w ij is the new ternarized weight, α is the learning rate E is the output error, y j is the output signal and δ i is the error signal.
where ∅(x) are the new ternarized activation functions and y is the output signal.

Areas of Applications of Intelligent Embedded Systems
Indeed, numerous application areas of embedded systems and the breakthrough of machine learning methods have further widened and deepened the range of applications where embedded systems and machine learning methods are currently actively adopted.Some areas of applications we consider in this survey are intelligent sensor systems and IoTs, deep learning in mobile devices, deep learning training using general-purpose GPUs, deep learning in heterogeneous computing systems, Embedded field programmable gate arrays, energy-efficient hardware design, and architectures.

Intelligent Sensor Systems and IoTs
There is a growing interest revolving around the efficient implementation of machine learning algorithms within embedded environments of sensor networks and the internet.Diverse machine learning algorithms such as SVMs, GMMs, and DNNs are finding useful applications in cogent areas such as the network configuration of mobile networks, analysis of sensor data, power consumption management, etc.A list of applications of machine learning executed within these environments is presented in Table 18.Although machine learning techniques have found useful applications in embedded systems domains, there are major drawbacks that entail limited available computational and memory resources in embedded computing systems.

Deep Learning In Mobile Devices
DNNs are finding very useful applications in mobile devices for speech recognition, computer vision, and natural language processing, respectively, indoor navigation systems, etc.A list of application areas of deep learning models in mobile devices is presented in Table 19.The computational and memory demands required for training and inferencing deep learning models make current mobile devices unsuitable for these models.Thus, more research is being carried out towards the inferencing of the models on mobile devices.There are a lot of energy-intensive applications on current mobile devices which compete for the limited available power and thus, more research is being carried out to optimize these deep learning models so they can efficiently fit within mobile devices.This research investigates the automatic configuration of the radio and IP stack of a MANET at runtime using machine learning techniques.To achieve this, the SVM algorithm was implemented within two communication controllers with general-purpose processors (the ARMv7 and IBM PPC440GX).
To deploy the SVM efficiently, certain optimizations were carried out on the algorithm, and the corresponding effect of each optimization technique was observed whilst comparing each ablation with a baseline.The result of the proposed system showed improved performance reducing the runtime of the system in most cases.
In this research, accuracy was traded off for execution speed.The accuracy of SVM algorithms depends on their floating-point computations.However, floating-point operations were avoided by the optimization scheme, thereby reducing the overall accuracy.Also, computationally intensive machine learning methods are often executed using hardware acceleration units.This increases the accuracy and execution speed of the scheme.to analyze the huge amount of sensor data for classification and clustering to detect the required processes.The intelligent sensor system is used to monitor the conditions of an intelligent stove and a water circulation system.
The GMM algorithm is optimized using the expectation-maximization (EM) algorithm and implemented using the minimum description length (MDL) criterion due to the intensive computational vector and matrix operations required and the limited computational power available.
Furthermore, the fixed point number representation was used in the computation of the sensor data, while the hardware implementation was done using the imperfect floating-point unit.
The number representation adopted improves the speed of the implementation at the expense of accuracy.training the different datasets and consequently inferencing the trained model using an SoC-based hardware accelerator.A unique FC-NN topology is modeled for each application for high performance and trained using preprocessed datasets as required by the application.
The NN model is manually mapped to the hardware accelerator by extracting the weights and biases and converting them from 32-bit floating-point to 16-bit fixed-point values owing to the accelerator constraints indicated in the paper.The quantization method introduced to reduce the precision of the model reduces the prediction accuracy of the system which parameter retraining would address.The performance of this model was measured using accuracy, harmonic mean score, weighted harmonic mean score, and power consumption.The results of the research show that deep learning models can be executed efficiently using specialized hardware processors Model scalability and battery life performance which are major concerns in IoT developments are not considered in the research.

[71] Intelligent Wearable Systems
This research investigates the integration of a machine learning algorithm executed on a wearable device to extract, analyze and classify sensor data on the embedded device, thereby saving the energy required to send for centralized processing.To achieve this, the research work entailed (1) sampling the data generated to avoid the generation of redundant input data, (2) carrying out appropriate feature extraction using the Integral of the modulus of acceleration (IMA) embedded machine learning method, which avoids sending the raw sensor data to a centralized processor for knowledge extraction and (3) executing the classification process using support vector machines (SVM).This machine learning sensing system was employed for the energy-efficient long-term monitoring of the physical activities of house occupants.
The results obtained were fascinating, revealing that employing adequate machine learning methods extends the battery life of a wearable sensor by 987 days when compared to transferring raw sensor data through the network.The research makes use of an SoC-based wearable sensor for data collection.Data sampling reduces system accuracy because machine learning models depend on large sensor data for effective training.Also, SoCs are highly energy-efficient and demonstrate good performance, and they require a very high time-to-market compared to other embedded architectures.This research investigates an adaptive machine learning approach to selecting the most optimal DNN for a given input, thereby saving inference time.To achieve this, the research work trained an offline predictor (premodel) using the K-nearest neighbor classification models on 12 DNN models measuring the predictions under two metrics-response to the input data and precision required.The inference time for a DNN model is a function of these two parameters.
The premodel is then executed on embedded hardware to select from an option of 14 pre-trained CNN models, the most optimal model to employ for a particular input data and precision requirement, thereby reducing the inference time.
The ML system was implemented on the NVIDIA Jetson TX2 deep learning platform, which had an embedded GPU GPUs provide the necessary speed and efficiency for deep learning implementations but are power-hungry, making them unsuitable for embedded applications.

[138] Mobile Computing
This work investigates a novel optimization method for accelerating deep neural network models for resource-constrained environments like mobile devices.
The research exploits the observation that deep learning execution time is majorly slowed by the non-tensor layers in the model (pooling, LRN, and normalization layers).An optimization framework, "DeepRebirth," is proposed, which targets the regeneration of the non-tensor layers in the model using either streamline or branch slimming methods.The optimization method involves merging non-tensor layers with bottom tensor units and further fine-tuning the regenerated layer in streamline streaming.Branch slimming involves merging non-tensor layers with tensor units at the same level.
The performance of the optimized model was measured over the state of the art networks like GoogLeNet, AlexNet, and ResNet on different high-end and low-end mobile computing processors.
The results of DeepRebirth reveal that the optimization method yields faster execution time, minimal accuracy loss, and reduced power consumption making the technique appropriate for heterogeneous mobile devices.
It is, however, observed that the optimization technique has little influence on mainstream ARM CPUs, which are the most ubiquitous processors in mobile devices.This limits the impact of optimization in the mobile computing world.More so, the inference was carried out during airplane mode, and thus, the metric for power consumption on mobile devices is not accurately measured.This metric is very important because battery life is a great concern in mobile computing and can constitute a major challenge to deep learning inferencing on mobile computing platforms.

[174] Indoor Navigation
A framework is presented in this work to improve WiFi-based fingerprinting indoor localization using commodity smart mobile devices.The research proposes the utilization of a CNN model in predicting the location of a user in an indoor environment.The WiFi access points are used in creating the image database, which models the different locations on the navigation path, which are then used to train the CNN model.
The real-time AP data are subsequently used by CNN to predict the location of the user at run time.
The CNN model used in the research "CNN-LOC" eliminates the pooling layer owing to the size of the grids, thereby eliminating the need for subsampling.This consequently reduces the training and inference times and power consumption during execution, making it suitable for resource-constrained mobile computing.The research also introduces scalability by proposing a hierarchal classifier for large area localization.The performance of this research is measured by comparing the prediction accuracy across other indoor localization frameworks using SVR, KNN, and DNN approaches.The results show that the CNN-LOC framework outperforms the others.
However, the research overlooks the limitation of battery life in smartphones during the execution of computationally intensive deep learning models like CNN using WiFi APs.WiFi has been observed to be one of the applications responsible for draining the power in mobile devices, thereby shortening the battery life.This work investigates a novel low precision arithmetic approach to optimize the power consumption and memory utilization of DCNN inferencing in the embedded system domain.To achieve this, the work explored the use of the posit number system over the contemporary fixed and floating-point number representation.The process involves converting the posit number to a decimal floating-point number and subsequently a binary floating-point number for reading and writing to memory and vice versa during the processing period.
The system is executed using three datasets and compared to a reference point which is the single floating-point representation.The results are fascinating in terms of memory utilization during inferencing, and the accuracy obtained despite using low precision estimation The hardware specifications are not given in the research.

Deep Learning Training Using Graphic Processors (GPUs)
DNNs are computationally intensive.State-of-the-art hardware for training deep learning models are graphic processors because of their high-level parallel processing and high floating-point capability.Deep learning algorithms are largely dependent on parallel processing operations, which GPUs are adequately developed to target.Although GPGPUs have very good performance, they are highly power-hungry and expensive to implement, and this thus makes them unsuitable for embedded systems design and development.This owes to the fact that a key design metric for embedded devices is that they must consume very low power and must be economical.Table 20 presents an area of application in training deep learning models on GPGPUs.GPUs are one of the major hardware acceleration tools for Deep learning algorithms owing to their suitability for high thread-level parallelism.This research Investigates an optimal system configuration for GPGPUs in training CNN models effectively.Two image classification CNN models (LeNet and MiniNet) are executed using different GPU configurations to determine the most optimal configuration.The research employs GPGPU-sim, a GPU simulator, in executing the CNN model whilst observing the effect of modifying the configuration of the GPU in terms of the NoC architecture used, the size of the L1D and the L2D caches, and network traffic intensity on the different layers of the CNNs (convolutional, pooling and fully connected layers).
The result of the research reveals that the Mesh Network outperforms the Perfect Network at the fully connected layers.Also, modification in the size of the L1 affects the performance of the CNN, while changes in the L2 cache do not influence the CNN performance.Also, the network traffic intensity differs across the layers.
GPGPUs are very cost-intensive and energy-intensive.Thus, they are unsuitable for embedded applications.

Deep Learning Using Heterogeneous Computing Systems
Multicore and many-core architectural enhancement is a modification made by computer architects to address the performance wall and memory wall facing CPU technology.Multicore technology is also called homogenous computing systems, while many core architectures are used in heterogeneous computing systems.Heterogeneous computing systems are systems with more than one type of processor core.Most heterogeneous computing systems are used as acceleration units for offloading computationally intensive operations from the CPU, thereby increasing the system's overall execution speed.Table 21 presents an area of application of deep learning training in heterogeneous computing systems.A critical drawback in heterogeneous computing systems pivots around the sharing of memory resources, data bus, etc.If designed inefficiently, it can result in data traffic and thus increase latency and power consumption.The research focuses on a software approach to the problem and not a hardware perspective.

Embedded Field Programmable Gate Arrays (FPGAs)
FPGAs are gaining popular interest in the computing world due to their low cost, high performance, energy efficiency, and flexibility.They are often used to design acceleration units and pre-implement ASIC architectures.Table 22 presents certain areas of applications where FPGA architectures are adopted to accelerate deep learning model execution.FPGAs, although programmable and reconfigurable, are difficult to program.This is a critical limitation to their ubiquitous utilization in the embedded computing design.This work proposes a development scheme to accelerate the execution of convolutional neural networks in resource-constrained FPGAs by exploring the design space of the system.The roofline model, which uses a computation and communication approach to measure the performance of the system, was adopted to explore the design space to affect the optimization of the computation and memory access process.
The computation process is accelerated by loop unrolling, pipelining, and tile sizing using a polyhedral-based optimization technique.Memory access is optimized by adopting loop promotion and transformations for efficient data reuse.Optimal parameters are first determined for single convolutional layers and then unified unroll factors are used for CNNs with multiple convolutional layers.The performance of the proposed architecture is estimated by observing the computational performance, resource utilization, and power consumption.The proposed scheme reveals brilliant results in outperforming prior works.
The research, however, focused on accelerating the feedforward inference process and optimized only the convolutional layers neglecting the pooling, normalization, and fully connected layers which are required in real-world scenarios.Besides, tiling introduces some accuracy loss to the model execution.The research focuses on hardware design alone.However, FPGAs, although flexible owing to their reconfigurable characteristic, are difficult to program employing a hardware/software approach that is more suitable for developers.
More so, power consumption can be further reduced by employing low precision approximation techniques for deep learning models which are not employed in the paper.

[51]
Embedded FPGA A hardware/software co-design approach to accelerate large-scale convolutional neural networks in the embedded system domain is considered in this work.The framework proposed entails the decomposition and quantization of large-scale CNN models at the software level and the design of an FPGA-based CNN accelerator at the hardware level.The optimization at the software level targets the high memory space required by fully-connected layers and high computational demand required by the convolutional layers in the CNN model, while the hardware level optimization entailed the design of an application-specific FPGA architecture to meet the computational and high memory bandwidth required for the efficient execution of the model.To achieve this, the Singular Value Decomposition (SVD) technique was adopted to accelerate the fully connected layers, and dynamic-precision quantization using 16-bit fixed-point representation was used for the quantization process.OpenCL-based FPGA Hardware Acceleration This work considers the acceleration of large-scale convolutional neural networks using an OpenCL-based FPGA Architecture focusing on optimizing the rate at which data is processed to improve throughput.To achieve high throughput, a precision study is carried out using the Caffe tool of two CNN models (AlexNet and VGG) for the convolution and fully connected layers to determine the optimal precision at which inference can be carried to avoid accuracy loss.At the hardware level, scalable modules are designed using OpenCL software design skit for accelerating each of the different layers in the CNN (convolution, pooling, normalization, and fully connected layers).In the OpenCL implementation phase, the convolution layer is accelerated by developing a scalable convolution block that performs the multiply and accumulates operations required by the convolution layer.
The Normalization layer is accelerated by developing a module that performs exponent operations using a piece-wise linear approximation function.based on a long short term memory design approach is proposed in this work.The proposed solution targets the constraint involving vanishing or exploding gradients while executing RNNs by recommending a variation to the LSTM design approach.The recommended LSTM design method entails the definition of three gates (input, forget, and output) that learn to either remember, forget or output a result.This LSTM architecture is implemented using the FPGA which uses fixed-point approximations for computations.
The matrix-vector multiplications are implemented using Multiply Accumulate units while the non-linear functions are computed using a MAC unit and a comparator.Direct memory access units are incorporated with a stream synchronization module to foster efficient communication of data between the processing logic units and the external memory.
The module integrates a driver software to incorporate data reuse to avoid constant external memory accesses which consumes power.
The proposed architecture is implemented using Zedboard's FPGA with dual ARM Cortex-A9 processor and compared to platforms using CPU and GPU computing units for performance evaluation outperforming them.
However, fixed-point representations introduce accuracy errors to the results, which parameter retraining or model fine-tuning ought to address.Besides, external memory accesses consume a lot of power owing to the high bandwidth requirement.Furthermore, the research focuses on the learning phase alone, neglecting inferencing.

Energy Efficient Hardware Design and Architectures
The urgency for novel energy-efficient hardware designs cannot be overemphasized.Table 23 presents different application-specific architectures and current research issues involved in targeting the high-performance applications required in this big data era.Application-Specific Architectures, although highly efficient, are difficult to design and implement, having high time-to-market.Their good performance owing to their specificity makes them very suitable for embedded and machine learning applications.The research survey suggests low precision approximations for machine learning models to enhance their execution and reduce their power consumption.Furthermore, the research survey reveals how machine learning techniques could also be used in the exploration of the design space for ASIC development.
The research survey focused on the hardware perspective alone.However, there are current software and hardware/software co-design approaches to enhance deep learning.More so, Photonics is a current promising research thrust owing to the promising characteristics of photonics-based architecture over electronic-based architecture.

Research Directions and Open Issues
Embedded machine learning research is still in its early days.Thus, there remains a large range of opportunities to explore in this research direction, which is critical to the development of IoT devices of the future.Some research directions in the areas of Computer Architecture, Deep Learning Optimizations, Hardware Security, Energy Efficiency, and Power Management are presented in Table 24.Additionally, the key lessons learned are highlighted in Section 6.1.With the end of Dennard's scaling and Moore's law significantly slowing down, combined with the performance wall of Silicon, there remains an urgent need for innovation in the design and development of hardware architecture to meet the target for high-performance computing that face us today.In this respect, [176] suggests domain-specific architectures in the form of GPUs, FPGAs, TPUs, etc. as a research direction for future high-performance architectures.DSAs are different from ASICs in that DSAs introduce flexibility to the design due to their programmability.This is an open research area in computer architecture.Also, to generate highly efficient application-specific ASIC architecture designs, the design space is required to be explored efficiently, which is currently a stringent task carried out by human experts.However, [177,178] presents a frontier of employing machine learning techniques in exploring the design space at design time and runtime.Ruben et al. [179] proposed a machine learning-based prediction for the dynamic, runtime, and architectural optimizations of Embedded Systems.This is another open area of research-adopting machine learning in efficient computer architecture design and development 2

Deep Learning Optimizations
Deep learning models are computationally and memory intensive, and thus, implementing them within resource-constrained environments is tasking.There is, therefore, an opportunity for highly efficient optimization techniques to compress deep learning models efficiently with minimal or no accuracy loss.Although many research works have explored optimization techniques, optimization methods are infinite, and thus there remains an opportunity to optimize deep learning models still.Some optimization techniques are but are not limited to pruning, clustering, layer acceleration, quantization, and numeric precision.Some optimizations combine one or more of these techniques to compress a model successfully.

3
Hardware Security Software security has been greatly explored, but little work has been done in hardware security which is a major concern in embedded systems development [180].State-of-the-art embedded hardware architectures are prone to Trojan attacks, and thus, this creates the need for research in the design and development of secure embedded architectures for embedded applications.Energy Efficiency is a critical issue in embedded computing systems because most embedded devices run on a battery [181].Thus, to effect the continuous functionality of embedded devices, there remains the need to adequately design energy-efficient architectures and also adopt innovative power management techniques.This is a crucial research thrust in embedded computing technology, particularly to meet the requirements of high-performance machine learning applications [182][183][184].

Silicon Photonics
Current Silicon technology is reaching its limit for performance and thus [185] surveys the exploration of photonics-based architectures as a substitute for silicon technology owing to the high bandwidth and data transfer speed of photonics-based architectures [186,187].This is a key research direction for high-performance architectures

Lessons Learned
In this section, we present a comprehensive summary of the lessons learned from this survey.Our summary covers embedded systems computing architectures, machine learning techniques, deep learning models, optimization techniques, and energy efficiency and power management techniques.
Lesson one: As of today, even the most expensive embedded system platforms do not have the computational and memory capacity to execute expensive machine learning algorithms efficiently.Thus, to bring these models into the embedded space where it becomes a part of our everyday life, which may be found in mobile devices and other IoTs, huge hardware architectural modifications and algorithm optimizations, are required.To approach the issue of overall efficiency properly, optimization approaches ought to tackle the key performance constraints of embedded systems: low power consumption, memory footprint, and latency and throughput concerns.Some optimization techniques are network pruning, data quantization, tiling, layer acceleration, etc.
Lesson two: The hardware architectural modifications required to accelerate stateof-the-art deep learning and other machine learning models greatly depend on the ML Model architecture.For example, the CNN model architecture is such that is computationcentric because there are many convolution layers in a CNN architecture, while a fully connected DNN is memory-centric because the algorithm architecture is such that it places great demand on the memory for both storage and throughput.Thus, when hardware acceleration units are being developed, it must be done with a sound understanding of the algorithm architecture to accelerate the right operations, whether vector products (convolutions) or multiply and accumulate (fully connected).This improves the overall efficiency of the hardware architecture.
Lesson three: Most machine learning models are much bigger than standard embedded on-chip and off-chip memory sizes.Thus, to address the memory concern, optimizations may be carried out using network model pruning to reduce the number of parameters, so they can be stored within the available memory, data quantization, which reduces the bit precision of the network parameters so they could fit into standard DRAM and SRAM sizes.Also, direct memory access units may be adopted to reduce the latency of data transfer from the external memory to the processing logic to inform high execution speeds.
Lesson four: To address the energy-efficiency bottleneck, which is primarily due to the type of computation carried out, and the power required to fetch parameters from off-chip memory, optimizations may involve reducing the total number of parameters of the network so that computation may be done as close as possible to the compute unit.Some techniques for compressing the model involve network pruning, clustering, and data quantization.Also, bit reduction using quantization introduces accuracy errors to the overall prediction process.It is worthy of note that precision should not be reduced below a particular threshold which should preserve the model accuracy.To address the accuracy concerns, the quantized parameters of the model may be retrained and fine-tuned to restore prediction confidence.
Lesson five: To tackle latency concerns that are a result of off-chip memory transfers, optimizations may be carried out such that model parameters may be cache on-chip for data reuse.This optimization can be done using techniques such as tiling or simple vector decomposition, where input data may be partitioned into bits or tiles that can fit into on-chip memory and may be reused for computation when required.This technique avoids frequent off-chip memory transfers, which is a major concern for both latency and power consumption.Hardware acceleration units may be designed to integrate a Tiling Unit to carry out this operation at the hardware level.Some other techniques to inform high throughput involve pipelining, on-chip buffer optimization, data access optimizations, etc.
Lesson six: Although hardware acceleration using custom FPGA logic, GPUs, or CPUs addresses compute power demands, a most promising solution is to develop applicationspecific architectures using ASICs.Interestingly, every processor architecture has its pros and cons such as the energy efficiency and reconfigurability of FPGAs, but they are slow and hard to program, the high performance of GPU processors but they are power-hungry, the flexibility of general-purpose CPU architectures but are slow with ML computations, etc.Of all these processor architectures, ASICs possess the best performance in terms of energy efficiency because they are hardwired designs to target a specific application.They consume very low power and incur very low costs too.They, however, trade-off flexibility for performance and take a lot of time to market.ASICs are thus gaining renewed interest in the design and development of application-specific machine learning architectures, with Google TPU being a successful case study.

Conclusions
Machine learning models are fast proliferating embedded devices with limited computational power and memory space.These machine learning models are compute and memory intensive and thus, face the critical limitation of available hardware resources in embedded and mobile devices.In this paper, optimization techniques and various applications of machine learning algorithms within resource-limited environments are presented.We first survey the embedded machine learning space to determine the common machine learning algorithms adopted and select key compute and memory-intensive models such as HMMs, k-NNs, SVMs, GMMs, and DNNs.We survey specialized optimization techniques commonly adopted to squeeze these algorithms within resource-limited environments.Also, we present different hardware platforms such as microcontroller units, mobile devices, accelerators, and even TinyML frameworks, which are used to port these algorithms to resource-limited MCUs.Furthermore, we survey the challenges encountered in embedded machine learning and present a more detailed exposition on certain hardware-oriented and algorithm-oriented optimization schemes to address these bottlenecks.Additionally, an exciting look is given to different hardware and algorithm-based optimization techniques, including model pruning, data quantization, reduced precision, tiling, and others to determine which optimization technique best suits the different ML algorithms.Interesting and viable application areas, open research issues, and key take-away lessons are presented in this intersection of embedded systems and machine learning.Conclusively, this survey attempts to create awareness for the passionately interested researcher to kick-start an adventure into this promising landscape of embedded machine learning.
Institutional Review Board Statement: This article does not contain any studies with human participants or animals performed by any of the authors.
Informed Consent Statement: Not applicable.

Figure 1 .
Figure 1.The layout of Embedded Machine Learning Computing Architectures and Machine Learning and Optimization Techniques.

Figure 1 .
Figure 1.The layout of Embedded Machine Learning Computing Architectures and Machine Learning and Optimization Techniques.

Figure 2 .
Figure 2. Description of a spectrum of certain CNN models to reveal their compute and memory demand: (a) Describes the Memory Demand of these models in terms of the number of weight parameters in (millions) (b) Computational Demand of these models in terms of their number of operations (GOPs).

Figure 2 .
Figure 2. Description of a spectrum of certain CNN models to reveal their compute and memory demand: (a) Describes the Memory Demand of these models in terms of the number of weight parameters in (millions) (b) Computational Demand of these models in terms of their number of operations (GOPs).

Figure 3 .
Figure3.This graph describes the energy consumption and prediction accuracy of a DNN as a function of the Arithmetic Precision adopted for a single MAC unit in a 45 nm CMOS[124].It may be deduced from the graph that lower number precisions consume less power than high precisions with no loss in prediction accuracy.However, we can observe that when precision is reduced below a particular threshold (16 bit fp), the accuracy of the model is greatly affected.Thus, quantization

Figure 3 .
Figure3.This graph describes the energy consumption and prediction accuracy of a DNN as a function of the Arithmetic Precision adopted for a single MAC unit in a 45 nm CMOS[124].It may be deduced from the graph that lower number precisions consume less power than high precisions with no loss in prediction accuracy.However, we can observe that when precision is reduced below a particular threshold (16 bit fp), the accuracy of the model is greatly affected.Thus, quantization may be performed successfully to conserve energy but quantizing below 16-bit fp may require retraining and fine-tuning to restore the accuracy of the model.

Table 2 .
Machine Learning Techniques in Resource-Constrained Environments.

Table 3 .
Problems and Algorithms of HMM.

Table 4 .
Optimization Schemes for HMM.

Table 9 .
Description of some DNN models and their parameters.

Table 10 .
Parameters for popular CNN models.

Table 10 .
Parameters for popular CNN models.

Table 14 .
Some literature on mobile machine learning.

Table 17 .
Energy Consumption in (pJ) of performing operations.

Table 17 .
Energy Consumption in (pJ) of performing operations.

Table 18 .
Intelligent sensor systems and IoTs.

Table 19 .
Deep learning in mobile devices.

Table 20 .
Deep learning training in general purpose graphic processing units (GPGPUs).

Table 21 .
Deep Learning in heterogeneous computing systems.
The pooling and fully connected layers are accelerated by implementing single work-item kernels.

Table 23 .
Energy-efficient hardware design and architectures.

Table 24 .
Future research directions.