Machine Learning in Wireless Sensor Networks for Smart Cities: A Survey

Artificial intelligence (AI) and machine learning (ML) techniques have huge potential to efficiently manage the automated operation of the internet of things (IoT) nodes deployed in smart cities. In smart cities, the major IoT applications are smart traffic monitoring, smart waste management, smart buildings and patient healthcare monitoring. The small size IoT nodes based on low power Bluetooth (IEEE 802.15.1) standard and wireless sensor networks (WSN) (IEEE 802.15.4) standard are generally used for transmission of data to a remote location using gateways. The WSN based IoT (WSN-IoT) design problems include network coverage and connectivity issues, energy consumption, bandwidth requirement, network lifetime maximization, communication protocols and state of the art infrastructure. In this paper, the authors propose machine learning methods as an optimization tool for regular WSN-IoT nodes deployed in smart city applications. As per the author’s knowledge, this is the first in-depth literature survey of all ML techniques in the field of low power consumption WSN-IoT for smart cities. The results of this unique survey article show that the supervised learning algorithms have been most widely used (61%) as compared to reinforcement learning (27%) and unsupervised learning (12%) for smart city applications.


Introduction
A smart city is an urban area that uses remote sensors and the Internet of Things (IoT) enabling technologies to collect data from different locations and uses to enhance the quality of life of the people. The low power, low data rate wireless sensor networks (WSN) are used for monitoring and control applications in smart cities. The WSN nodes are used as the underlying technology infrastructure in the IoT. In the IoT, the "things" refer to the tiny embedded physical sensing devices (i.e., WSN nodes) connected to the internet to perform a specific application. Currently, a new revolutionary technique known as artificial intelligence (AI) and machine learning (ML) is evolving as the future of fully automated IoT applications. Machine learning is a part of AI, in which, the computer algorithms learn by themselves by improving from past experiences. A detailed survey of ML algorithms was performed in [1] until the year 2013. As the ML and IoT, technologies are emerging rapidly, therefore, the authors extend their survey work also. The IoT applications in smart cities are smart traffic monitoring [2], smart grids [3], smart waste management [4], smart agriculture [5], smart medical healthcare [6], etc. Table 1 provides a full form of all important abbreviations used in this paper. The major problems in WSN based IoT (WSN-IoT) are fully autonomous operation, maximum network lifetime, energy efficiency, quality of service (QoS), cross-layer optimization, high bandwidth requirement, sensor data analysis, cloud computing, communication protocol design, etc. Currently, the industrial IoT (IIoT) or industry 4.0 is the biggest revolution for smart industries, smart manufacturing sector, automobile sector, smart cities and medical healthcare sector. Worldwide, various major companies like Microsoft, Google and Amazon are working on the development of AI and ML-based algorithms in advanced IoT applications for smart cities.
Machine learning can be applied in WSN-IoT for dynamic updating of routing tables in WSNs, node localization in mobile WSN-IoT nodes, identification and separation of faulty nodes for network optimization and prediction of the amount of energy harvesting in energy harvesting WSN (EH-WSN). Through this paper, the authors have tried to answer the following research questions: Why machine learning methods are used in WSN-IoT? What is its superiority of using ML over traditional optimization methods in WSN-IoT? Why is the smart city a typical use case of IoT applications?
IoT offers new opportunities for smart cities to use data to manage traffic, reduce pollution and make better use of infrastructure. The following are the advantages of using machine learning in traditional WSN-IoT: • WSNs are generally deployed in a dynamically changing environment. Therefore, self-adaption to the new environment is expected from a fully automated IoT scenario. • Unknown parameter monitoring requires automatic adjustment of network topology and configurations, e.g., temperature measurement in a glacier or volcano monitoring. • Lack of accurate mathematical models of the unknown parameters in WSN-IoT. • WSN-IoT deals with a large amount of sensor data, therefore the correlation between different data set may be of critical concern.
Electronics 2021, 10, 1012 3 of 22 • Integration of WSN in IoT using cloud-based services for better monitoring and control. • Future predictions and possible actions in WSN-IoT.

•
The IoT generates a large amount of data from millions of sensor nodes. Machine learning is powered by data and generates useful information from previous data. Machine learning uses past IoT data to identify hidden patterns and builds models that help predict future behavior and events.
As WSN-IoT are resource-limited (finite bandwidth and power availability) therefore, there are some limitations for running ML-based inferences on IoT nodes also such as: • A large number of computations are required to process the more amount of data, hence computation complexity increases. • Additional power consumption.

•
Training of WSN-IoT nodes for various ML algorithms requires complex operations and multi-domain skilled programmers.
The following are the contributions of this survey article in the field of WSN-IoT: • In this paper, ML techniques are proposed as an optimized solution for traditional WSN-IoT problems in smart cities.

•
Design guidelines of the WSN-IoT framework using AI and ML have been proposed. • An in-depth literature survey of WSN-IoT in smart cities is presented in detail for ML engineers and data scientists.
This paper is organized as follows: Section 2 provides operation of WSN-IoT in smart cities, Section 3 provides machine learning for WSN-IoT, Section 4 provides open research issues in WSN-IoT, which can be solved by machine learning techniques, Section 5 provides a literature survey of machine learning in WSN, Section 6 provides a summary of ML techniques in WSN-IoT, Section 7 provides the survey report and, finally, Section 8 provides the conclusion and future work.

Wireless Sensor Networks Based Internet of Things (WSN-IoT)
The operation of WSN-IoT in a smart city is shown in Figure 1. Here, the WSN nodes are deployed in smart city applications such as smart traffic monitoring, smart grids in buildings, remote health care monitoring, smart agriculture and industrial applications.
The function of an IoT-WSN node deployed in a smart city is to continuously monitor and control any physical quantity like temperature, humidity, pressure, acceleration, etc. The main function of these sensor nodes is to sense the data and send it to the main WSN-IoT gateway node. From the gateway node, the data is sent to the cloud server. At the IoT cloud, cloud computing takes place. The IoT cloud is directly connected to remote servers, user mobile phones, computers, mobile phone towers, etc. The IoT and machine learning tasks require a large amount of data processing and memory requirements. Therefore, the IoT cloud server is designed as a high processing, high-performance computer with huge storage capacity. However, the WSN end nodes have small computing capabilities with limited processing, small storage and finite non-rechargeable battery power supply.
The WSN-IoT end nodes based on the IEEE 802.15.4 standard have the maximum data rate of 250 kbps only. In WSN-IoT, the end nodes are powered by two AA-size batteries (1.5 volts, 1000 mAh), and the gateway is connected to the mains power supply. Furthermore, if the WSN-IoT nodes are powered by renewable energy harvesting power supplies [7], then machine learning algorithms can also be utilized to predict the future available energy in IoT-WSNs. In the battery management system, machine learning techniques can be used for tracking maximum power point technique (MPPT) algorithms [8,9]. As these WSN sensor nodes generate a lot of sensor data, therefore, machine learning algorithms can also be applied to them for data analysis, data prediction and other suitable tasks.
Furthermore, if the WSN-IoT nodes are powered by renewable energy harvesting power supplies [7], then machine learning algorithms can also be utilized to predict the future available energy in IoT-WSNs. In the battery management system, machine learning techniques can be used for tracking maximum power point technique (MPPT) algorithms [8,9]. As these WSN sensor nodes generate a lot of sensor data, therefore, machine learning algorithms can also be applied to them for data analysis, data prediction and other suitable tasks. In smart cities, the WSN-IoT networks contain the connectivity technologies/protocols as shown in Table 2. Table 2 shows IoT communication technologies such as Bluetooth [10], based on IEEE 802.15.1 standards [11], RFID [12], IEEE 802.16 [13], ZigBee [14] based on IEEE 802.15.4 [15], Wi-Fi [16], based on IEEE 802.11 [17], LoRa-WAN [18] based on IEEE 802.11 series [19], 4G/5G cellular networks [20], based on WiMAX standard IEEE 802.16e [21] and 5G based on IEEE 1941 [22,23], respectively.
At the WSN-IoT node device level, the machine learning algorithms can be applied from the cloud for its autonomous operation. The IoT device sends the sensor data to the cloud server. From the IoT cloud, the user can monitor and control the application using a mobile phone, laptop or desktop PC and personal digital assistant. Currently, many popular cloud service companies provide free, but the limited amount of sensor data to be stored in their cloud storage. For example, Microsoft Azure IoT, Amazon web service In smart cities, the WSN-IoT networks contain the connectivity technologies/protocols as shown in Table 2. Table 2 shows IoT communication technologies such as Bluetooth [10], based on IEEE 802.15.1 standards [11], RFID [12], IEEE 802.16 [13], ZigBee [14] based on IEEE 802.15.4 [15], Wi-Fi [16], based on IEEE 802.11 [17], LoRa-WAN [18] based on IEEE 802.11 series [19], 4G/5G cellular networks [20], based on WiMAX standard IEEE 802.16e [21] and 5G based on IEEE 1941 [22,23], respectively.
At the WSN-IoT node device level, the machine learning algorithms can be applied from the cloud for its autonomous operation. The IoT device sends the sensor data to the cloud server. From the IoT cloud, the user can monitor and control the application using a mobile phone, laptop or desktop PC and personal digital assistant. Currently, many popular cloud service companies provide free, but the limited amount of sensor data to be stored in their cloud storage. For example, Microsoft Azure IoT, Amazon web service (AWS), Google cloud platform, Cisco IoT cloud connect, IBM Watson IoT and Thing speak IoT by Mathworks Incorporation, USA. Table 3 shows open system interconnect (OSI) model layer data communication network protocols of an upper higher level separately. Some very popular higher-level IoT protocols are message query telemetry transport (MQTT) protocol [24], advanced message queuing protocol (AMQP) [25], constrained application protocol (CoAP) [26] and data distribution service (DDS) protocol [27]. These IoT protocols are based on the IEEE P1451-99 [28] standard for harmonization of internet of things (IoT) devices and systems. In smart city IoT applications, these protocols are used with TCP, UDP and cloud-based services. In WSN, a smart transducer interface protocol [29] is used for sensor management. It is based on an IEEE standard called the P21451 interoperability interface standard [30].

Training Process
Here, first of all, data is acquired from a particular application. The features are extracted from this raw data. For example, if the data is image data then the colors, pixels, brightness and contrast of all images database are extracted. Then the features are classified according to the requirement of the machine learning process. Now some training

Training Process
Here, first of all, data is acquired from a particular application. The features are extracted from this raw data. For example, if the data is image data then the colors, pixels, brightness and contrast of all images database are extracted. Then the features are classified according to the requirement of the machine learning process. Now some training examples are applied to the basic initial algorithms for their learning or improvement. Thus, algorithms are trained and optimized according to the data patterns.

Testing Process
Now the next step is to deploy this trained WSN in any real-life application. In real life, the unknown data is taken as input and the features are extracted from it. These extracted features are applied to an already trained algorithm. The output of the trained algorithm is classified as data predictions.

WSN Actions
Finally, based on predicted output data the necessary actions by the WSN are decided.

Open Research Problems in WSN-IoT Which Can Be Solved by Machine Learning Techniques
The following are the currently open research issues in WSN-IoT, which can be solved by ML techniques.

IoT Node Localization
In a WSN scenario, the current location identification of a sensor node is called node localization. In mobile WSN nodes, path planning is a very important step. Node localization is considered a classification problem because all the nodes are divided (classified) into range-based and range-free nodes. Several ML algorithms like, SVM, K-NN and RL-based techniques (Q-learning, SARSA) are used in WSN-IoT for node localization as a classification problem [31].

IoT Node Coverage and Connectivity
In a WSN scenario, the sensing coverage is the field of interest (FOI), in which at least one sensor node covers all the points. Therefore, the optimal placement of sensor nodes is a design issue. To maximize the WSN lifetime, the connectivity should be proper between the neighbor nodes [32].

Routing Layer Issues
The processing for sending the data packets from one node to another via intermediate nodes is called routing. In the routing process, long routing tables are maintained by the gateway nodes, which consists of the source and destination address of all the packets in the network. In WSN, the sensed data is sent towards the main gateway node by the end nodes. If the routing path is very long then unnecessary energy is wasted in a WSN network. Therefore, smart routing algorithms need to be designed carefully to find the optimal routes between end node and gateway nodes. Several machine learning techniques such as decision tree, random forest, ANN, SVM and Bayesian learning are used to find the optimal path in WSNs [33].

MAC Layer Issues
The MAC layer controls the medium accessing technique in WSN. The sensor MAC (SMAC) protocol is generally used in WSN. Reinforcement learning (RL) based algorithms are used for MAC protocol design in WSN. The RL-MAC techniques control the sleep, wake, transmission and reception in sensor networks [34].

Sensor Data Aggregation
In a smart city, thousands of small sensor nodes are deployed to measure the same physical quantity, e.g., temperature, humidity, light, carbon dioxide (CO 2 ) gases, etc. Several sensors may report the same information to the gateway. This large amount of data is difficult to handle by the gateway. Therefore, sensor data aggregation is important in WSNs for smart city applications. Data aggregation means collecting and summarizing useful information from multiple sources. In this process, the data redundancy and data accuracy are improved. Data aggregation saves the power consumption of WSN nodes and hence improves the network lifetime also. Machine learning is useful for data aggregation. The cluster aggregates the data from the cluster head and transmits it to the base station. ML techniques based on artificial neural networks (ANN) and quality (Q)-learning algorithms are useful for data aggregation tasks in WSN-IoT [35].

Event Monitoring and Target Detection
In a smart city application, the WSNs are deployed for event monitoring and target detection such as intrusion detection and traffic monitoring. In WSN, node failure, target recovery and tracking latency from sensing nodes are required. Various ML techniques like Bayesian, Q learning and genetic algorithms are used for event monitoring and target tracking in WSNs. Applying, ML techniques in WSNs can be useful to detect an event or target from the complex image sensor data [36].

Energy Harvesting
Energy harvesting is the process of extracting environmental energy from the sun, wind, tides, radio waves, etc., and converts it into the corresponding electrical energy. The broad objective of energy harvesting is to save our limited available fossil fuels (coal, oil and gases). However, in the smart city application, energy harvesting can also be used for achieving maximum network lifetime in rechargeable battery-based WSN-IoT nodes. Furthermore, the ML techniques are used in energy harvesting WSN-IoT for future available energy prediction tasks. The ML algorithms like regression technique and reinforcement learning techniques (Q-learning) are suitable for energy harvesting applications. Generally, solar energy, radio frequency (RF) waves and wind energy are used with rechargeable battery-based WSNs. Harvested energy prediction, battery power management are the tasks that can be optimized by using the ML algorithms in traditional WSN-IoT [37].

Node Query Processing
In WSNs, the end nodes, cluster heads and gateway nodes perform various types of queries such as sensor data aggregation, routing paths, synchronization and control operations, packet delivery with each other, etc. The k-nearest neighborhood (k-NN) based ML techniques are used for sensor data queries in WSN. Table 4 shows some WSN-IoT research issues with ML-based solutions [38].  To predict the amount of battery energy required to maximize network lifetime, Prediction of energy harvesting availability in the future.
8 Node Query processing [38] k-NN Node Beacon sending, Handshake for data transfer

Literature Survey of Machine Learning in WSN-IoT
Our literature survey of machine learning algorithms for WSN-IoT is shown in Figure 4. Our literature survey is divided into the following categories: 1.

Literature Survey of Machine Learning in WSN-IoT
Our literature survey of machine learning algorithms for WSN-IoT is shown in Figure  4. Our literature survey is divided into the following categories: 1. Supervised Learning, 2. Unsupervised learning, 3. Reinforcement learning.

Literature Survey of Supervised Machine Learning for WSN-IoT
In supervisor learning, data are labeled. In other words, we provide an input data variable (x) to the system. The system predicts output data (y) depending upon the type of input and system function. The objective of supervised learning is to approximate the mapping function so that when a new unknown input data is applied then, the output (y)

Literature Survey of Supervised Machine Learning for WSN-IoT
In supervisor learning, data are labeled. In other words, we provide an input data variable (x) to the system. The system predicts output data (y) depending upon the type of input and system function. The objective of supervised learning is to approximate the mapping function so that when a new unknown input data is applied then, the output (y) can be predicted [39,40].
In this section, we will discuss various supervised learning algorithm, which can be applied in WSN-IoT applications. The supervised learning algorithms are used in WSN for target tracking, localization of Nodes, event monitoring, data security, fault detection, etc.
There are two types of supervisor learning as regression and classification.

Regression
In regression, the output variable (y) has some continuous numerical value like rupees, height, weight, etc. Regression is applied to solve various issues in WSNs such as localization, connectivity problem, data aggregation and energy harvesting [41].

Classification
In classification, the output variable(y) is a category of objects like the type of colors (e.g., red or yellow), type of diseases (e.g., fever or fracture), etc. There are the following types of classification problems as k-NN, decision tree, ANN, Bayesian learning, etc [42].
(A). K-nearest neighborhood (k-NN) [43]: This is an instance-based supervised learning algorithm. Here, all training instances are stored in a master database. When a new instance query (x q ) arrives then this new query is compared with the stored database and classified results are derived. In the k-NN algorithm the distance between each data points is calculated by using the Euclidian distance formula where q and p are data points and d is the distance between them. In [44] data streaming in IoT using the k-NN algorithm is proposed. The KNN algorithm is also used for the early detection of agriculture pests, diseases, sensor node failure and fault detection issues [45].
(B). Decision tree (DT) [46]: In the DT algorithm, the main task is to calculate the attribute of the root node from each level. This process is called attribute selection. There are two methods for attribute selection as information gain and Gini index. The average amount of information is called entropy and is given as Gini index is a metric to measure how often a randomly chosen element would be incorrectly identified.
where pi is the probability of occurrence of an i th event, n is the number of training examples. It is a classification method for predicting labels of data by iterating the input data through a learning tree. During this process, the feature properties are compared relative to decision conditions to reach a specific category. In WSNs, decision tree (DT) algorithms like ID3 are used to identify the link reliability, mean time to failure (MTTF) and mean time to repair (MTTR). Figure 5 shows an example of the DT algorithm, in which the decisions are taken by a WSN node system for different conditions. Here, if the 1st condition of WSN node battery voltage is greater than 2.7 volts (i.e., V battery > 2.7 volts) is not true, then all decision flows towards the left-hand side of the tree. On the left-hand side, the 2nd condition to be checked is supply availability. On the other hand, if the 1st condition (i.e., V battery > 2.7 volts) is true, then all decisions flow towards the right-hand side of the tree. (C). Artificial neural networks (ANNs) [47][48][49]: ANN is inspired by the human brain architecture learning mechanism. The basic unit of AAN is perceptron, which is equivalent to a neuron in the human brain as shown in Figure 6. Backpropagation is most the common learning algorithm in ANN. An activation function is any step function, a cosine function or a sigmoid function, etc. The error ( ) in backpropagation is calculated until the convergence meets as where Tk is the target value, Ok is the actual output value and (n) is the number of training examples. In backpropagation algorithms, the learning is determined by the sum of weighted links between the inputs layer, middle layer and output layer. Finally, the weights (w) are updated using the rule as:  (C). Artificial neural networks (ANNs) [47][48][49]: ANN is inspired by the human brain architecture learning mechanism. The basic unit of AAN is perceptron, which is equivalent to a neuron in the human brain as shown in Figure 6. Backpropagation is most the common learning algorithm in ANN. An activation function is any step function, a cosine function or a sigmoid function, etc. The error (E) in backpropagation is calculated until the convergence meets as where T k is the target value, O k is the actual output value and (n) is the number of training examples. In backpropagation algorithms, the learning is determined by the sum of weighted links between the inputs layer, middle layer and output layer. Finally, the weights (w) are updated using the rule as: (C). Artificial neural networks (ANNs) [47][48][49]: ANN is inspired by the human brain architecture learning mechanism. The basic unit of AAN is perceptron, which is equivalent to a neuron in the human brain as shown in Figure 6. Backpropagation is most the common learning algorithm in ANN. An activation function is any step function, a cosine function or a sigmoid function, etc. The error ( ) in backpropagation is calculated until the convergence meets as where Tk is the target value, Ok is the actual output value and (n) is the number of training examples. In backpropagation algorithms, the learning is determined by the sum of weighted links between the inputs layer, middle layer and output layer. Finally, the weights (w) are updated using the rule as:  (D). Support Vector Machine (SVM) [50,51]: SVM is the most popular algorithm, which can be used for both classification and regression of data. Generally, it is used for classification in the WSN-IoT scenario. In SVM, we plotted each data value as a point in an n-dimensional X-Y plane as shown in Figure 7.
Electronics 2021, 10, x FOR PEER REVIEW 13 of 24 (D). Support Vector Machine (SVM) [50,51]: SVM is the most popular algorithm, which can be used for both classification and regression of data. Generally, it is used for classification in the WSN-IoT scenario. In SVM, we plotted each data value as a point in an n-dimensional X-Y plane as shown in Figure 7.
where, w is the width between two support vectors planes, (xi) is the data point value, (yi) is the assigned label to each data point respectively, (b) is the distance of hyperplane from the origin and (n) is the total number of training examples. In WSN, the SVM technique is used for the localization of mobile sensor nodes. (E). Bayesian Learning [52][53][54]: Bayesian learning is an important statistical technique to calculate the conditional probability of an event. Bayesian learning is based on the very popular Bayes' theorem for conditional probability as: where, P(A) is called the probability of occurrence of event A, P(B) is called the probability of occurrence of event B, P(A|B) is called conditional probability of occurrence of event A if B is given and P(B|A) is called conditional probability of occurrence of event B if A is given earlier. In WSN, it is used for calibration, clustering and detecting outlier (i.e., outsider or different types) nodes in a group of several nodes deployment.

Literature Survey of Unsupervised Machine Learning for WSN-IoT
In unsupervised machine learning, only the input data variable (x) is known to the machine. The output data variable (y) is not known to the machine or system. The objective of unsupervised ML is to know the probability distribution model of input data [55]. such that, where, w is the width between two support vectors planes, (x i ) is the data point value, (y i ) is the assigned label to each data point respectively, (b) is the distance of hyperplane from the origin and (n) is the total number of training examples. In WSN, the SVM technique is used for the localization of mobile sensor nodes.
(E). Bayesian Learning [52][53][54]: Bayesian learning is an important statistical technique to calculate the conditional probability of an event. Bayesian learning is based on the very popular Bayes' theorem for conditional probability as: where, P(A) is called the probability of occurrence of event A, P(B) is called the probability of occurrence of event B, P(A|B) is called conditional probability of occurrence of event A if B is given and P(B|A) is called conditional probability of occurrence of event B if A is given earlier. In WSN, it is used for calibration, clustering and detecting outlier (i.e., outsider or different types) nodes in a group of several nodes deployment.

Literature Survey of Unsupervised Machine Learning for WSN-IoT
In unsupervised machine learning, only the input data variable (x) is known to the machine. The output data variable (y) is not known to the machine or system. The objective of unsupervised ML is to know the probability distribution model of input data [55].

Clustering Technique
In this method, the inherent groups in the data are identified, such as a particular type of customer database that purchases a specific type of goods from specified stores in smart cities. Figure 8 shows three clusters of class red, green and blue [56][57][58]. To make a cluster between data points of two vectors X and Y the Euclidian distance (d) is calculated as: The k-means clustering [59][60][61], PCA [62][63][64] and ANN [65,66] algorithm are most widely used for clustering in WSN-IoT.

Clustering Technique
In this method, the inherent groups in the data are identified, such as a particular type of customer database that purchases a specific type of goods from specified stores in smart cities. Figure 8 shows three clusters of class red, green and blue [56][57][58]. To make a cluster between data points of two vectors and the Euclidian distance ( ) is calculated as: The k-means clustering [59][60][61], PCA [62][63][64] and ANN [65,66] algorithm are most widely used for clustering in WSN-IoT.

Literature Survey of Reinforcement Learning (RL) for WSN-IoT
In the reinforcement learning algorithm, there are two main components called the agent and the environment as shown in Figure 9 [67,68]. The agent refers to the WSN-IoT node and the environment refers to the type of application deployment, e.g., temperature monitoring, pollution monitoring, smart agriculture monitoring, etc [69]. The WSN-IoT node agent performs the actions (At) of sensing and actuation at time state (t). The application environment changes its state (St) and provides a reward (Rt) to the agent as shown in Figure 9. There are many types of reinforcement learning algorithms given as (A) Qlearning [70,71], (B) SARSA [72], temporal difference (TD) [73] and deep Q learning [74]. The Q value is defined as a sum of discounted rewards starting from the state (sk), taking action (ak) following any policy (π) as: where, E = expectation, s = state, a = action, k = numerical integer value (0 to N-1), γ = discount rate and r = rewards value.

Literature Survey of Reinforcement Learning (RL) for WSN-IoT
In the reinforcement learning algorithm, there are two main components called the agent and the environment as shown in Figure 9 [67,68]. The agent refers to the WSN-IoT node and the environment refers to the type of application deployment, e.g., temperature monitoring, pollution monitoring, smart agriculture monitoring, etc [69]. The WSN-IoT node agent performs the actions (A t ) of sensing and actuation at time state (t). The application environment changes its state (S t ) and provides a reward (R t ) to the agent as shown in Figure 9. There are many types of reinforcement learning algorithms given as (A) Q-learning [70,71], (B) SARSA [72], temporal difference (TD) [73] and deep Q learning [74]. The Q value is defined as a sum of discounted rewards starting from the state (s k ), taking action (a k ) following any policy (π) as: where, E = expectation, s = state, a = action, k = numerical integer value (0 to N-1), γ = discount rate and r = rewards value. Electronics 2021, 10, x FOR PEER REVIEW 15 of 24 Figure 9. Reinforcement learning (RL) for WSN-IoT.

Summary of Literature Survey of ML Techniques for WSN-IoT
Finally, this section provides a summary of all machine learning (ML) techniques proposed as an optimized solution for WSN-IoT problems. In a smart city, the major challenges are as follows: smart education, smart classrooms, smart traffic monitoring, rain water harvesting, smart grids in smart buildings smart healthcare in hospitals, smart agriculture, industrial IoT (or Industry 4.0), smart waste management, smart governance, smart environment monitoring, etc. Now, we will map each problem of smart cities with the solution provided by machine learning algorithms in WSN-IoT. The WSN node localization problem is considered as a classification or multivariate regression task in the ML domain. Therefore, SVM classification [50] or SVM regression model [51] algorithms are applied as a solution for the node localization problems in WSN-IoT.
The security issues are tackled by correlation techniques and handled by using the Bayesian learning technique as [52][53][54]. Cluster head selection tasks in WSN-IoT are considered clustering tasks in the ML domain. The k-NN [59][60][61], PCA [62][63][64] and ANN [65,66] have been used for clustering. WSN node energy management is considered a prediction problem in the ML domain. The Q-Learning [71,72] has been applied to predict the energy issues. Similarly, in energy harvesting based WSN (EH-WSN) predicts future energy availability using reinforcement learning algorithms like Q-Learning [73], SARSA [74] and deep Q-learning [75] have been applied. Event monitoring and fault detection problems are considered as classification models. These are solved by SVM [76,77] and rule-based Learning [78][79][80][81] algorithms.
The routing of data packets in WSN-IoT is considered a classification problem in the domain of machine learning. The routing optimization algorithms such as genetic algorithms [82] and classification algorithms such as Markov decision process (MDP) in decision tree [83], random forest [84] and Q-learning (QELAR) [85] have been used.
At the MAC layer, the packet scheduling task is considered a regression task in the machine learning context. Therefore, SVM [86], deep neural networks [87] have been applied at the MAC layer. The QoS (latency, bandwidth and coverage) in IoT is considered a prediction problem in IoT. Therefore, Q-Learning, ANN [88] and SVM [89] have been used as the solution. Spectrum sharing in WSN-IoT is a self-learning problem, which can be solved by a deep reinforcement learning [90,91] technique. Data aggregation is a technique in which redundant information is removed before processing by the server. Data aggregation is treated as a regression problem and is solved by SVM [92] and reinforcement [93] respectively. Table 5 shows the summary of the literature survey of ML techniques for WSN-IoT.

Summary of Literature Survey of ML Techniques for WSN-IoT
Finally, this section provides a summary of all machine learning (ML) techniques proposed as an optimized solution for WSN-IoT problems. In a smart city, the major challenges are as follows: smart education, smart classrooms, smart traffic monitoring, rain water harvesting, smart grids in smart buildings smart healthcare in hospitals, smart agriculture, industrial IoT (or Industry 4.0), smart waste management, smart governance, smart environment monitoring, etc. Now, we will map each problem of smart cities with the solution provided by machine learning algorithms in WSN-IoT. The WSN node localization problem is considered as a classification or multivariate regression task in the ML domain. Therefore, SVM classification [50] or SVM regression model [51] algorithms are applied as a solution for the node localization problems in WSN-IoT.
The security issues are tackled by correlation techniques and handled by using the Bayesian learning technique as [52][53][54]. Cluster head selection tasks in WSN-IoT are considered clustering tasks in the ML domain. The k-NN [59][60][61], PCA [62][63][64] and ANN [65,66] have been used for clustering. WSN node energy management is considered a prediction problem in the ML domain. The Q-Learning [71,72] has been applied to predict the energy issues. Similarly, in energy harvesting based WSN (EH-WSN) predicts future energy availability using reinforcement learning algorithms like Q-Learning [73], SARSA [74] and deep Q-learning [75] have been applied. Event monitoring and fault detection problems are considered as classification models. These are solved by SVM [76,77] and rule-based Learning [78][79][80][81] algorithms.
The routing of data packets in WSN-IoT is considered a classification problem in the domain of machine learning. The routing optimization algorithms such as genetic algorithms [82] and classification algorithms such as Markov decision process (MDP) in decision tree [83], random forest [84] and Q-learning (QELAR) [85] have been used.
At the MAC layer, the packet scheduling task is considered a regression task in the machine learning context. Therefore, SVM [86], deep neural networks [87] have been applied at the MAC layer. The QoS (latency, bandwidth and coverage) in IoT is considered a prediction problem in IoT. Therefore, Q-Learning, ANN [88] and SVM [89] have been used as the solution. Spectrum sharing in WSN-IoT is a self-learning problem, which can be solved by a deep reinforcement learning [90,91] technique. Data aggregation is a technique in which redundant information is removed before processing by the server. Data aggregation is treated as a regression problem and is solved by SVM [92] and reinforcement [93] respectively. Table 5 shows the summary of the literature survey of ML techniques for WSN-IoT. Spectrum sharing is Self-learning, rewards system tasks in ML.

11
Pandemic medical treatment (e.g., COVID-19 or Corona Virus) Data Aggrigation SVM [92] and Reinforcement [93] Data Aggregation is treated as a Regression task in the ML context.
The ML and WSN-IoT pair can act as a boon for the medical healthcare sector in smart hospitals in smart cities. For example, as per a world health organization (WHO) report [94], an international pandemic called corona virus disease (COVID-19) caused the death of 716,075 people worldwide until the end of the year 2020. In smart hospitals, the advanced ML techniques with efficiently deployed WSN-IoT can be applied for the treatment of infected patients placed in quarantine. The sensors attached to biomedical instruments can send patient's data over the internet to the doctors for medical diagnosis. Thus, doctors need not go near to the patients and hence avoid/reduce the virus spread in the smart cities and society.

Our Survey Report of ML Techniques in WSN-IoT
We performed this survey task by visiting various websites, journals, magazines and research papers for machine learning in WSN-IoT for smart cities. By surveying on the internet for ML techniques addressing WSN-IoT problems, we included many papers available as shown in Table 6 from the year 2010 to 2021. Table 7 shows category-wise ML algorithms addressing WSN-IoT issues with percentage contribution. Figure 10 shows the graphical representation of major ML algorithms in WSN-IoT.    Figure 11 shows a quick look and easy to understand graphical representation of major ML algorithms used in WSN-IoT.    Figure 11 shows a quick look and easy to understand graphical representation of major ML algorithms used in WSN-IoT. Figure 11. Graphical representation of major ML algorithms used in WSN-IoT. Figure 11. Graphical representation of major ML algorithms used in WSN-IoT. Table 8 shows the dominance of ML techniques in WSN-IoT problems. Figure 12 shows a quick graphical representation of dominance of RL, supervised and unsupervised learning algorithms in WSN-IoT.  Table 8 shows the dominance of ML techniques in WSN-IoT problems. Figure 12 shows a quick graphical representation of dominance of RL, supervised and unsupervised learning algorithms in WSN-IoT. Unsupervised Learning 12% Figure 12. Graphical representation of RL, supervised and unsupervised learning algorithms in WSN-IoT.

Conclusions and Future Work
In this paper, we outlined various machine learning algorithms in WSN-IoT for smart city applications. In this paper, we performed a detailed survey of ML techniques in WSN-IoT for the smart city challenges. From this survey, it was concluded that the supervised

Conclusions and Future Work
In this paper, we outlined various machine learning algorithms in WSN-IoT for smart city applications. In this paper, we performed a detailed survey of ML techniques in WSN-IoT for the smart city challenges. From this survey, it was concluded that the supervised learning algorithms have been used with the highest amount, i.e., 61% as compared to RL at 27% and unsupervised at 12% only. The ML algorithms are so versatile and powerful that a single type of ML algorithm can be used for multiple tasks in WSN-IoT in smart cities. For example, the powerful SVM algorithm can be used for classification and for regression tasks in WSN-IoT. In the future, a more powerful and complex algorithm will emerge, which minimizes human intervention. The futuristic IoT-based solution for smart cities will include machine learning techniques. For example, the heart stroke rehabilitation system [95] in smart health care using LDA, MLP and SVM algorithms. Next-generation smart cities will have ultra-dense cellular IoT networks using high-performance machine learning algorithms [96][97][98][99][100].
Funding: This research receives no external funding.