Examining the Performance of Fog-Aided, Cloud-Centered IoT in a Real-World Environment

The fog layer provides substantial benefits in cloud-based IoT applications because it can serve as an aggregation layer and it moves the computation resources nearer to the IoT devices; however, it is important to ensure adequate performance is achieved in such applications, as the devices usually communicate frequently and authenticate with the cloud. This can cause performance and availability issues, which can be dangerous in critical applications such as in the healthcare sector. In this paper, we analyze the efficacy of the fog layer in different architectures in a real-world environment by examining performance metrics for the cloud and fog layers using different numbers of IoT devices. We also implement the fog layer using two methods to determine whether different fog implementation frameworks can affect the performance. The results show that including a fog layer with semi-heavyweight computation capability results in higher capital costs, although the in the long run resources, time, and money are saved. This study can serve as a reference for fundamental fog computing concepts. It can also be used to walk practitioners through different implementation frameworks of fog-aided IoT and to show tradeoffs in order to inform when to use each implementation framework based on one’s objectives.


Introduction
Cloud computing is an emerging technology that offers high computational power and permanent storage for the Internet of Things (IoT). In cloud-based IoT environments, as the number of IoT devices increases, the amount of data generated from the IoT also increases. This causes high latency due to the long distances between the IoT devices and the cloud. The exchange of data between IoT devices and the cloud increases the utilization of bandwidth and requires increasing resources as the number of IoT devices increases. In addition, operations such as IoT device authentication and authorization, as well as encryption, add computation overheads to the cloud. This means the computation capability must be brought closer to the IoT devices and the resource-demanding tasks must be reserved for the cloud. As such, fog computing emerged to satisfy the demands for frequent computation, communication, and storage by the IoT layer [1,2].
According to the commercial sector, fog computing is a layer of computing that extends the cloud, bringing it closer to the things that generate and process IoT data. Any device with computing, storage, and network connectivity can be a fog node, while fog layer nodes can be deployed anywhere with a network connection (e.g., on top of a traffic light, alongside a railway track [3]). Fog computing involves the following characteristics [1]: • Low latency: The amount of time it takes for data to send from source to destination is known as network latency. In cloud-based IoT environments, latency is typically high due to the distance between the IoT devices and the cloud. This increases the cloud response time, especially as the number of IoT devices increases, making the cloud unable to support the real-time demands of IoT devices. Fog computing decreases the latency by moving data to the edge of the network and nearer to end users to meet the high processing demands; • Higher scalability: The ability of a system to manage an increasing quantity of work by adding resources to the system is known as scalability. In cloud-based IoT environments, as the number of IoT devices increases, it becomes challenging for the cloud to accommodate the heavy computation and bandwidth demands of the devices. When the number of IoT devices grows, fog computing can solve this problem by distributing several fog nodes, which can reduce the heavy computation demands on the cloud and provide hierarchal scalability; • Location awareness: The ability of a device to determine its location, either passively or actively, is referred to as location awareness. This feature is important because it allows applications to provide services better suited to the user and device location, thereby lowering latency; • Mobility: Computing mobility is the ability to perform computing operations while a connected device is able to move, communicating from any location through a wireless channel. This includes the mobility of IoT nodes as well as fog nodes in cloud-based IoT environments; • Decentralized architecture: A decentralized network is a network of interconnected devices in which no single entity is the sole authority. Workloads in distributed architectures are distributed among several machines instead of relying on a single central server. This is an important feature of fog computing because applications and services on the fog can process and store data from any end devices, whether it is a fog node or a sensor (i.e., IoT node); • Heterogeneity: Heterogeneity in networking refers to connecting devices made by different manufacturers running different operating systems using multiple network architectures and protocols. Fog computing heterogeneity is a topological feature that is of particular importance in cloud-based IoT environments, as it enables devices to exchange information, meaning the exchanged information can be used without restrictions; • Bandwidth optimization: Optimization of the network bandwidth refers to the overall inbound and outbound bandwidth improvements in a network. This allows fog nodes to handle the traffic from billions of devices to prevent congestion and latency problems. This is due to the fact that the massive amounts of data collected by IoT devices may be processed locally rather than being sent to the cloud.
Despite the benefits offered by fog-aided IoT, researchers and practitioners are faced with challenges in the implementation and performance of fog-aided IoT systems.
First, there is a lack of real-life implementation of the many theoretical studies, as highlighted in our earlier work in [1]. Although simulation-based experiments provide easy access to practical results regarding the performance of computing systems, observations and research outcomes may not be generalizable to all scenarios due to the variety of IoT platform providers and device manufacturers; their different implementation frameworks, service specifications, and configurations; and differences in the network architectures and protocols. As such, in order to develop a profound and general insight into the tradeoffs involved in a particular system, it is important to use real IoT platforms built on top of real-world networks (i.e., the Internet) when obtaining analytical results regarding the performance of fog-aided IoT implementation. In addition, it would be interesting to explore the performance differences of fog implementation frameworks interacting with different commercial IoT platforms, such as Amazon IoT and Azure IoT.
Second, due to the diversity of fog-aided IoT environments and the lack of consensus among practitioners and hobbyists on a standard fog computing implementation framework [1], there is a lack of resources showing how to implement efficient fog-aided IoT systems. Most of the implementation frameworks available are either domain-specific, complex, or too abstract to be useful in all scenarios.
Third, although fog computing offers promising solutions to many of the performance and security problems of IoT [4], it also involves various security and privacy risks. For instance, while fog computing is crucial for spreading risks across distributed fog nodes, it also has the untoward effect of increasing the attack surface. This is made worse because fog computing devices interact with devices only; that is, the fog nodes receive IoT data from sensors and send the data to the cloud, and vice versa. This means that no humans are involved in the communication. Although this can be considered an advantage because these interacting devices do not have screens or an on-device user interface, which reduces the attack surface, it can lead to failures or targeted attacks that cannot be easily detected and deterred. Other security and privacy issues in fog-aided IoT also deserve our attention. In this work, we aim to better understand fog-aided IoT environments in order to pave the way for further research to address interesting confidentiality, integrity, and availability violations. This paper makes the following contributions:

•
We present two cloud-based IoT environment architectures. The first architecture has a fog layer applied between the IoT devices and the cloud, whereas the second architecture publishes the data directly to the cloud without a fog layer; • We use two benchmarks to measure the performance of the cloud-based IoT architectures. The first benchmark involves Mosquitto message broker metrics, which are used to measure the performance of the IoT-fog-cloud at the fog computing level. The second benchmark involves AWS message broker metrics, which are used to measure the performance of the two architectures (IoT-cloud and IoT-fog-cloud) to show the impact of the additional middle layer (i.e., the fog layer) on cloud-centered IoT environments; • We discuss the security and privacy implications of the two architectures presented in this paper, showing what triggers these implications and suggesting methods to address them; • This study serves as a tutorial reference for fundamental fog computing concepts and aims to walk practitioners through different implementation frameworks for fog-aided IoT and to reveal tradeoffs that inform when to use each implementation based on one's objectives.
The remainder of this paper is structured as follows. In Section 2, we discuss the background in detail. In Section 3, we present the related work and discuss existing industrial and Message Queuing Telemetry Transport (MQTT) benchmarks. In Section 4, we describe the experiment setups of the two architectures in detail. In Section 5, we provide a description of the AWS IoT metrics and Mosquitto message broker metrics that are used to measure our two architectures. In Section 6, we explain the analysis methods used to perform the experiments on both architectures. In Section 7, we describe the results of the experiments performed on both architectures. In Section 8, we provide an evaluation of the experimental results based on the analysis methods. In Section 9, we present the threats to validity. In Section 10, we provide a discussion and outline the limitations of our work. Finally, we conclude the paper by provided future research directions in Section 11.

Background
Today, IoT sensors are used everywhere and have become crucial in many domains of life. As shown in Figure 1, different kinds of sensors can be found in our homes, cars, workplaces, and other areas, which are sold independently (e.g., smoke sensors, light sensors, temperature sensors, motion sensors, proximity sensors, touch sensors, ultrasonic sensors, humidity sensors, IR sensors, pressure sensors, gyroscope sensors) or as integral parts of sophisticated devices, such as smartphones, which may contain dozens of sensors. These sensors are developed by major manufacturers and are deployed in many sectors, including healthcare, education, communication, transportation, and manufacturing. workplaces, and other areas, which are sold independently (e.g., smoke sensors, light sensors, temperature sensors, motion sensors, proximity sensors, touch sensors, ultrasonic sensors, humidity sensors, IR sensors, pressure sensors, gyroscope sensors) or as integral parts of sophisticated devices, such as smartphones, which may contain dozens of sensors. These sensors are developed by major manufacturers and are deployed in many sectors, including healthcare, education, communication, transportation, and manufacturing. Manufacturers have developed IoT platforms to help organizations build fully functional IoT environments. According to AT&T [5], an IoT platform is an end-to-end software framework that pulls together information from sensors, devices, networks, and software, which work together to unlock valuable, actionable data. IoT platforms enable management and automation of connected devices within the IoT universe. There are several proprietary IoT platforms available, including AWS IoT Core [6], Microsoft Azure IoT [7], IBM IoT [8], and Google IoT Core [9], as well as open source IoT platforms such as IoTivity [10], Zetta [11], Arduino IDE [12], DeviceHive [13], and openremote [14]. These IoT platforms usually reside and run on a virtual machine in the cloud that efficiently pulls, processes, and stores the data received from the massive number of IoT sensors.
Classical IoT environments are configured so that IoT sensors are directly connected to the IoT platform and the cloud. In modern architectures, a fog layer is introduced between the IoT sensors and the IoT platform (the cloud) for extra-efficient computation, communication, and storage. Case studies showing the tradeoffs between the two implementation frameworks will be discussed extensively in this paper.
Devices in all layers of IoT environments (see Figure 1) communicate using different protocols. These protocols are listed below [15]: MQTT is a lightweight many-to-many communication protocol for IoT that is designed to be a publish-subscribe messaging transport protocol. MQTT is ideal for Manufacturers have developed IoT platforms to help organizations build fully functional IoT environments. According to AT&T [5], an IoT platform is an end-to-end software framework that pulls together information from sensors, devices, networks, and software, which work together to unlock valuable, actionable data. IoT platforms enable management and automation of connected devices within the IoT universe. There are several proprietary IoT platforms available, including AWS IoT Core [6], Microsoft Azure IoT [7], IBM IoT [8], and Google IoT Core [9], as well as open source IoT platforms such as IoTivity [10], Zetta [11], Arduino IDE [12], DeviceHive [13], and openremote [14]. These IoT platforms usually reside and run on a virtual machine in the cloud that efficiently pulls, processes, and stores the data received from the massive number of IoT sensors.
Classical IoT environments are configured so that IoT sensors are directly connected to the IoT platform and the cloud. In modern architectures, a fog layer is introduced between the IoT sensors and the IoT platform (the cloud) for extra-efficient computation, communication, and storage. Case studies showing the tradeoffs between the two implementation frameworks will be discussed extensively in this paper.
Devices in all layers of IoT environments (see Figure 1) communicate using different protocols. These protocols are listed below [15]: MQTT is a lightweight many-to-many communication protocol for IoT that is designed to be a publish-subscribe messaging transport protocol. MQTT is ideal for connecting remote devices with minimal memory consumption and network bandwidth. MQTT is used in a wide variety of domains, such as in industry, health care, and transportation. Port 1883 is the default MQTT port, whereas port 8883 is the default MQTT port over TLS (i.e., secure-mqtt), both of which are registered with the Internet Assigned Numbers Authority (IANA) for Secure MQTT; • Constrained Application Protocol (CoAp) is a one-to-one User Datagram Protocol (UDP) protocol for transferring state information between the client and server. Despite its ability to preserve resources, CoAP is best-suited to a state transfer model. Since CoAp uses UDP, it does not guarantee the delivery of datagrams. In addition, CoAp is unencrypted. The default CoAP port registered at IANA is 5683; • Extensible Messaging and Presence Protocol (XMPP) is a secure and near-real-time communication protocol for message-oriented middleware based on XML that enables the exchange of structured but extensible data between any two or more devices over a network. XMPP is mainly used by instant messaging applications such as WhatsApp [16] and Telegram [17]. XMPP offers persistent decentralized connection between devices. As such, no central XMPP servers are needed to communicate; however, to establish a connection between two devices, one of the devices is considered an xmpp-client and communicates over port 5222, while the other is considered an xmpp-server and uses port 5269. XMPP can also use port 5280 for two-way communication. This is called xmpp-bosh, meaning Bidirectional Streams Over Synchronous HTTP (BOSH). XMPP has been used in the literature to network IoT devices, such as in [18,19]. Despite its features and potential, XMPP has some limitations. First, XMPP does not have a Quality of Service (QoS) mechanism. In addition, XMPP streams data in XML format, which introduces overhead due to the text-based communication.
These reasons, among others, make MQTT a more popular protocol for IoT, since it has a mechanism for QoS and uses lightweight binary-based communication; • MQTT For Sensor Networks (MQTT-SN) [20] is considered a modified version of MQTT that is adapted to the attributes of a wireless connection, such as a lossy wireless network. It is designed specifically for wireless sensor networks with scale in mind. MQTT-SN was developed to support non-TCP networks such as UDP. This is another advantage because it makes the communication lighter by eliminating the TCP handshakes; • The Advanced Message Queuing Protocol [21] (AMQP) is an open standard application layer protocol for middleware. AMQP is designed with more advanced features that introduce more overhead than when using MQTT. These features include message orientation, queuing, routing, reliability, and security. The registered port number for AMQP at IANA is 5672, while for AMQPS (i.e., TLS/SSL encrypted AMQP) it is 5671. For more information on which protocols function best for IoT networks based on the messaging requirements, see [21].
This study focuses on commercial IoT sensors (i.e., DHT11 temperature and humidity sensors) connected using a proprietary IoT platform (i.e., AWS IoT Core). We chose proprietary IoT platforms over the free open source ones because they are popular among industry users due to their faster time to market and lower initial cost.
Amazon Web Services (AWS) provides reliable, scalable, and inexpensive on-demand cloud computing services to individuals, companies, and governments around the world [22]. Customers can benefit from the cloud datacenters distributed in the different locations in many ways, including the low cost (pay as you go) and the massive cloud infrastructure, in order to perform experiments and deploy new applications. AWS offers many services. In this study, we used (1) AWS IoT, (2) Amazon S3, and (3) AWS CloudWatch. AWS IoT allows secure communication and messaging exchange over MQTT for Internet-connected devices such as sensors and micro-controllers in real time [23]. Amazon CloudWatch [24] monitors devices and applications connected to AWS in real time using several metrics [23] to track the connected devices and measure the performance, security, and scalability, among other criteria.
The MQTT protocol is a widely used protocol that is supported by the most platforms and commercial sensors; therefore, we used MQTT as the communication protocol. In addition, we used a Raspberry Pi board to simulate fog nodes for the scenarios in which fog nodes are introduced. To enable the Raspberry Pi boards used in this experiment to communicate over MQTT, we used an MQTT broker software, Eclipse Mosquitto [25]. An MQTT broker is a server that receives all the messages from the IoT devices and publishes them to other devices. MQTT broker also has other benefits, including (1) supporting scalability with many IoT devices, (2) managing credentials and certificates that are used for authentication, (3) decreasing network strain on the cellular network without decreasing security, and (4) excluding the connection of insecure and vulnerable devices. Many MQTT brokers are available, including Eclipse Mosquitto [26], RabbitMQ [26], and ActiveMQ [27]. In this paper, we used Eclipse Mosquitto because it is the most popular and has a lot of resources for implementation. In addition, it is lightweight and suitable for use on all devices, from low-power, single-board computers to full servers. An MQTT broker feature called SYS-Topics [28] is widely used to monitor the Mosquitto MQTT broker by providing metrics about Mosquitto, as well as to track the devices connected to the broker.
In this paper, we present numerical results based on an experiment that uses a realworld IoT platform, sensors, and network (not a simulation) to show the performance tradeoffs of various IoT implementation frameworks and discuss the results. Notably, this experiment in a real environment is vulnerable to real-life cyber or physical attacks, as well as performance failures.

Related Works
The rapid adoption of cloud-based IoT environments in large scale and with intensive use has induced, among other factors, a growing need to simulate real-life environments to measure the security and performance of the IoT-fog-cloud environments to provide suitable support for the construction of efficient access control models [1]. Benchmarks are one of the ways to measure the security and performance of cloud-based IoT environments. There are several widely used general industry benchmarks that are adopted in many commercial solutions. The Standard Performance Evaluation Corporation (SPEC) provides benchmarks for a wide range of IT components, such as cloud, CPU, storage, power, and virtualization components [29]. The Transaction Processing Performance Council (TPC) also offers a suite of widely used IT industry benchmarks [30]. TPCx-IoT is one of the TPC benchmarks that measures the operating, data storage, and management systems to provide the industry with performance metrics and other available metrics for IoT systems [31,32]. Moreover, HP developed IoTABench, an IoT analytics benchmark for big data scenarios that is used to evaluate the performance and scalability of big data platforms [33]. The benchmark was demonstrated using a smart metering IoT use case and evaluated on the HP Vertica 7 analytics platform, which can handle data for an "electric utility with 40 million smart meters".
For MQTT, different benchmarks to measure the performance and security of cloudbased IoT environments have been proposed in the literature [34,35]. Two IoT platforms, Things Board and Site Where, have been evaluated using different metrics [36]. In addition, an evaluation of the message transmission processes (i.e., Subscribe and Publish) of the MQTT protocol via wireless and wired clients was presented in [37]. The end-to-end delay and message loss when transmitting messages were analyzed with different quality of service levels and different payload sizes. The results of their experiment showed that end-to-end delay is related to the message loss with different sizes of payloads.
Moreover, Aazam et al., in a recent survey, evaluated the performance of fog computing using performance metrics such as processing delay, processing costs, and processing power, and derived the performance gains obtained in comparison to a cloud computingonly approach [38]. In the healthcare sector, Alsubaei et al. evaluated security in the Internet of Medical Things (IoMT) [39]. In addition, Kafhali et al. evaluated the response times for accessing medical data stored in a fog-based IoMT implementation framework [40]. They also proposed a queuing model to predict the minimum number of computing resources (both fog and cloud nodes) required to meet the service level agreement (SLA) for response time. Another study in the healthcare field was presented by Vilela et al., who compared the performance of fog-based computing to the conventional cloud computing model in a realtime healthcare monitoring system [41]. Edge Bench is another benchmark for serverless edge computing platforms that is used to measure the performance of two edge computing platforms, Greengrass and Azure [42]. In addition, DeFog, a fog computing benchmark, was proposed to provide a standard methodology and facilitate the understanding of the target platform by collecting a catalogue of relevant metrics for a set of benchmarks [43]; however, most experiments in these studies were carried out using simulators that rely on generic metrics or that focus on one domain, which do not represent real IoT-fog-cloud systems across different domains.
In this study, we extend the previous studies by implementing real-life experiments and analyzing performance metrics from a popular cloud provider (AWS) and IoT protocol (MQTT). We implement two real-life cloud-based IoT environments to measure their performance. In addition, we use different numbers of IoT devices to increase the numbers of subscribers and publishers in order to understand how these factors impact the results.

Experiment Setup
This section presents the hardware and software components used to set up our experiment. In this paper, we present several IoT-cloud implementation frameworks, namely (1) IoT-cloud, (2) IoT-fog-cloud using a bridge, and (3) IoT-fog-cloud using Python. The hardware and software configurations used in these implementation frameworks are discussed in the following subsections.

Hardware
As shown in Figure 2, the experiment discussed in this section involved two architectures. The first architecture is IoT-fog-cloud, while the second is IoT-cloud. In this section, we first describe the devices that were used in the two cloud-based IoT environments, then explain the two architectures in detail. In this experiment, we used DHT11 devices [44] and a Raspberry 3 Pi model B [45]. The DHT11 device is a low-cost sensor that is used to measure the temperature and humidity of the surrounding air. The purpose of the DHT11 sensor in this experiment was to generate real data for the experiment. The Raspberry Pi is a low-cost, single-board computer with built in WiFi and processing capability that is used across several domains, such as in weather monitoring, smart homes, and smart health care. In this experiment, the purpose of the Raspberry Pi was to provide light computation capability to the DHT11 sensor data. In addition, it provided light storage for the DHT11 configurations. Moreover, the Raspberry Pi can be easily moved to different locations. A complete list of the hardware used in this experiment is available in Table 1.   In the first architecture, each DHT11 sensor is connected to only one Raspberry Pi. The Raspberry Pi is used here to enable WiFi connectivity, since the DHT11 sensors are not equipped with network interfaces. Each sensor is connected to Raspberry Pi board (i.e., connectivity enabler), which is considered an IoT device in the IoT layer. Each IoT device is connected via WiFi to another Raspberry Pi board that acts as a fog node in the fog layer. Communication between the IoT devices, the fog node, and the cloud uses the

First Architecture
In the first architecture, each DHT11 sensor is connected to only one Raspberry Pi. The Raspberry Pi is used here to enable WiFi connectivity, since the DHT11 sensors are not equipped with network interfaces. Each sensor is connected to Raspberry Pi board (i.e., connectivity enabler), which is considered an IoT device in the IoT layer. Each IoT device is connected via WiFi to another Raspberry Pi board that acts as a fog node in the fog layer. Communication between the IoT devices, the fog node, and the cloud uses the MQTT protocol. For this, an MQTT broker called Eclipse Mosquitto [25] is installed in the Raspberry Pi, acting as a fog device. The Mosquitto MQTT broker exchanges all messages using the subscribe-publish model presented in [1]. The Mosquitto MQTT broker is also used to filter all messages based on topics. A topic refers to an UTF-8 string that the broker (i.e., Mosquitto) uses to filter messages for each connected IoT device. Each data type (i.e., humidity, temperature) in our experiment is considered a separate topic. The data generated by the sensors and collected by the three IoT devices are transmitted over the Internet to the Raspberry Pi acting as the fog node. This Raspberry Pi, which contains the MQTT broker Mosquitto on it, is then connected over the Internet to the AWS cloud. The communication between the three layers occurs through the Internet. Figure 3 shows the hardware used to implement the first architecture.
(i.e., Mosquitto) uses to filter messages for each connected IoT device. Each data type (i.e., humidity, temperature) in our experiment is considered a separate topic. The data generated by the sensors and collected by the three IoT devices are transmitted over the Internet to the Raspberry Pi acting as the fog node. This Raspberry Pi, which contains the MQTT broker Mosquitto on it, is then connected over the Internet to the AWS cloud. The communication between the three layers occurs through the Internet. Figure 3 shows the hardware used to implement the first architecture.

Second Architecture
In the second architecture, all three IoT devices, consisting of sensors and Raspberry Pi boards (i.e., connectivity enablers), are directly connected to the AWS cloud; therefore, the real data received by device one, device two, or device three are forwarded wirelessly to the AWS cloud layer. The hardware used in the second architecture is identical to that used in Figure 3, except that no fog node is used in this architecture.

Software
We installed Arduino IDE [46] software on top of the three Raspberry Pi boards used to connect the IoT devices. Arduino IDE is a cross-platform application that is written in functions from C and C++ and is used to write and upload programs to Arduinocompatible boards such as Raspberry Pi. We used Arduino IDE to read the data collected by the sensors (i.e., temperature and humidity) and then publish it over the Internet to the fog node. We used C++ scripts in Arduino IDE to perform certain operations on the IoT devices, namely tagging, authentication, publishing, and subscription.

First Architecture
In the first architecture, Algorithm 1 gathers data from the IoT devices and forwards it to the fog. This involves authenticating the IoT devices to communicate with the MQTT broker on the fog device and publish data to it. The following two sections present two ways for fog nodes to communicate with the cloud.

Second Architecture
In the second architecture, all three IoT devices, consisting of sensors and Raspberry Pi boards (i.e., connectivity enablers), are directly connected to the AWS cloud; therefore, the real data received by device one, device two, or device three are forwarded wirelessly to the AWS cloud layer. The hardware used in the second architecture is identical to that used in Figure 3, except that no fog node is used in this architecture.

Software
We installed Arduino IDE [46] software on top of the three Raspberry Pi boards used to connect the IoT devices. Arduino IDE is a cross-platform application that is written in functions from C and C++ and is used to write and upload programs to Arduinocompatible boards such as Raspberry Pi. We used Arduino IDE to read the data collected by the sensors (i.e., temperature and humidity) and then publish it over the Internet to the fog node. We used C++ scripts in Arduino IDE to perform certain operations on the IoT devices, namely tagging, authentication, publishing, and subscription.

First Architecture
In the first architecture, Algorithm 1 gathers data from the IoT devices and forwards it to the fog. This involves authenticating the IoT devices to communicate with the MQTT broker on the fog device and publish data to it. The following two sections present two ways for fog nodes to communicate with the cloud.

• Bridging
In our fog-aided IoT implementation, the MQTT broker (i.e., Mosquitto) is installed on top of a Raspberry Pi board that serves as a fog node. In such cases, the MQTT broker needs to be very close to where the sensors are deployed. The Mosquitto MQTT broker has a built-in capability that allows the received data to be sent directly to the cloud (AWS IoT Core) by specifying the address of the AWS IoT core service used. This operation is called bridging (please see [47] for more information). Algorithm 2 illustrates the connection of IoT devices to the AWS IoT core using a bridge connection. Algorithm 1. Gather data generated from the IoT device and forward it to the fog node. First architecture/*. This algorithm authenticates the IoT device to the fog device, generates temperature and humidity data via DHT11 sensors, and publishes them to the Mosquitto MQTT broker on the fog device 1: Define the Type of DHT Sensor, which is DHT11 2: Define the Input/Output Pins of the Raspberry Pi to which the DHT11 is Connect to Internet 12: If (the Connection is Established) then 13: Print "Connected" 14: Connect Publish Temperature Topic with Its Read Value to MQTT Broker on Fog Device 27: End while Algorithm 2. Bridge every message received from the IoT devices based on the topics of the messages to the AWS broker on the cloud/*. This algorithm authenticates the IoT device to the MQTT broker on the fog device, bridges to the data received from the sensors via the MQTT broker, filters them based on topics, authenticates to the AWS via certificates, and then publishes the filtered data to the AWS IoT core service Using the same Mosquitto MQTT broker, we developed a Python script on the fog device to replace the built-in Mosquitto bridging capability to simultaneously authenticate the MQTT broker and AWS, receive the data from the sensors, and filter and publish the data received to the AWS IoT core service. This Python script provides more flexibility for future improvements in security and performance. The following algorithm depicts the operations implemented in our Python script. Algorithm 3 is used to authenticate IoT devices to fog devices, gather the temperature and humidity data from the DHT11 sensors, and publish the data to the Mosquitto MQTT broker on the fog device. The algorithm also authenticates the fog node to interact with AWS IoT Core (i.e., the cloud). Afterward, the algorithm filters the collected data and publishes it to the cloud. Define a Connection Function 11: Subscribe for All Topics in each IoT Device 12: Print "Connected" when the Connection is Established 13: Print "Error" when the Connection is Disconnected 14: Define a Message Function 15: Get Topic of The

Second Architecture
In the second architecture, Arduino IDE is installed on top of the three Raspberry Pi boards used to connect the IoT devices. Arduino IDE is used to read the data collected by the sensors (i.e., temperature and humidity) and then publish it over the Internet directly to the cloud. This operation requires authentication to the cloud before the data can be published. The process used to authenticate the IoT device (i.e., DHT11 + Raspberry Pi) to access the cloud (AWS IoT Core) using the AWS certificate and then to publish the data generated by the sensors to the cloud is illustrated in Algorithm 4.
To monitor the Mosquitto MQTT broker, Algorithm 5 illustrates how $SYS-Topics is used to provide metrics. Algorithm 4. Gather data generated from the IoT device and forward it to the cloud-Second architecture/*. This algorithm authenticates the IoT device to the AWS cloud via certificates, generates temperature and humidity data via the DHT11 sensors, and publishes them to the AWS cloud

Descriptions of Metrics
Many metrics can be used to measure the performance in IoT systems. The following subsections describe the metrics that we utilized to measure the performance in the cloud and fog layers.

Cloud Layer: AWS IoT Metrics
Many metrics can be used to measure the performance of cloud-based IoT systems from the cloud layer. Since we used the AWS IoT as the cloud service provider for our experiments, we used Amazon CloudWatch to measure the performance. Amazon CloudWatch has been used in the literature to monitor performance [48][49][50]. Amazon CloudWatch processes and analyzes data in real time and provides the following metrics to measure our two cloud-based IoT environments:

•
Connect Success: This metric is used to collect the number of successful connections from our IoT nodes or fog nodes to the AWS message broker.

•
Ping Success: This metric is used to collect the number of ping messages received by the AWS message broker. These ping messages are received from the fog node(s) in the first architecture and from the IoT node(s) in the second architecture.

•
PublishIn Success: This metric is used to collect the number of publish requests successfully processed by the AWS message broker. As with the ping messages, these messages are received from the fog node(s) in the first architecture and from the IoT node(s) in the second architecture. • PublishOut Success: This metric is used to collect the number of publish requests successfully made by the AWS message broker to the fog nodes in the first architecture and to the IoT nodes in the second architecture. • Subscribe Success: This metric is used to collect the number of successful subscribe requests processed by the AWS message broker. These requests are made by the fog node in the first architecture and made directly by the IoT devices in the second architecture.

•
PublishIn Clienterror: This metric is used to collect the number of publish requests rejected because they did not meet the AWS IoT requirements. • Unsubscribe Success: This metric is used to collect the number of unsubscribe requests that were successfully processed by the AWS message broker. These unsubscribe requests are made by the fog node in the first architecture and made directly by the IoT devices in the second architecture. • Throttle Exceeded: This metric is used to collect the number of requests that have been throttled because the client (i.e., the IoT node or fog node) has sent too many messages and exceeded the allowed message rate. • PublishOut Throttle: This metric is used to collect the number of publish requests that have been throttled because the client (i.e., IoT node or fog node) has exceeded the allowed message rate.

Fog Layer: Eclipse Mosquitto Broker Metrics
Many metrics can be used to measure the performance of the fog layer. MQTT has been used in the literature as a lightweight protocol to communicate between messages [34,35,51]. Since we used the MQTT protocol for the message broker on the fog node, we utilized Eclipse Mosquitto [25] in our experiments. Eclipse Mosquitto [25] provides some metrics as $SYS topics, which are described below:

Analysis Methods
We used two benchmark metrics to analyze the performance of the two IoT architectures implemented in this paper. In the two cloud-based IoT architectures, we set the number of subscribers and publishers to two for each device. This is because the IoT devices (i.e., the DHT11 sensors) generate two types of data, namely (1) temperature and (2) humidity data; therefore, as the number of sensor devices increase, the number of subscribers and publishers should also increase. This provides more accurate, consistent, and real results about the environment performance and scalability. In this section, we present the methods that we used to perform the experiments on both architectures.

Architecture 1 vs. Architecture 2
The performance of the two cloud-based IoT architectures was analyzed using the AWS benchmark and the obtained results were compared. The experiment was performed using a different number of IoT devices each time (1, 2, or 3 IoT devices) to compare and analyze the results in order to show the impacts of the fog layer in the first architecture on IoT environments. Figure 2 shows the experimental setup and metrics applied to measure the performance of the two cloud-based IoT environments. AWS, as with any IoT platform provider, requires that any device be authenticated before communicating with it. AWS uses certificates to authenticate devices. As shown in Figure 2, the location where the certificate is stored is different in the two architectures due to the structure of the environments. Since the first architecture has a fog layer between AWS and the IoT devices, the certificate is stored in the fog device (i.e., the Raspberry Pi board serving as the fog layer). In the second architecture, however, since the IoT devices are directly connected to AWS, the certificates are stored in each IoT device.

Architecture 1 Implementation: Python Script vs. Bridging
The first architecture of the IoT-fog-cloud layer was implemented using two different experiments, as shown in Figure 4. The first experiment was based on a Python script shown in Algorithm 3 that manually receives, filters, and forwards the messages to the cloud. The second experiment (Algorithm 2) bridges all of the messages received from the IoT devices based on their topics to AWS. The purpose of using the bridge in the first architecture is to connect two brokers, the MQTT message broker "mosquitto" and the AWS message broker, to exchange messages based on the different topics and to validate the results of the first architecture. AWS benchmarks were used to analyze the performance of the first architecture in two different implementation frameworks. Although bridging allows faster implementation, the Python script provides more flexibility to add features to optimize performance. The objective of this analysis is to show the impacts of both implementation frameworks. Sensors 2021, 21, x FOR PEER REVIEW 17 of 33

Architecture 1 Measurement: Mosquitto Metrics vs. AWS Metrics
The performance of the first architecture (i.e., IoT-fog-cloud) was measured from both the fog side and the cloud side. The fog-side metrics (i.e., Mosquitto broker metrics) were measured over different durations (30 s, 1 min, 5 min, 15 min, 30 min, 45 min, and 1 h), while the cloud-side metrics (i.e., AWS metrics) were measured over 30 s, 1 min, 5 min, 15 min, and 1 h (due to the AWS platform constraints, it was difficult to unify the experiment durations). Figure 5 presents the different metrics used to measure the performance of the IoT environment that uses a fog layer from two sides. Figure 5a presents the metrics applied to measure the performance of the IoT environment at the fog layer, whereas Figure 5b presents the metrics applied to measure the performance of the IoT environment at the cloud layer. This was done to ensure that the performance of the IoT environment remains consistent in the fog and cloud layers using different setups.  The performance of the first architecture (i.e., IoT-fog-cloud) was measured from both the fog side and the cloud side. The fog-side metrics (i.e., Mosquitto broker metrics) were measured over different durations (30 s, 1 min, 5 min, 15 min, 30 min, 45 min, and 1 h), while the cloud-side metrics (i.e., AWS metrics) were measured over 30 s, 1 min, 5 min, 15 min, and 1 h (due to the AWS platform constraints, it was difficult to unify the experiment durations). Figure 5 presents the different metrics used to measure the performance of the IoT environment that uses a fog layer from two sides. Figure 5a presents the metrics applied to measure the performance of the IoT environment at the fog layer, whereas Figure 5b presents the metrics applied to measure the performance of the IoT environment at the cloud layer. This was done to ensure that the performance of the IoT environment remains consistent in the fog and cloud layers using different setups. Sensors 2021, 21, x FOR PEER REVIEW 18 of 33

Results and Description of Experiments
In this section, the results of the two architectures of cloud-based IoT environments will be analyzed using the two benchmarks metrics (Mosquitto and AWS) based on the selected architecture using one, two, or three IoT devices with increasing numbers of subscribing and publishing requests. In the following sections, the benchmark metrics of each architecture are described in detail, along with observations.

Descriptions of the Three Experiments on the First Architecture with One, Two, or Three IoT Devices Using AWS Benchmark Metrics (Cloud Layer)
The Amazon CloudWatch monitor has a variety of metrics that can be used to analyze the cloud layer, which were used in the experiments on the first architecture, as shown in Tables 2 and 3 Tables 2 and 3. Before analyzing the results obtained using the AWS metrics, a number of assumptions should be taken into consideration. First, the number of successful connections from either IoT devices or fog devices using the Connect Success metric must be equal to the number of subscribe requests received from either fog devices or IoT devices using the Subscribe Success metric because the loss of connections will lead to the loss of subscribe requests in each device and the subscriptions will be renewed automatically once the connection is reestablished. Second, the number of unsubscribe requests received from either IoT devices or fog devices using the Unsubscribe Success metric must reflect the numbers of subscription and publishing requests generated in each IoT device. In our experiments, there were three IoT devices, each of which generated two types of data, namely temperature and humidity. Asusch, the number of IoT devices was three and the number of all subscription and publishing requests was six. Third, the number of publishing requests received from either fog devices or IoT devices using the Publishin Success metric should be close to or the same

Results and Description of Experiments
In this section, the results of the two architectures of cloud-based IoT environments will be analyzed using the two benchmarks metrics (Mosquitto and AWS) based on the selected architecture using one, two, or three IoT devices with increasing numbers of subscribing and publishing requests. In the following sections, the benchmark metrics of each architecture are described in detail, along with observations.

Descriptions of the Three Experiments on the First Architecture with One, Two, or Three IoT Devices Using AWS Benchmark Metrics (Cloud Layer)
The Amazon CloudWatch monitor has a variety of metrics that can be used to analyze the cloud layer, which were used in the experiments on the first architecture, as shown in Tables 2 and 3 Tables 2 and 3. Before analyzing the results obtained using the AWS metrics, a number of assumptions should be taken into consideration. First, the number of successful connections from either IoT devices or fog devices using the Connect Success metric must be equal to the number of subscribe requests received from either fog devices or IoT devices using the Subscribe Success metric because the loss of connections will lead to the loss of subscribe requests in each device and the subscriptions will be renewed automatically once the connection is reestablished. Second, the number of unsubscribe requests received from either IoT devices or fog devices using the Unsubscribe Success metric must reflect the numbers of subscription and publishing requests generated in each IoT device. In our experiments, there were three IoT devices, each of which generated two types of data, namely temperature and humidity. Asusch, the number of IoT devices was three and the number of all subscription and publishing requests was six. Third, the number of publishing requests received from either fog devices or IoT devices using the Publishin Success metric should be close to or the same as the number of publish requests made by AWS to either fog devices or IoT devices using the Publishout Success metric.
The experiments using the first architecture, which included a fog layer, were conducted with one, two, or three IoT devices for different time periods (30 s, 1 min, 5 min, 15 min, and 1 h), as shown in Table 2. The first experiment involved one IoT device attached to one fog device, with the results showing that the Connect Success and Subscribe Success numbers reflected the numbers of subscribing and publishing requests for one IoT device, with two subscribe and publish requests. This was because the connection was not disconnected and the subscribe request was not lost from either the IoT or the fog device. Moreover, there was only one IoT device connected to only one fog device, which was then authenticated to the AWS cloud, and no other devices interrupted them. In addition, the PublishIn Success and PublishOut Success numbers were expected and stable because the data came from the IoT device and were filtered based on the fog device, then sent to the AWS cloud. It takes time for the data to be transferred between the three layers of IoT, fog, and cloud. In the second experiment, on the other hand, two IoT devices were attached to the fog device, as shown in Table 2. The results indicated that the Connect Success and Subscribe Success numbers were equal to the numbers of subscribing and publishing requests for both IoT devices, since two IoT devices were linked to one fog device with four subscribing and publishing requests; therefore, the PublishIn Success and PublishOut Success numbers were significantly increased with two IoT devices compared to the previous experiment using only one IoT device attached to one fog device. This was because two IoT devices were connected to only one fog device serving as the middle layer between the IoT devices and the AWS cloud, which was used to authenticate to the AWS cloud; thus, the fog device increased the number of messages published to the AWS cloud and reduced the number of IoT devices that needed to be authenticated to the AWS cloud. The third experiment used three IoT devices connected to one fog device, as shown in Table 2. The Connect Success and Subscribe Success numbers still reflected the actual numbers of subscribing and publishing requests for all three IoT devices, even though there was only one fog device; thus, the PublishIn Success and PublishOut Success numbers were significantly increased due to the high volumes of published messages from the three IoT devices. As such, the fog device made a significant impact in filtering and transferring the actual volumes of data from the three IoT devices to the AWS cloud and decreasing the number of authentications required for the AWS cloud.  In Table 2, it can be seen that for one IoT device, the Connect Success and Subscribe Success numbers are low because there is only one IoT device with two subscribe and publish requests authenticated to the AWS and with no other IoT devices attached with it; thus, the connection and subscription are established at the same time. In addition, the Pub-lishIn Success and PublishOut Success numbers are consistent and high due to the single authenticated IoT device. With two IoT devices with four subscribe and publish requests, however, the Connect Success and Subscribe Success numbers increase because there are two IoT devices attempting to authenticate to the AWS cloud at one time, which increases the number of connections and subscriptions. The PublishIn Success and PublishOut Success numbers are not substantially change compared to the previous experiment even though the numbers of subscribe and publish requests increase when the second IoT device is added because the second IoT device tries to authenticate to the AWS cloud while the first IoT device tries to publish messages to the AWS cloud. As a result, publish messages are lost in each period. The third experiment, using three IoT devices with six subscribe and publish requests, shows that the number of connections and subscriptions increases while the PublishIn Success and PublishOut Success numbers are low. This is because the three IoT devices try to authenticate to the AWS cloud at once, causing a huge number of connection and subscription requests.

Descriptions of the Three Experiments on the First Architecture with One, Two, or Three IoT Devices Using Mosquitto Benchmark Metrics (Fog Layer)
The MQTT Mosquitto broker using SYS-Topics has several metrics that were used in the experiments on the first architecture from the fog layer, as shown in Tables 4-6. The Mosquitto broker metrics are (1) load/messages/received, (2) load/messages/sent, (3) load/publish/received, (4) load/publish/sent, (5) load/bytes/received, (6) load/bytes/ sent, (7) load/sockets, (8) load/connections, (9) messages/stored, (10) store/messages/ bytes, (11) subscriptions/count, (12) heap/current, (13) messages/received, (14) messages/sent, (15) publish/messages/received, (16) publish/messages/sent, (17) bytes/ received, (18) bytes/sent, (19) publish/bytes/received, and (20) publish/bytes/sent. These metrics are described in detail in Section 5.2 and were obtained over one hour, starting from 30 s, as shown in Tables 4-6. Before analyzing the Mosquitto broker metrics on the fog device, a number of hypotheses should be discussed. First, the number of subscribe requests received from each IoT device using the subscription/count metric must reflect the number of subscribe and publish requests established in each IoT device. Second, the number of publish messages received from each IoT device in the first architecture using the publish/messages/received metric should be equal to the number of publish messages received from the fog device using the Publishin Success metric. Third, the number of publish messages sent by the Mosquitto broker on the fog device using the publish/messages/sent metric should increase as the number of IoT devices increases due to the broker capabilities and the high computational power of the fog device to process and publish many messages. The experiments on the first architecture, which had a fog device between the IoT device(s) and the AWS cloud, were performed using one, two, or three IoT devices for different periods of 30 s, 1 min, 5 min, 15 min, 30 min, 45 min, and 1 h. The first experiment involved one IoT device attached to the fog device connected to the AWS cloud, as shown in Table 4. The results showed that the number of subscribe requests using the subscriptions/count metric on the fog device reflected the number of subscribe and publish requests of the single IoT device with two subscribe and publish requests. This was because the fog device subscribes to the topics of each message received from IoT devices, filters them based on the topics of the messages, then publishes them to the AWS cloud. As such, each message is identified by its topic in each layer. In addition, the number of publish messages made by the fog device (publish/messages/sent) is high, even though there is only one IoT device, with two subscribe and publish requests. This is because the fog device has high capability to filter and publish many messages, since it is located close to the IoT device that is generating the real data. Moreover, the size of the heap memory used by Mosquitto on the fog device is stable over different durations from 30 s to 1 h. The second experiment was conducted using two IoT devices with four subscribe and publish requests attached to the fog device, which connects to the AWS cloud, as shown in Table 5. The results showed that the number of subscribe requests increased since the number of IoT devices increased to two. In addition, the number of publish messages made by the fog device (publish/messages/sent) significantly increased because two IoT devices with four subscribe and publish requested were attached. The third experiment was conducted using three IoT devices with six subscribe and publish requests attached to a fog device and then to the AWS cloud, as shown in Table 6. The results demonstrated that the number of subscribe requests was six when the number of IoT devices increased to three. In addition, the number of publish messages made by the fog device (publish/messages/sent) was significantly higher when the number of IoT devices increased to three.

First Architecture vs. Second Architecture
The first experiment on each of the two architectures was performed by connecting one IoT device to the cloud. We used two subscribe and two publish commands to send the data. The results showed that the numbers of subscribe and publish commands were the same for both architectures and matched the defined numbers of subscribe and publish commands for one IoT device; however, the number of published messages (PublishIn Success and PublishOut Success, shown in Figures 6a and 7a) in the first architecture was slightly lower than the number of published messages in the second architecture. This was because there was an additional hop (i.e., fog layer) in the middle of the first architecture that processed the messages before transmitting them to the AWS cloud. As a result, the messages took longer to be delivered to the AWS cloud. In contrast, the second architecture did not have a fog layer and the messages were forwarded directly to the AWS cloud, so the number of published messages was slightly higher. Moreover, the IoT devices must be authenticated to the AWS before publishing the messages. The Connect Success and Subscribe Success metrics shown in Figures 8a and 9a show that when using one IoT device, authentication did not impact these two metrics in either architecture.

First Architecture vs. Second Architecture
The first experiment on each of the two architectures was performed by connecting one IoT device to the cloud. We used two subscribe and two publish commands to send the data. The results showed that the numbers of subscribe and publish commands were the same for both architectures and matched the defined numbers of subscribe and publish commands for one IoT device; however, the number of published messages (PublishIn Success and PublishOut Success, shown in Figures 6a and 7a) in the first architecture was slightly lower than the number of published messages in the second architecture. This was because there was an additional hop (i.e., fog layer) in the middle of the first architecture that processed the messages before transmitting them to the AWS cloud. As a result, the messages took longer to be delivered to the AWS cloud. In contrast, the second architecture did not have a fog layer and the messages were forwarded directly to the AWS cloud, so the number of published messages was slightly higher. Moreover, the IoT devices must be authenticated to the AWS before publishing the messages. The Connect Success and Subscribe Success metrics shown in Figures 8a and 9a show that when using one IoT device, authentication did not impact these two metrics in either architecture.      The second experiment was conducted using two IoT devices and four subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands for two IoT devices only in the first architecture, as shown in Figures 8b and 9b (Connect Success and Subscribe Success); the number of subscribe and publish commands was significantly increased in the second architecture. This is because in the second architecture, the two IoT devices need to be authenticated separately to the AWS cloud, whereas in the first architecture, only one device (the fog node) needs to be authenticated because the IoT devices are authenticated to the fog node in a different, simpler process. Additionally, this high number of connect and subscribe requests in the second architecture causes a loss in the number of messages published to the AWS cloud. This is because while one of the IoT devices is subscribed and publishing, the other device remains trying to connect, as shown in Figures 6b and 7b (PublishIn Success and PublishOut Success). Notably, the rate of published messages in the first architecture is much better than in the second architecture. This is because in the first architecture, there was no failure to subscribe and the fog node was always able to publish messages successfully to the AWS cloud, as shown in Figures  8b and 9b (Connect Success and Subscribe Success).
In the third experiment, we used three IoT devices with six subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands (PublishIn Success and PublishOut Success in Figures 6c and 7c) in the first architecture. In the second  The second experiment was conducted using two IoT devices and four subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands for two IoT devices only in the first architecture, as shown in Figures 8b and 9b (Connect Success and Subscribe Success); the number of subscribe and publish commands was significantly increased in the second architecture. This is because in the second architecture, the two IoT devices need to be authenticated separately to the AWS cloud, whereas in the first architecture, only one device (the fog node) needs to be authenticated because the IoT devices are authenticated to the fog node in a different, simpler process. Additionally, this high number of connect and subscribe requests in the second architecture causes a loss in the number of messages published to the AWS cloud. This is because while one of the IoT devices is subscribed and publishing, the other device remains trying to connect, as shown in Figures 6b and 7b (PublishIn Success and PublishOut Success). Notably, the rate of published messages in the first architecture is much better than in the second architecture. This is because in the first architecture, there was no failure to subscribe and the fog node was always able to publish messages successfully to the AWS cloud, as shown in Figures  8b and 9b (Connect Success and Subscribe Success).
In the third experiment, we used three IoT devices with six subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands (PublishIn Success and The second experiment was conducted using two IoT devices and four subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands for two IoT devices only in the first architecture, as shown in Figures 8b and 9b (Connect Success and Subscribe Success); the number of subscribe and publish commands was significantly increased in the second architecture. This is because in the second architecture, the two IoT devices need to be authenticated separately to the AWS cloud, whereas in the first architecture, only one device (the fog node) needs to be authenticated because the IoT devices are authenticated to the fog node in a different, simpler process. Additionally, this high number of connect and subscribe requests in the second architecture causes a loss in the number of messages published to the AWS cloud. This is because while one of the IoT devices is subscribed and publishing, the other device remains trying to connect, as shown in Figures 6b and 7b (PublishIn Success and PublishOut Success). Notably, the rate of published messages in the first architecture is much better than in the second architecture. This is because in the first architecture, there was no failure to subscribe and the fog node was always able to publish messages successfully to the AWS cloud, as shown in Figures 8b and 9b (Connect Success and Subscribe Success).
In the third experiment, we used three IoT devices with six subscribe and publish commands. The results show that the number of subscribe and publish commands matches the defined number of subscribe and publish commands (PublishIn Success and PublishOut Success in Figures 6c and 7c) in the first architecture. In the second architecture, however, the number of subscribe and publish commands is significantly higher. This is because the fog node in the first architecture authenticates three IoT devices and the AWS cloud authenticates only the fog node. In contrast, in the second architecture, the AWS cloud authenticates three IoT devices separately, which increases the number of subscribe and publish commands. This also negatively affects the number of published messages in the second architecture due to the time spent by the IoT devices that are not connected trying to connect, as shown in Figures 6c and 7c  In this experiment, we found that the performance of the second architecture was better than that of the first architecture when using one IoT device. However, when using more than one IoT device, the first architecture outperforms the second architecture in terms of performance. Moreover, when using more than one IoT device, the resource utilization in the first architecture was better than in the second architecture because all of the IoT devices were able to successfully connect to the fog node simultaneously. Overall, with an increased number of IoT devices, the first architecture outperforms the second architecture.

Architecture 1 Implementation: Python Script vs. Bridging
In this experiment, we evaluated the first architecture (i.e., IoT-fog-cloud) using two different implementation frameworks (Python Script vs. bridging) using AWS metrics. Three experiments were performed using one, two, or three IoT devices. Using two subscribes and two publishes per device, both implementation frameworks showed insignificant differences in performance using the AWS benchmark, as shown in Figures  10-13. Comparing the performance results of the same architecture using two different implementation frameworks shows that the results of our experiment are accurate. In this experiment, we found that the performance of the second architecture was better than that of the first architecture when using one IoT device. However, when using more than one IoT device, the first architecture outperforms the second architecture in terms of performance. Moreover, when using more than one IoT device, the resource utilization in the first architecture was better than in the second architecture because all of the IoT devices were able to successfully connect to the fog node simultaneously. Overall, with an increased number of IoT devices, the first architecture outperforms the second architecture.

Architecture 1 Implementation: Python Script vs. Bridging
In this experiment, we evaluated the first architecture (i.e., IoT-fog-cloud) using two different implementation frameworks (Python Script vs. bridging) using AWS metrics. Three experiments were performed using one, two, or three IoT devices. Using two subscribes and two publishes per device, both implementation frameworks showed insignificant differences in performance using the AWS benchmark, as shown in Figures  10-13. Comparing the performance results of the same architecture using two different implementation frameworks shows that the results of our experiment are accurate.

Architecture 1 Measurement: Mosquitto Metrics vs. AWS Metrics
In this section, we evaluate the first architecture of the cloud-based IoT environment using AWS and Mosquitto benchmarks. This is done to prove that we monitored the first architecture from two sides, the cloud layer and the fog layer, as shown in Figure 4.
The first experiment was conducted using one IoT device with two subscribe and two publish commands. The results showed that the number of subscribe and publish commands were the same for both the AWS (i.e., Subscribe Success and Connect Success) and Mosquitto (i.e., $SYS/broker/subscriptions/count) benchmarks and reflected the defined numbers of subscribe and publish commands for one IoT device, as shown in Figures 8a, 9a and 14i. This experiment was implemented using the subscribe-publish mode presented in [1]. In the architecture with three layers (IoT, fog, and cloud), the fog node subscribes to all of the message topics sent from the IoT device(s). These messages are then processed and published to the AWS cloud. Again, the cloud layer (i.e., the AWS cloud) subscribes to all of the message topics sent from the fog device. This is important, as it ensures that the IoT device(s) are connected and able to send data to the fog and then

Architecture 1 Measurement: Mosquitto Metrics vs. AWS Metrics
In this section, we evaluate the first architecture of the cloud-based IoT environment using AWS and Mosquitto benchmarks. This is done to prove that we monitored the first architecture from two sides, the cloud layer and the fog layer, as shown in Figure 4.
The first experiment was conducted using one IoT device with two subscribe and two publish commands. The results showed that the number of subscribe and publish commands were the same for both the AWS (i.e., Subscribe Success and Connect Success) and Mosquitto (i.e., $SYS/broker/subscriptions/count) benchmarks and reflected the defined numbers of subscribe and publish commands for one IoT device, as shown in Figures 8a, 9a and 14i. This experiment was implemented using the subscribe-publish mode presented in [1]. In the architecture with three layers (IoT, fog, and cloud), the fog node subscribes to all of the message topics sent from the IoT device(s). These messages are then processed and published to the AWS cloud. Again, the cloud layer (i.e., the AWS cloud) subscribes to all of the message topics sent from the fog device. This is important, Figure 13. AWS IoT message broker Subscribe.Success metric with 1, 2, and 3 IoT devices on North Virginia datacenter (cloud layer).

Architecture 1 Measurement: Mosquitto Metrics vs. AWS Metrics
In this section, we evaluate the first architecture of the cloud-based IoT environment using AWS and Mosquitto benchmarks. This is done to prove that we monitored the first architecture from two sides, the cloud layer and the fog layer, as shown in Figure 4.
The first experiment was conducted using one IoT device with two subscribe and two publish commands. The results showed that the number of subscribe and publish commands were the same for both the AWS (i.e., Subscribe Success and Connect Success) and Mosquitto (i.e., $SYS/broker/subscriptions/count) benchmarks and reflected the defined numbers of subscribe and publish commands for one IoT device, as shown in Figures 8a, 9a and 14i. This experiment was implemented using the subscribe-publish mode presented in [1]. In the architecture with three layers (IoT, fog, and cloud), the fog node subscribes to all of the message topics sent from the IoT device(s). These messages are then processed and published to the AWS cloud. Again, the cloud layer (i.e., the AWS cloud) subscribes to all of the message topics sent from the fog device. This is important, as it ensures that the IoT device(s) are connected and able to send data to the fog and then to the cloud without loss.
The number of published messages on the fog device using the Mosquitto metric (publish/messages/received) is equal to the number of published messages on the AWS cloud using the AWS metric (PublishIn Success), as shown in Figures 6a and 14m. This is because the messages generated from one IoT device are received by both the fog and the AWS cloud and no messages are lost during transmission; however, the number of messages processed by the Mosquitto broker ($SYS/broker/publish/messages/sent) on the fog device is significantly higher than the number of messages processed by the AWS broker (PublishOut Success) on the AWS cloud, as shown in Figures 7a and 14n. This is because the fog device is closer to the IoT devices, which reduces the latency of transmitting all of the messages to the AWS cloud. In addition, fog devices play an important role in decreasing the computation overhead caused by the IoT devices due to its close proximity. It also reduces AWS resource consumption by bringing computation closer to the IoT devices; therefore, the memory consumption ($SYS/broker/heap/current) on the fog node remains stable even when the number of processed messages increases, as shown in Figure 14j.
The second and third experiments on the first architecture were performed using two or three IoT devices with four or six subscribe and publish commands, respectively. The results showed that the number of subscribe and publish commands remained identical using the Mosquitto metric ($SYS/broker/subscriptions/count) and the AWS metrics (Subscribe Success and Connect Success) and reflected the defined number of subscribe and publish commands, as shown in Figure 8b Figure 14m; however, we noticed that when we increased the number of IoT devices, the number of messages sent and processed by the Mosquitto broker on the fog device significantly increased without affecting the performance, as shown in Figure 14n. important role in decreasing the computation overhead caused by the IoT devices due to its close proximity. It also reduces AWS resource consumption by bringing computation closer to the IoT devices; therefore, the memory consumption ($SYS/broker/heap/current) on the fog node remains stable even when the number of processed messages increases, as shown in Figure 14j. The second and third experiments on the first architecture were performed using two or three IoT devices with four or six subscribe and publish commands, respectively. The results showed that the number of subscribe and publish commands remained identical using the Mosquitto metric ($SYS/broker/subscriptions/count) and the AWS metrics (Subscribe Success and Connect Success) and reflected the defined number of subscribe and publish commands, as shown in Figures 8b,c, 9b,c and 14i. In addition, the number of published messages on the fog device remained equal to the number of published messages on the AWS cloud, as shown in Figures 14m and 6b,c; however, we noticed that when we increased the number of IoT devices, the number of messages sent and processed by the Mosquitto broker on the fog device significantly increased without affecting the performance, as shown in Figure 14n.
Overall, in the first experiment with one IoT device, the number of messages processed by the fog device (i.e., received from the IoT device and sent to the cloud) was higher than the number of messages received by the AWS cloud. Similarly, as the number of IoT devices increased (i.e., using two and three IoT devices), the number of messages made by the Mosquitto broker on the fog device remained higher than the number of messages made by the AWS broker on the AWS cloud. This was because the messages were transmitted through the three layers of the environment (i.e., IoT, fog, and cloud); thus, it took more time for the messages to be delivered to the AWS cloud due to the additional intermediate fog layer. Moreover, since the fog device was located close to the IoT devices, message latency decreased and the processing of the messages on the fog increased. As such, using fog computing was very beneficial when connecting more than one IoT device to the cloud. Overall, as the number of IoT devices increases, the process ability of the fog device in the first architecture outperforms that of the AWS cloud.

Threats to Validity
The experiments in this paper were implemented using real commercial sensors interacting with a real-world cloud through a commercial IoT service. The environment was vulnerable to real attacks and simulates a real-life IoT environment. We used Raspberry Pi boards to enable the sensors to send collected data through the Internet to the cloud or the fog node. We also used a Raspberry Pi board as a fog node and as the intermediate layer between the sensors and the cloud in the first architecture. The first architecture consisted of three layers (IoT, fog, and cloud), whereas the second architecture consisted of only two layers (IoT and cloud). Every layer operated on a separate network and all networks were connected to the Internet to simulate real-life Overall, in the first experiment with one IoT device, the number of messages processed by the fog device (i.e., received from the IoT device and sent to the cloud) was higher than the number of messages received by the AWS cloud. Similarly, as the number of IoT devices increased (i.e., using two and three IoT devices), the number of messages made by the Mosquitto broker on the fog device remained higher than the number of messages made by the AWS broker on the AWS cloud. This was because the messages were transmitted through the three layers of the environment (i.e., IoT, fog, and cloud); thus, it took more time for the messages to be delivered to the AWS cloud due to the additional intermediate fog layer. Moreover, since the fog device was located close to the IoT devices, message latency decreased and the processing of the messages on the fog increased. As such, using fog computing was very beneficial when connecting more than one IoT device to the cloud. Overall, as the number of IoT devices increases, the process ability of the fog device in the first architecture outperforms that of the AWS cloud.

Threats to Validity
The experiments in this paper were implemented using real commercial sensors interacting with a real-world cloud through a commercial IoT service. The environment was vulnerable to real attacks and simulates a real-life IoT environment. We used Raspberry Pi boards to enable the sensors to send collected data through the Internet to the cloud or the fog node. We also used a Raspberry Pi board as a fog node and as the intermediate layer between the sensors and the cloud in the first architecture. The first architecture consisted of three layers (IoT, fog, and cloud), whereas the second architecture consisted of only two layers (IoT and cloud). Every layer operated on a separate network and all networks were connected to the Internet to simulate real-life implementation frameworks. In both architectures, up to three sensors were used to capture temperature and humidity data and send it to the cloud.
Each experiment was performed for an hour, during which the data captured by the sensors were sent to the cloud (i.e., the AWS North Virginia datacenter) and stored there. It would be of interest to investigate whether the results can be applied to other cloud service providers or to other datacenters in different geographical areas. In addition, three sensors were used to conduct the experiments by either connecting the sensors to the cloud directly or by connecting the sensors through a fog node to the cloud. Further experiments are needed to determine how many sensors a fog node can handle before the performance is impacted.
Moreover, our results indicate that the first architecture outperforms the second architecture in publishing the data captured by the sensors to the cloud. This is attributed to the fact that in the second architecture, all of the sensors need a certificate for authentication, whereas in the first architecture, only one certificate for authentication is needed and is placed on the fog node. Notably, this leaves the communication between the IoT layer and the fog layer without a proper authentication mechanism. We plan to utilize the computation and storage capability of the fog node to implement an authentication and authorization model for the sensors that interact with the fog nodes in the future.

Discussion and Limitations
Fog computing has evolved to support cloud-based IoT environments in many ways. It is known for its ability to lower the communication latency, optimize the communication bandwidth, and enable higher network scalability and heterogeneity [52]. In addition to these advantages, fog computing has many valuable characteristics, such as fog node mobility and location awareness, in addition to the computational ability that IoT devices lack. In this paper, we demonstrated that fog computing has a substantial impact on cloudbased IoT environments in reducing latency and improving communication performance. In addition to these benefits, fog computing has a great deal of advantages that were not extensively discussed in our results.
First, fog-computing-aided IoT environments are known for their higher scalability, since every group of IoT devices is connected to a fog node. This hierarchical structure enables better management, tracking, and monitoring of IoT devices. In addition to this, compared to environments in which IoT devices are connected directly to the cloud, fogaided IoT environments improve resource utilization. This is attributed to the savings in processing capability, which are wasted when the cloud authenticates a massive number of IoT devices. At first sight, one might say that the same processing time and resources will be consumed at the fog node to authenticate IoT devices. While this is true, with a fog node, consumers of cloud services will not be overcharged for services that are wasted on the cloud to authenticate a significant number of IoT devices, especially because providers charge based on consumption.
From a security standpoint, fog computing spreads risks across distributed fog nodes in fog-aided IoT environments. In addition, authenticating IoT devices at the fog layer provides more flexibility by adding sophistication to the authentication and authorization process, for example through encryption-based access control.
Although the convenience of having a fog layer with semi-heavyweight computation capability has a higher capital cost, in the long run it saves a lot of resources, time, and money.

Conclusions and Future Works
In this paper, we proposed two architectures of cloud-based IoT environments using a real environment. In one architecture, we used a fog layer between IoT devices and the cloud, whereas in the second, IoT devices published data directly to the cloud. In order to validate our results, we also examined two ways of implementing fog-aided IoT-cloud environments, namely (1) bridging and (2) using a Python script to forward the data to the cloud. For each architecture, we conducted several experiments and increased the number of IoT devices as well as the number of subscribe and publish commands in each experiment. To evaluate the experimental performance, we used two sets of benchmark metrics, namely (1) AWS message broker metrics and (2) Mosquitto message broker metrics. The performance was evaluated based on the following analysis methods. First, we compared the performance of the first and second architectures. Second, we compared the performance of the two implementation frameworks of the fog-aided IoT environments (i.e., Python script vs. bridging). Finally, to validate our results, the performance of the first architecture was analyzed using Mosquitto metrics vs. AWS metrics. The results showed that the performance in the IoT-cloud with a fog layer was significantly better than without the fog layer, as the number of IoT devices and the number of subscribe and publish commands increased. The results also showed that the use of a Python script or fog-aided IoT-cloud environment resulted in the same performance. The results of our third analysis showed that as the number of IoT devices increased, the processability of the fog device in the fog-aided IoT-cloud architectures outperformed that of the AWS cloud. This study aimed to educate readers on different methods that can be used to implement IoT cloud environments and to compare the performance for each. It can also guide researchers by providing different ways to implement fog-aided IoT-cloud systems.
In the future, we plan to extend this study by using the same implementation frameworks presented in this paper to analyze the performance using different cloud providers, protocols, fog devices, and IoT devices. The presented implementation frameworks will also be used to investigate and address security issues in fog-aided IoT-cloud systems.