Comparative Performance Analysis of Data Transmission Protocols for Sensor-to-Cloud Applications: An Experimental Evaluation ^†

Abstract

This paper examines some of the most popular protocols for transmitting sensor data to cloud structures from publish/subscribe and request/response IoT models. The selection of a highly efficient message transmission protocol is essential, as it depends on the specific characteristics and purpose of the developed IoT system, which includes communication requirements, message size and format, energy efficiency, reliability, and cloud specifications. No standardized protocol can cover all the diverse application scenarios; therefore, for each developed project, the most appropriate protocol must be selected that meets the project’s specific requirements. This work focuses on finding the most appropriate protocol for integrating sensor data into a suitable open-source IoT platform, ThingsBoard. First, we conduct a comparative analysis of the studied protocols. Then, we propose a project that represents an experiment for transmitting data from a stationary XBee sensor network to the ThingsBoard cloud via HTTP, MQTT-SN, and CoAP protocols. We observe the parameters’ influence on the delayed transmission of packets and their load on the CPU and RAM. The results of the experimental studies for stationary sensor networks collecting environmental data give an advantage to the MQTT-SN protocol. This protocol is preferable to the other two protocols due to the lower delay and load on the processor and memory, which leads to higher energy efficiency and longer life of the sensors and sensor networks. These results can help users make operational judgments for their IoT applications.

1. Introduction

We are witnessing times when modern wireless sensor networks generate vast amounts of sensor data every hour, every day, from various sources such as smart cities, transportation, healthcare, military industry, agriculture, industrial production, and others. During the different stages of transmission, processing, and storage of this data, guarantees for security and long network life are demanded. It is known that wireless sensor networks (WSNs) do not have the necessary hardware and firmware capacity. Therefore, the data passing through the sensors leads to rapid consumption of their energy resources and a decrease in the overall life of the networks [1,2]. Trends related to solving the above challenges lead to designing and producing increasingly intelligent sensors that allow operation as end devices. These smart sensors differ from traditional sensors with integrated platforms, including microprocessors for data processing and storage, diagnostic software, and communication tools, capable of transforming the collected data into a ready-made analysis for decision-making. Since smart sensors can process signals directly, measurement results or execution of specific analytical applications can be performed immediately, and decisions can be made immediately, which turns these sensors into near-end devices [3,4].

The possibility of processing and storing sensor data in cloud structures, based on the “Sensor–Cloud” integration paradigm, is gaining popularity and is increasingly being applied in practice. Clouds are flexible, powerful, and cost-effective technology that provides real-time data to users at any time, with massive coverage and quality [5,6]. Cloud platforms offer the appropriate technology to process large volumes of collected sensor data, enable real-time data monitoring, and have the necessary equipment to fulfill integration requirements [7]. Today, organizations that decide to move to cloud storage and data processing benefit from a simplified IT infrastructure and administration, remote access to their data from anywhere on the Internet, and at a reasonable price. In [8], we presented a diagram of integrating data from sensor networks in the cloud, which solves essential questions about the cost of sensor networks and the possibilities of storing sensor data in specialized cloud databases. We analyze the features of sensor data and the specific requirements and models for their storage and processing. The most appropriate way to build an integration infrastructure is to apply the IEEE 1888 standard [9], where data is transmitted by protocol.

One of the severe challenges is the interaction between the WSN and the cloud during the transmission of the sensor data to the cloud. An appropriate integration protocol must be chosen when using the cloud, which inherently determines how the data arrives from the sensors to the cloud (at what speed, format, delay, etc.). Sensor–cloud communication can be implemented through different architectural models depending on the adopted approach for data transmission through a gateway or coordinator, or directly between sensor networks and clouds, some of which we have explored in our work [10]. The most challenging task in designing IoT systems is building a communication channel between local devices, gateways, and cloud platforms. Data can be exchanged using different protocols [11,12]. In [13], the delay in transmitting sensitive patient data via AMQP and CoAP protocols in remote patient monitoring systems was investigated.

The right choice of communication protocol is important, as it significantly determines the effective collection, transmission, and processing of data in the cloud, which is crucial for real-time decision-making and forecasting. Poor protocol selection can lead to increased network congestion, lower performance and security, higher power consumption, and reduced efficiency of the designed system, compromising the benefits of cloud-based analytics and storage.

This paper explores considerations for selecting the optimal protocol for integrating sensor data into a cloud. The characteristics of commonly used communication protocols are analyzed, real experiments are conducted to study the performance of the HTTP, CoAP, and MQTT-SN protocols supported by the ThingsBoard cloud, and recommendations are provided based on experimental results. The obtained experimental results allow for clarification of the protocols’ influence on the design of sensor–cloud structures and the most appropriate choice of communication protocol.

2. Related Works

The characteristics of commonly used communication protocols are analyzed, real experiments are conducted to study the performance of the HTTP, CoAP, and MQTT-SN protocols supported by the ThingsBoard cloud, and recommendations are provided based on experimental results. The obtained experimental results allow for clarification of the influence of the protocols on the design of sensor–cloud structures and the most appropriate choice of communication protocol. Sensor connectivity and cloud technologies provide the network infrastructure and communication capabilities needed by devices to transport, collect, and exchange data over the Internet and for remote monitoring and control. Connectivity is implemented through a range of communication technologies, such as the following [14]:

• LPWAN technologies such as Sigfox, LoRa, and NB-IoT;
• Cellular technologies, such as 2G, 3G, 4G, and 5G;
• Short-range wireless technologies such as Wi-Fi, Bluetooth, and Zigbee;
• Satellite technologies such as VSAT, BGAN, and the latest satellite-based LPWAN.

Countless interconnected devices communicate with each other and produce significant data. However, more than a connection is needed for the devices to communicate. They must speak the same “language.” This is where communication protocols come in. Regardless of the type of technology, the biggest challenge in developing sensor–cloud connections is creating a communication channel between devices, gateways, servers, and cloud platforms. So, to build secure communication, it is necessary to use different protocols.

Research on the applicability of communication protocols for IoT devices published in [14] shows that there is no single global standard for a protocol implementing communication. On the other hand, each protocol includes multiple layers that must be compatible. For example, the communication protocol layers include a physical connectivity layer, a data transport layer, and an application layer, and above them, there is data processing aggregation, data processing (e.g., Kafka, RapidMQ), data storage and databases (e.g., MongoDB), and finally analytics and use cases relevant applications (e.g., machine vision, asset management). A significant problem in effective communication is that specific devices may only support proprietary protocols. These protocols are created and owned by a single organization or individual, often making them incompatible with other standard hardware. In such cases, a protocol converter is needed [14]. In [13], the authors show increased use of protocols specifically designed for IoT and sensor devices, such as HTTP, CoAP, MQTT, MQTT-SN, DDS, XMPP, and AMQP. According to the study, the most significant interest is in applying the MQTT and CoAP protocols, due to their ability to work in networks with remote configuration. In [15], a comparative analysis of the characteristics and evaluation of the MQTT, CoAP, AMQP, and HTTP protocols for transmitting messages from sensor devices to cloud structures was made. According to the authors, CoAP performs better than HTTP and offers more functionality than MQTT, but it all depends on the experimental conditions. The authors in [16] empirically study the effects of edge-based and cloud-based services in mobile sensor vehicle networks using HTTP, CoAP, and MQTT protocols. Different scenarios are used in which vehicles interact with the network infrastructure to exchange small-sized data comparable to the vehicle sensors. The effectiveness of the protocols is evaluated, depending on the speed of the vehicles. It is concluded that CoAP outperforms MQTT (at QoS1 and QoS2) and HTTP in the case of small message transmission in terms of throughput and latency in different case studies. In [17], it is stated that MQTT-SN is better than CoAP when transmitting periodic messages due to its smaller packet size, which balances the overhead of connection and topic registration. Conversely, if clients connect spontaneously, stochastically, or with high dynamics, CoAP would be the better choice due to its communication model.

Table 1. Protocols for WSN–cloud data integration.

Features	MQTT/ MQTT-SN	CoAP	HTTP
Communication model	Publish/Subscribe	Request/Response or Publish/Subscribe	Request/Response
Protocol type	Many-to-many	P2P (Peer-to-peer)/one-to-one	Peer-to-Peer
Transp. Protocol	TCP/UDP	UDP	TCP
Header Size	2-Byte	4-Byte	Undefined
Security	TLS/SSL	DTLS, IPSec	TLS/SSL
Default Port	1883/8883 (TLS/SSL)	5683 (UDP Port) 5684 (DLTS)	80/443 (TLS/SSL)
IoT and Organizational Support	Facebook, IBM, Cisco, ThingsBoard, ThingSpeak, Software AG, Tibco, ITSO, M2Mi, Amazon, AWS, InduSoft, Fiorano	Cisco, Azure, AWS, ThingsBoard, ThingSpeak, Large Web Community, Support, Cisco, Contiki, Erika, IoTivity	Not reliable for IoT, but maintained by Azure, AWS, ThingsBoard, ThingSpeak, and Global Web Protocol Standard
Licensing Model	Open Source	Open Source	Free

summarizes some of the characteristics of commonly used protocols for transmitting sensor data to the cloud. Hardware solutions such as IEEE 802.15.4/6, Z-wave, 802.11 Wi-Fi, Bluetooth Low Energy (BLE), Zigbee, and HTTP typically define the communication channel layer in sensor networks.

2.1. HTTP

HTTP defines how messages on the Internet and websites are transmitted and formatted. HTTP is based on the RESTful request/response web architecture [18]. HTTP is unsuitable for transferring small packets in sensor or IoT device communications over TCP/IP. HTTP uses the client–server model. A suitable method for implementing HTTP in IoT may be to define the IoT device as an ad hoc client rather than server-dependent. So, building an IoT device that makes ad hoc (on-demand) connections rather than receiving ones from servers is better. Like CoAP, HTTP uses URIs instead of topics.

2.2. MQTT-SN

MQTT-SN is an adjusted version of MQTT for WSN, making it fit for sensor devices due to its low power, bandwidth limitation, and compact messages [12]. MQTT-SN works on the UDP/IP transport communication protocol, which is lighter than TCP/IP. The privilege of using UDP is removing the “handshakes” of TCP, so the communications become connectionless; thus, they are more lightweight. There are three types of MQTT-SN components: MQTT-SN clients, MQTT-SN GW gateways, and MQTT-SN forwarders. MQTT-SN clients connect to the MQTT broker/server via the MQTT-SN GW using the MQTT-SN protocol. MQTT-SN forwarders are responsible for transporting messages to the GW [19,20].

2.3. CoAP

CoAP is a lightweight M2M and binary-based protocol. CoAP relies on both request/response and resource/observe (a publish/subscribe variation) architecture. CoAP is mainly designed to interact with HTTP and the RESTful Web through simple proxies [21]. In contrast to MQTT, CoAP uses a URI instead of topics. The publisher publishes data to the URI, and the subscriber subscribes to an individual resource indicated by the URI. When a publisher publishes new data to the URI, all the subscribers are alerted about the unique value shown by the URI. CoAP works on UDP and ensures a minimum packet size since UDP uses up to 8 bytes as a packet header. Additionally, CoAP adds only 4 bytes as the application header. It is Internet Protocol (IP)-based, so it uses the standard IP stack and is natively compatible with the Internet. CoAP uses Datagram Transport Layer Security (DTLS), so clients and servers communicate through connectionless datagrams with less reliability. CoAP uses Representational State Transfer (REST), a communication model familiar with HTTP. This model made it easier for CoAP to interact with HTTP without a complex translator. In general, CoAP supports two modes: reliable and non-reliable. Reliability in CoAP is processed by using Confirmable (CON) messages. Acknowledge (ACK) packages will reply to every CON message sent to the server from the client. CoAP also supports the “Piggybacked Response” ACK message, which means an ACK can be included in response messages [22].

3. Materials and Methods

First, we design a hardware platform that creates an entire XBee/Zigbee sensor network, working in Mesh network mode, that gathers local environmental sensor data. The network uses Raspberry Pi (RPI) as a gateway to send the collected data to the cloud over different protocols (HTTP, MQTT-SN, CoAP), according to Figure 1. The delay is measured at the RPI4 using the delay function, which measures the time from sending the packets to receiving the ASK.

To establish the collection of sensor data, we used the Wall router, which is composed of temperature and light sensors, as well as a Zigbee radio module [23,24,25] (

Table 2. Technical characteristics of the hardware components.

Name	Technical Parameters
Wall Router	Protocol: Zigbee; Frequency: 2.4 GHz; Sensor sensitivity: −102 dBm; Sensor type: Temperature and light; Transmission speed/rate: 250 Kbps; Zigbee module’s range: from 40 m to 120 m; Temp. Sensor Range: 0 °C to 70 °C; Light Sensor Range of spectral bandwidth: 360 to 970 nm; Indoor Range: Up to 90 m; Outdoor RF Line of sight range: Up to 1.6 km; Transmit Power: 50 mW
Raspberry Pi4 A	CPU: Quad-core 1.5 GHz Core type: Cortex-A72 (ARM v8) 64-bit RAM: 1–8 GB USB Ports: 2 USB 2.0, 2 USB 3.0 Ethernet: Yes Wi-Fi: Yes Bluetooth: Yes

). The XCTU software platform controls the technical settings of the Wall router. Our project uses the Raspberry Pi 4 Model A. It offers high CPU speed, multimedia performance, and intelligence compared to the previous-generation Raspberry Pi 3 Model A, while maintaining backward compatibility and similar power consumption. For the end user, the Raspberry Pi 4 Model A provides desktop performance comparable to base x86 PC systems [26,27]. Some of the essential technical characteristics of the hardware components involved in the experiment are listed in Table 2.

Second, we developed Python code (in RPI4) and used it to design different scenarios for conducting experimental research, collecting sensor data, generating the integration protocols, and creating variable parameters with different parameter values for each scenario: number of packets (P) transmitted, number of topics (T) number per packet, and byte value/size (bS) per topic.

Third, we create an account, develop settings, and connect to the ThingsBoard cloud. We execute the code designed to transmit, process, and store data in the cloud database.

Fourth, we use the variable parameters (number of packets transmitted, topic number per packet, and byte value per topic) and trace the conditions for data transmission, like packet delay (nanoseconds) and energy (CPU/RAM load in MHz/MB) consumption.

We use Wireshark software tools in the fifth step to trace the transmitted packets’ destination and size verification for each examined protocol.

4. Results and Discussion

4.1. Tracking the Impact of Parameters and Protocols on the Delay of Transmitted Data

Figure 2 shows that the studied protocols decrease their performance with increasing values of the variable parameters. MQTT-SN and HTTP maintain a stable proportional ratio. Since the size of HTTP messages is the largest, the most significant delay in data transmission is also observed here. The CoAP protocol ranks second in delay. An interesting trend is observed—increased delay values when the number of packets increases above 70 per second in peak values, which reduces the performance of CoAP. Even though when transmitting data through a gateway, the MQTT-SN protocol performs a complex conversion, which leads to an unavoidable delay in data transmission, the delay in MQTT-SN is the smallest.

Therefore, since the two protocols are “light”, small values of the variable parameters should be used to favor their maximum capability. On the other hand, this leads to a higher communication channel performance when transmitting sensor data in the cloud. The XBee/ZigBee modules also have a limited capability level for data transmission. One reason is the restricted time window in which XBEEs can transmit I/O data. The manufacturer recommends the time window settings, and we do not change them during our testing due to the risk of reducing XBee performance. Varying parameters (P, T, bs) result in different delay times, highlighting the importance of optimizing these parameters for efficient data transmission. MQTT-SN performance is the best among the other protocols despite the limited usage. Conversely, CoAP shows a higher level of “sensitivity” (dependency) from the variable values (especially values for topics and byte size in our cases). On average, when values are roughly above 70 packets, topics, and byte sizes, CoAP shows significantly lower results than HTTP.

4.2. Energy Consumption

As expected, MQTT-SN has the lowest CPU load compared to other protocols (Figure 3). Regarding CPU load, CoAP and HTTP show almost the same values when the transmitted packets are up to 70. Increasing the transmitted packets above 70 leads to a sharp increase in CPU load when communicating via the CoAP protocol. Therefore, when transmitting a small number of packets, up to 70, the MQTT-SN and CoAP protocols are almost the same in CPU load.

The RAM load results in Figure 4 show that MQTT-SN uses the least RAM compared to CoAP and HTTP. The CoAP and HTTP protocols show similar results but vary depending on the data transmitted. Again, a significant RAM load is observed when increasing the number of packages above 70.

4.3. Verification of the Functionality of the Program Code

Using the Wireshark tool, we trace the destination and check the transmitted packet size for each protocol studied. Figure 5 shows the transmitted data via HTTP, MQTT-SN, and CoAP protocols and proves the data sent via the UDP protocol, which MQTT-SN and MQTT-SN gateway use.

5. Conclusions and Future Work

The paradigm of integrating data from sensor networks to various cloud platforms enables us to overcome sensor networks’ hardware and firmware limitations and process the collected data. This integration significantly reduces the operating costs of sensor networks and increases their application through monitoring and diagnostics of remote objects, which is especially important for critical infrastructure. One of the challenges in implementing the sensor–cloud paradigm is building a communication channel through various protocols between sensor devices, gateways, and cloud platforms. Its importance is also evidenced by the increased interest in research on this problem by the business and scientific communities.

The present document shows the results of the experimental studies, which involved data transmission from a stationary sensor network to a public cloud. The sensor network is built from XBee/Zigbee sensor devices, adopting a mesh topology for data transmission. The collected environmental data is transmitted from the stationary sensor network to the cloud through the Raspberry Pi as a gateway through the HTTP, MQTT-SN, and CoAP protocols. An algorithm and Python programming code were developed to conduct the experiments under different scenarios.

The experiments on a real-built sensor network and its logical connection with a cloud allowed us to study essential parameters for the performance of the resulting communication channel, such as the delay of transmitted data and CPU and RAM load. The functionality of our developed code has been verified with the Wireshark software tool.

Our survey shows that each protocol reduces productivity when increasing the variable values (to one degree or another). MQTT-SN shows the lowest dependency on the variable values, and HTTP is in second place. CoAP shows the highest dependency on the increasing variable values and substantially lower results than HTTP. During the experiments, an interesting trend was found, leading to the determination of limit values for the maximum number of packets transmitted to the cloud platform for the implemented hardware configuration regarding the COAP protocol. Increasing the number of transmitted packets above 70 degrades the performance of the communication channel.

In conclusion, the MQTT-SN protocol is preferable to the other two protocols for collecting environmental data from stationary sensor networks due to its lower delay and CPU and RAM load. This results in higher energy efficiency and longer lifetimes of sensors and sensor networks. Of course, any experimental result depends on hardware, firmware, and software configurations.