Survey and Comparative Study of LoRa-Enabled Simulators for Internet of Things and Wireless Sensor Networks

The Internet of Things (IoT) is one of the most important emerging technologies, spanning a myriad of possible applications, especially with the increasing number and variety of connected devices. Several network simulation tools have been developed with widely varying focuses and used in many research fields. Thus, it is critical to simulate the work of such systems and applications before actual deployment. This paper explores the landscape of available IoT and wireless sensor networks (WSNs) simulators and compares their performance using the Low Power Wide Area Network (LPWAN) communication technology called LoRa (Long Range), which has recently gained a lot of interest. Using a systematic approach, we present a chronological survey of available IoT and WSNs simulation tools. With this, we categorized and content-analyzed published scientific papers in the IoT and WSNs simulation tools research domain by highlighting the simulation tools, study type, scope of study and performance measures of the studies. Next, we present an overview of LoRa/LoRaWAN technology by considering its architecture, transmission parameters, device classes and available simulation tools. Furthermore, we discussed three popular open-source simulation tools/frameworks, namely, NS-3, OMNeT++ (FLoRa) and LoRaSim, for the simulation of LoRa/LoRaWAN networks. Finally, we evaluate their performance in terms of Packet Delivery Ratio (PDR), CPU utilization, memory usage, execution time and the number of collisions.


Introduction
The recent rise of the Internet of Things (IoT)-connected devices is driving the increasing demand for advanced and new technologies. The IoT describes a vision in which billions of smart devices/things/objects are equipped with sensory and communication capabilities to autonomously sense, share and exchange information for intelligent decision making [1]. Such decisions can then be used in many applications such as agriculture, transportation, healthcare, climate change, supply chain management, etc. With little or no extensive infrastructure, wireless sensor networks (WSNs), a technology often used within an IoT system, play an important role in the IoT vision due to their robust design and self-organizing network concepts [2].
WSNs consist of several (hundreds or thousands) of low-power, low-cost tiny computers or sensor nodes deployed either randomly or in a predetermined manner in a given area of interest connected via wireless communication links [3][4][5][6][7]. They are specifically designed to sense some physical properties or conditions such as pressure, humidity, temperature, and vibration from their surrounding environment and send the collected data to at least a common gateway sensor node, called a sink or base station, via the internet in an IoT system [5][6][7].
Various communication technologies to interconnect IoT and WSNs devices have been developed. One such technology that has gained growing momentum and interest • We present a chronological survey of available IoT and WSNs network simulators. • We analyze and categorize recent studies between 2011 and mid-2021 with a focus on IoT and WSNs network simulation tools by highlighting the discussed simulators, study type, scope and performance measures of the studies. • We examine and compare three popular open-source simulation tools/frameworks for the simulation of LoRaWAN networks in terms of packet delivery ratio (PDR), CPU utilization, memory usage, execution time and the number of collisions.
The rest of the paper is organized as follows. Section 2 provides an overview of the IoT architecture, review process and survey of available IoT and WSNs simulators. In Section 3, we exhibit an overview of the most popular LPWAN technologies, end device classes, transmission parameters and available simulation tools to analyze LoRa/LoRaWAN networks. Section 4 describes the methodological approach used in this work. In Section 5, we present our performance evaluation and results discussion. Finally, conclusions are drawn in Section 6.

IoT: State-of-the-Art
Even though the IoT has no universally agreed-upon architecture, many researchers and industries have proposed various IoT architectures based on their own needs and requirements [11]. However, the three-layer architecture is the most generic or basic IoT architecture [12,13]. This architecture proposes three layers, namely, perception, network and application layer. The perception layer is the physical and main part of object identification and data collection [14]. It is sometimes called the sensing layer and has several sensor nodes, actuators and gateways that cooperatively sense, gather and exchange information about the environment. The network layer, also called the transmission layer, is responsible for transmitting and processing sensed data from the sensing layer to other network devices, servers and smart things/objects. This layer also handles all data transmission. On the other hand, the application layer is responsible for providing application-specific services to the end-user. This layer defines various IoT applications, such as smart agriculture, smart health systems, smart cities, etc. [11]. Moreover, many and different IoT architectures have been proposed in various literature, such as the four-layer [15], five-layer [16], and man-like neural network architecture [17].

Systematic Literature Review (SLR)
The SLR process used in this work is similar to that used in [18], and this is because it is well-suited for our purpose. The SLR protocol consists of four main steps: • Search for the works in the domain of WSNs simulation tools: This step involves searching for published papers that discussed or mentioned WSNs simulation tools. The search was conducted on some of the most popular academic databases such as ACM, Elsevier, MDPI, Springer, IEEE Xplore and other digital libraries. In addition, the search used the following keywords: survey, comparison, review, simulator-specific, simulation tools, analytical studies, case studies, analytical study, qualitative analysis, technical report and evaluation with a focus on IoT and WSNs simulation tools. This step helps with retrieving and finding relevant papers from the pool of available scientific literature. • Manually select the relevant papers: For this step, we manually select papers between 2011 and mid-2021, considering their relevance to the subject matter. All abstracts and conclusions sections were read to select the most relevant papers for the SLR process. • Read and evaluate selected papers: For the third step, we carefully analyzed and examined the contents of the selected papers. This includes the year of publication, references, discussed or cited network simulators/emulators, type of study, scope and performance measures. • Collect the most relevant data using the data extraction table: Finally, the most relevant data were collected using the data extraction table.

Categories of Selected Scientific Papers
Based on the type of study, we divided the selected papers into five groups: Group 1: Survey and Review papers. The survey papers provide a general knowledge of WSNs simulators, such as features, advantages, disadvantages and classifications. These papers include . In particular, the authors in [27,37] present a comprehensive survey of various simulation tools. In [27], the main features, advantages and disadvantages of four network simulators, namely, NS-2, J-Sim, NS-3 and OMNeT++, are discussed. The work presented in [37] describes 16 simulators, considering their features, limitations, methodology, test-beds and hardware platforms.
In [52], the authors present a review and comparison of 15 network simulators based on the type, network impairments, deployment mode and protocol support. They further proposed evaluation methodologies and techniques to help researchers choose the best simulation tool.
Reference [53] focused on the specifics of WSNs simulations, providing a state-of-art review, features and requirements of 11 well-known and used simulators suitable for WSNs simulations. The conclusion and recommendation drawn from the work are that WSN simulators require a proper energy model with harvesting simulation support, a model of the sensed environment, and a mobility framework with localization support. An in-depth overview of 24 simulation tools is presented in [54]. The work mainly focused on the components, features, structure, implementation and usage of the simulation tools.
Moreover, in [55], the seven most widely used simulation tools for WSNs based on a set of new preferred criteria, namely, scalability, accessibility, complexity, popularity, accuracy, models, protocols and extensibility are discussed. The work further identified key limitations of the simulators with emphasis on their suitability for simulating largescale WSNs. In [56], researchers review 20 simulators and identify their features. Based on the usage, they classified them into three major domains of use: education, research, and industrial development and design.
The authors in [57] present statistical information on the seven most popular network simulators gathered during a literature survey of several research articles between 2000 and 2013. Following a simple comparison approach, they present an overview, main properties and background information on the popularity of the simulators. Based on their findings, they concluded that NS-3 and OMNeT++ simulators are good choices for academic researchers, with the latter option being better for researchers as it is more intuitive, easier to use and has a well-designed Graphical User Interface (GUI).
More than 30 simulation tools are described in [58], where their architecture, features, interface/GUI, and performance comparison are presented. The authors in [59] review 130 simulation environments for Ubiquitous Sensor Networks (USNs). The work further summarized the performance of several studies on simulation tools. Seven simulation tools are described and compared based on their license, sensor platform support, simulation code exportable, scalability, protocol design/optimization, mobile network simulation, dynamic network topology change, network support, standards, Medium Access Control (MAC) and routing support in [60].
Reference [61] examines 19 experimental tools and techniques for various WSNs applications selection based on their capabilities, ease of use, and accuracy. Finally, in [62], a comprehensive review of 12 simulation tools focusing on experimental analysis, modeling, estimation and avoiding interference is presented. The authors also provide insight into dealing with interference avoidance methods and improving coexistence mechanisms among various wireless devices operating in the same frequency band.
Group 2: Comparison papers. This group of papers includes comparisons and comparative studies of WSNs simulators based on defined criteria such as architecture, models, interface accessibility, user support, applications, extensibility, scalability, comparison tables, etc., to evaluate the differences between simulators. In addition, they also describe WSN simulators in a general way. These papers include .
Particularly, the authors in [63,74,78,[81][82][83]87] perform a comparative analysis of various WSN simulation tools. In [63], the authors propose a comparative study of three simulation tools, namely, QualNet, OPNET and NS-2, using as reference a real testbed based on recent Imote2 sensors. In addition, they evaluate the impact of various MAC protocols with respect to the IEEE 802.15.4 standard. According to their findings, the NS-2 simulator gives the closest results to reality in the case of an indoor scenario, while in the case of an outdoor scenario, the OPNET simulator gives the best results.
A comprehensive study of 14 WSNs simulation tools is provided in [74]. Out of the 14 mainstream WSNs simulators discussed, the authors further performed a comparative study of six simulators (WSNet, Castalia, COOJA, MiXiM, PASES, and TOSSIM) by designing two simulation scenarios and comparing their performance based on the packet delivery ratio, network throughput, run-time performance, packet loss at the MAC layer, power consumption estimation accuracy and network latency. Their analysis shows that Castalia, COOJA and WSNet are very efficient for large-scale network modeling, while the computation resources and running time of MiXiM, TOSSIM, and PASES are large.
In [78], a survey and comparative study of 22 open-source WSNs is presented. The authors identified their characteristics and compared them in terms of energy consumption models, scalability, mobility model and extensibility. The authors in [81] compare 23 network simulators. They considered several perspectives, including features, supported protocols, components, simulation mode, platform, main applications, visual/visibility, accessibility, support, testing, advantages, disadvantages and limitations for their comparative analysis in multi-tables.
The work presented in [82] reviews the implementation and evaluation process in WSNs. They describe relevant testbeds, simulation tools and their features. Furthermore, they conducted an experimental study using these testbeds and simulations to highlight their pros and cons. They further implemented a localisation protocol as a used case to investigate the effectiveness of the Avrora and NS-2 simulators and two testbeds. Their work clarifies future work to improve the reliability, accuracy, and time consumption for better implementation.
In [83], the architecture, features, advantages and limitations of 10 network simulators are presented. The main objective of the study is to highlight the unique characteristics of a good simulator. In addition, the performance of MATLAB/Simulink, NS-2, NS-3 and OMNeT++ simulators using the Ad-hoc On-demand Distance Vector (AODV) routing protocol is evaluated.
Lastly, the authors in [87] present a study of 10 simulation tools for other ad hoc networks, such as Vehicular Ad hoc Network (VANET), WSN, Wireless Mesh Network (WMN), etc. They highlighted areas of strength, features, operating system, supported ad hoc technologies and degree of usability of the simulators.
Group 3: Evaluation papers. This group of papers focused on evaluating WSN simulation tools. These papers include [88][89][90][91]. The authors in [88] present evaluation approaches and requirements for sensor networks to enable credible, realistic and convenient WSN evaluation. They also compared simulation models and real-world wireless link behavior in various settings. In [89], an energy-aware model for WSNs is proposed. The proposed scheme ensures an energy consumption gain that considers time constraints. In [90], Cup-Carbon, a new WSN simulator, is presented. To evaluate the ease of use of the simulator, the authors proposed a modified version of the Dijkstra algorithm that includes the battery level of the nodes as an additional parameter for calculating the best route. The authors in [91] proposed a methodological approach to evaluate WSN simulators. Using this approach, they evaluated three WSNs simulators (NS-2, OMNeT++ and TOSSIM).
Group 4: Case study papers. This group of papers focused on exploring real-life contemporary or multiple WSNs systems. In [92], the authors evaluate five WSN simulation tools, namely, Castalia, MiXiM, PASES, WSNet and COOJA, using AODV protocol as a case study. They designed a multi-hop simulation scenario in each simulator and compared their performance. Despite the simulation analysis differences and the available component models, their results show the correctness of the benchmark methods adopted and proved the functional equivalence of the tools and their network model application for multi-hop. Reference [93] performs a quantitative and comparative analysis of six network simulators used for academic purposes. The study's main objective is to identify the tools used to solve specific engineering problems in teaching-learning processes. The authors highlighted the importance of using different simulation tools, especially at the university and research environment, to promote scientific and/or technological solutions.
Group 5: Analytical study and qualitative analysis papers. This group of papers includes analytical studies and qualitative analysis. The authors in [94] present an analytical study of various network simulation tools and platforms focusing on associated main features. The study explored evaluation criteria, type of simulation, classification/categorization, designed or modified, nearby realistic experimental results and future directions. In [95], a qualitative analysis of 15 simulators for WSNs is presented. The authors also provide a detailed study and background of various WSN simulators, key features and limitations. Moreover, they compare the simulators in terms of type, event, license type, general or specific simulator, GUI support, pros and cons. Table 1 presents a chronological overview contribution of the selected papers (2011-2021). Furthermore, Table 2 summarizes the comparative studies in [64,70,72,82,83,91,92,96,97] where the authors analyzed different performance measures such as delivery ratio, computational/execution time, memory usage, CPU utilization, delay, received packets, energy consumption, among others by simulating various test scenarios.

Statistical Analysis of Selected Papers
In total, 78 relevant papers were obtained between 2011 and mid-2021. Group 1 has a total of 44 papers, which represent 56.4% of the selected papers. Group 2 has a total of 26 papers, which means 33.3% of the papers. Lastly, Groups 3, 4 and 5 have 4, 2, and 2 papers, representing 5.1%, 2.6% and 2.6% of the total selected papers, respectively. Moreover, Figure 1 shows the yearly distribution of the selected research papers. The year 2020 has the most obtained papers with a total of 11, followed by 2013 with 10, and 2012 and 2017 have 9 papers each. Even though many available simulators exist, as can be seen from Table 1, some of these simulators have higher citations than others. Figure 2  Simulators Emulators

Low Power Wide Area Networks (LPWANs) Technologies
Today, LPWANs are becoming popular as a promising mechanism to connect billions of low-cost IoT devices. They are commonly used in many applications including smart environments [98], agriculture [99], environment monitoring [100], smart cities [101], and many more. Several LPWAN technologies are already present in the market, with Nar-rowband IoT (NB-IoT), LoRa/LoRaWAN, Sigfox and Long Term Evolution for Machines (LTE-M) accounting for over 96% of the global installed or deployed base of LPWANenabled active devices according to the market research conducted by IoT Analytics in 2021 [102]. According to their estimates, NB-IoT and LoRa lead with 47% and 36% (see Figure 3) of the global installed base, respectively.
Unlike NB-IoT and SigFox, LoRa/LoRaWAN allows for private network deployments and easy integration with various network platforms [103]. Since its introduction to the market, LoRaWAN has drawn the interest of many research communities and companies due to its unique features. In short, each LPWAN technology has distinct advantages over the others, especially considering various IoT factors. A comparison between LoRaWAN, NB-IoT, Sigfox, and LTE-M technologies can be found in [103][104][105].

Long Range (LoRa)
LoRa is a radio modulation technology in the category of LPWANs technologies used for IoT devices and applications [106][107][108][109][110][111][112][113][114][115][116][117]. It was first developed by a French company called Cycleo and later acquired in 2012 by Semtech Corporation [118]. Although LoRa and LoRaWAN are often used synonymously in the literature, they refer to two different concepts in the network. LoRa deals with only the physical (PHY) layer of the stack (see Figure 4), precisely, the wireless modulation used to utilize the long-range communication link. LoRaWAN, on the other hand, is the MAC layer protocol that acts mainly as an open networking protocol and is responsible for delivering secure bi-directional communication, localization services, security and mobility between LoRaWAN gateways and end-node devices [119,120]. Essentially, LoRaWAN enables IoT devices to communicate using the LoRa wireless technology. LoRaWAN is designed and maintained by the LoRa Alliance, which is an open, non-profit association of many companies and research institutions responsible for developing and standardizing the LoRAWAN specification.
Moreover, LoRa uses the Chirp Spread Spectrum (CSS) modulation technique, where information is carried using chirp signal [121]. A chirp is a signal whose frequency increases (up-chirp) or decreases (down-chirp) over time. LoRa operates in the unlicensed sub-GHz ISM (Industry, Science and Medical) radio frequency band that vary from country to country [121,122]. Table 3 shows the various unlicensed frequency bands and channel plans available for a given country or region. For example, the LoRaWAN networks in Europe are expected to operate between 863 and 870 MHz.

MAC Options
LoRa R Modulation Regional ISM Band ...

MACLayer P HY Layer
LoRaWAN R LoRa R Figure 4. LoRaWAN protocol stack. Furthermore, LoRaWAN has official regional parameters that can be found on the LoRa Alliance website [123], where various attributes of LoRaWAN link layer protocol specifications for different regions or regulatory environments worldwide are defined. These regional parameters specifications, which are maintained and provided by the LoRa Alliance, are aimed at assisting implementers in identifying the relevant LoRaWAN frequency bands and channel plans available by country. They include physical layer parameters such as channel frequencies, channel plans, join-request messages, data rates, and maximum payload size [123]. An overview of LoRa-Alliance regional parameters can be found in [124].
Currently, LoRa devices are used in various IoT applications to address some of the world's biggest challenges ranging from smart cities [125], transportation [126], energy management [127], health monitoring [128], pollution control [129] and smart farming [130].
Moreover, three classes of end-devices, namely, Class A, B, and C, are defined in the LoRaWAN specification. Class A is the mandatory class for all LoRaWAN devices and is considered when end-devices (EDs) send data to the gateway at any time using ALOHAbased LoRaWAN MAC protocol [121]. Class B and C are extensions to Class A devices specification. In contrast to the other two classes, Class A is the most energy-efficient end-service system. Table 4 summarizes the main features and common applications of these classes.  Figure 5) consists of four parts: LoRa end devices (EDs) or nodes, LoRa gateways, a network and an application server. The end nodes are LoRa devices with the LoRa radio modulation capability that run on powered batteries for several years. Typically, the EDs have embedded sensors, transponders and microcontrollers and are connected to the LoRa gateways using a star network topology. This is because long-range star architecture better preserves the battery lifetime [120]. After receiving LoRaWAN data from several LoRa nodes, the LoRa gateways channel the data to a network server and then to various application servers for end-user usage.

Network Server
Application Server Gateways Sensor Nodes Figure 5. A typical LoRaWAN network architecture. Furthermore, the communication between the nodes and the gateways is bi-directional, allowing the nodes to perform actuations. In addition, each node can transmit to multiple gateways. At the network server level, duplicate packets are automatically filtered out, and the appropriate data are forwarded to the correct application server. LoRaWAN technology is currently used in several IoT systems for solving many unlicensed wireless connectivity [133][134][135][136].

LoRa Transmission Parameters
Five configuration parameters, namely, Transmission Power (TP), Spreading Factor (SF), Bandwidth (BW), Coding Rate (CR) and Carrier Frequency (CF), characterize the communication between the LoRa EDs and LoRa gateway(s).

Transmission Power (TP)
The TP is the power with which the transmitter sends a signal. The LoRa radio TP ranges from −4 to 20 dBm with 1 dB steps [137]. However, due to hardware implementation constraints, this range is often limited to 2 to 20 dBm [138]. The lower the TP value is, the longer the battery lifetime. Consequently, a lower TP value can decrease the transmission range. Moreover, the TP value for a particular frequency band is also a regional-dependent parameter. For example, the typical maximum transmit power for EU868-870, KR920-923 and IN865-867 is +16 dBm EIRP (+14 dBm ERP), +10 dBm EIRP (or +14 dBm EIRP) and +30 dBm EIRP, respectively. However, it is important to note that such TPs cannot be exploited whenever the LoRaWAN standard is adopted, while they are appropriate for LoRa modulation.

Spreading Factor (SF)
The SF describes how the chirps would be spread out, i.e., the number of chirps generated by each symbol (chips/symbol) [139]. Its values range from 7 to 12. An SF of 8 (SF8) denotes that each chirp represents 8 bits. Higher SF values increase the Signal-to-Noise Ratio (SNR), network range, radio sensitivity and robustness against interference. However, the energy consumption and the packet airtime increase in this case [140]. On the other hand, a lower SF increases the payload, capacity and Time-on-Air (ToA) but decreases the transmission range by lowering the processing gain.
Moreover, because of its significant importance, the network uses SFs to control congestion. The SFs used by LoRa modulation are orthogonal; i.e., multiple spread signals can be transmitted on the same frequency channel simultaneously. Table 5 summarizes the effect of SF on the data rate, receiver sensitivity, battery life and ToA. The number of chips per symbol is calculated as 2 SF . With an SF10, 1024 chips/symbol are used. However, such SFs, i.e., from 7 to 12, are the ones related to LoRaWAN, while when only LoRa transmission is adopted, the values of SFs can be selected between 6 and 12 [140]. With this, the spreading rate ranges between 2 6 and 2 12 chips/symbol. The relationship between SF, BW and chirp duration (T s ) is given by [141]: The modulation bit rate (R b ) depends on the SF and is given by the relation [141]: The symbol rate (R s ) is the reciprocal of the T s expressed as: The CR refers to the LoRa modem's forward error correction (FEC) rate that provides security/protection against interference [138]. The CR can be calculated as 4 4+n where n ∈ {1, 2, 3, 4}. By substituting the values of n, the possible CR are 4/5, 4/6, 4/7 and 4/8. A CR of 4/5 (CR4/5) means that one bit of correction code will be added with every four bits of data. When CR = 0, no FEC is applied. A higher CR offers more protection against bursts of interference but increases the ToA and power consumption. LoRa radios with different CR settings can communicate with each other using an explicit header. This is because the CR payload stored in the header of the LoRa frame structure is always encoded at CR4/8 [142]. The nominal bit rate (R b ) of the data signal can also be expressed in terms of the CR and BW as [141]: where SF ∈ {7,. . . ,12} and CR ∈ {1,. . . ,4} and rate code can be defined as 4 4+CR . Using Equation (4), the different nominal data rates computed with 125, 250 and 500 kHz are shown in Table 6. Clearly, a lower SF (for example, SF7) provides a higher bit rate than a higher SF (for example, SF12).

Carrier Frequency (CF)
The CF refers to the central frequency between 137 and 1020 MHz (with steps of 61 Hz). This range may be limited to 860 to 1020 MHz depending on the LoRa chip and region. For example, the LoRaWAN protocol in Europe uses eight uplink channels defined inside the EU863-870 MHz free ISM band [143]. The Uplink and downlink channels can be used interchangeably on the first receiving window. Furthermore, a ninth uplink and downlink channel are defined at 868.8 MHz and 869.525 MHz, respectively. The ninth uplink channel uses the Frequency-Shift Keying (FSK) modulation, while the ninth downlink channel is only used for the second receiving window [143].

Bandwidth (BW)
The BW describes the frequencies transmission band ranges over which LoRa's chirps are spread. BW is one of the main parameters of the LoRa modulation and determines the chip rate of transmission according to Equation (1). A chip rate of 125 kcps corresponds to a bandwidth of 125 kHz. The LoRa network usually operates at either 125 kHz, 250 kHz or 500 kHz. The higher the BW, the higher the data rate, but the lower the radio sensitivity. In contrast, a lower BW results in higher radio sensitivity and lower data rate. Table 7 shows the possible bit rate and the maximum application payload size for the EU863-780 MHz ISM Band. The table shows that higher SF values decrease the bit rates, and lower SF values increase bit rates. However, for the same SF, doubling the BW also causes the data rate to double. Moreover, parameters such as the ToA and payload size of a packet can be derived from the previous parameters. Figure 6 shows the LoRa packet structure. The header in the structure can be either implicit or explicit. In most cases, the CR and Cyclic Redundancy Check (CRC) are known (enabled by default) and do not change, i.e., do not need to be specified (implicit header mode) [144]. The transmission time of a PHY layer packet or ToA can be calculated using Equations (5)-(8) as follows [144]: where T preamble is the preamble duration given by Equation (6) and T payload is the time to transmit payload given by Equation (7).
T preamble = (n preamble + 4.25) · T sym (6) where n preamble is the programmed preamble length and T sym = 2 SF BW is the transmission time for one symbol.
T payload = N payload · T sym (7) where N payload is the number of payload symbols expressed as where PL is the packet length in bytes, SF is the spreading factor, CRC is the cyclic redundancy check used for error detection of the LoRaWAN packet (CRC = 1 if enabled, 0 otherwise) and IH is the Implicit Header (0 if enabled, 1 otherwise). The DE value is set to 1 when the low data rate optimization is enabled; otherwise, it is disabled (DE = 0). Figure 7 shows

An Overview of LoRa/LoRaWAN Simulation Tools
Simulation is undoubtedly essential for designing and evaluating of LoRa/LoRaWANbased applications and networks before real deployment. Over the years, several Lo-RaWAN simulation tools have been developed by researchers for examining different LoRa applications and scenarios. While some are based on discrete events, others are developed specifically for LoRa/LoRaWAN networks. An overview of commonly used open-source simulation tools with a LoRa/LoRaWAN focus is presented in [145][146][147][148]. The most widely used simulation tools are LoRaSim, NS-3, OMNeT++ (FLoRa), CupCarbon, PhySimulator, SimpleIoTSimulator and Mbed OS Simulator. Table 8 compares LoRa/LoRaWAN simulators for IoT in terms of programming language, target domain (network generic or LoRa/LoRaWAN specific), operating system and available GUI.
Specifically, for this work, we will examine in detail the simulation tools that support the LoRa/LoRaWAN framework for carrying out LoRa/LoRaWAN network simulations. With this in mind, we have chosen NS-3, OMNeT++ (FLoRa) and LoRaSim for our analysis. The reasons for the selection is discussed in Step 1 (Section 4).

LoRaSim
LoRaSim is a python-based discrete-event simulator designed to analyze the scalability of a LoRa network [137]. LoRaSim allows the deployment of N LoRa nodes (EDs) and M LoRa sinks (LoRa gateways or base stations) in a two-dimensional grid or random space. The channel model in LoRaSim is implemented based on the well-known log-distance path loss. Although LoRaSim is a simple simulator that provides great insights in terms of the network performance, however, acknowledgements (ACK) are not implemented [150]. Thus, it cannot be used to investigate the different aspects of network performance, especially when the nodes switch their SF based on the presence or absence of feedback from the gateway [150]. Moreover, LoRaSim only supports uplink transmissions and cannot be used to evaluate the Adaptive Data Rate (ADR) mechanism, which is essential for optimizing the network performance. It is worth mentioning that LoRaSim offers the possibility to run networks with multiple gateways by adjusting the SF and transmit power of the end node based on its distance from the gateway. For LoRaSim to work smoothly, packages such as SimPy, matplotlib and NumPy are required. It also offers a visualization plot of the network deployments but no graphical interface. Users can see much simulation information on the Command-Line Interface (CLI). LoRaSim has proved to be a big success in many research works. Many researchers have extended or improved it to suit their needs [156,157,161,163].

Framework for LoRa (FLoRa)
FLoRa is a simulation framework that utilizes the OMNeT++ simulator and the INET framework for carrying out end-to-end simulations for LoRa networks [153]. It allows complete simulation of the LoRa/LoRaWAN network with its main components. FLoRa is implemented based on the LoRaWAN specification for class A EDs with unconfirmed transmission mode. Through the ADR mechanism, the network server and nodes support the dynamic management of configuration parameters [153]. The ADR mechanism controls the SF, BW and TP parameters of EDs. In contrast to other simulators, FLoRa provides a friendly user interface and a graphical representation of the network scenarios.
Moreover, FLoRa offers an accurate LoRa physical layer model and an end-to-end simulation with one (or more) gateways. The communication between the gateway(s) and the network server(s) is via the Internet Protocol (IP). The physical layer between the gateway(s) and the network server can be realized with the existing INET framework modules. However, FLoRa has its limitations and drawbacks. For example, it does not take into account any interference and mobility. Moreover, the ADR algorithm implemented in FLoRa does not support unconfirmed transmission mode, and the network server's assigning of SFs is also not supported. To address some of the aforementioned problems, researchers in [165] have proposed a new simulator called Advanced Framework for LoRa (AFLoRa) based on the FLoRa simulator. AFLoRa is an updated version of the original FLoRa simulator with significant enhancements and additional LoRaWAN features. Many researchers have also validated their work using the FLoRa framework [166][167][168][169][170][171].

LoRaWAN Module for NS-3
NS-3 is an open-source discrete-event network simulator designed primarily for educational and research purposes [172]. It is an extensible network simulation platform used under the GNU GPLv2 license. One of the fundamental design goals of NS-3 was to improve the realism of the models by allowing the model's implementation closer to the actual software or real-world implementations that they represent. The core and models of NS-3 are implemented in the C++ programming language, with an optional Python Scripting API interface. Users can either use C++ main() or Python program to write their simulation scripts.
The LoRaWAN module for NS-3 is an extension of the NS-3 module for the simulation of LoRaWAN networks. Each LoRa end device and gateway of the LoRaWAN module for NS-3 contain a single LoRaWAN MAC/PHY pair component, and the interac-tion/communication between each end device's PHY layer with its respective gateway's PHY layer is through the spectrum channel module [151]. It supports LoRaWAN Class A EDs specifications. Moreover, the capture effect is the basis for the collision model used in the NS-3 LoRaWAN module. This effect occurs when two simultaneous uplink transmissions with the same frequency and SF collide, and the stronger signal captures the weaker signal. As a result, the gateway only receives the frame with the strongest received signal power. Many researchers over the years have developed different versions of NS-3 modules for the simulation of LoRaWAN networks. For the first time, the authors in [173] present a comprehensive survey of four different implementations of LoRaWAN modules in the NS-3 simulator. They labeled them as Module I through IV based on the date they were made publicly available and further compared them to highlight the most appropriate scenarios for each module. The four modules are available and free to download at GitHub, an internet code hosting platform for software development and version control. Most of the LoRaWAN specifications not found in the FLoRa framework are implemented in the NS-3 LoRaWAN module. In addition, compared to NS-3 LoRaWAN, FLoRa implementation is more difficult. Many researchers have validated, improved or extended their work using either the different implementations of the NS-3-based LoRaWAN modules or their proposed LoRaWAN modules in the NS-3 simulator [174][175][176][177][178][179][180][181][182][183][184][185][186]. A comparison of NS-3, FLoRa and LoRaSim with a focus on the LoRa/LoRaWAN framework is given in Table 9.

Methodological Approach
The methodological approach used to analyze and evaluate the selected LoRa/LoRaWAN simulators (i.e., OMNeT++ (FLoRa), LoRaSim and NS-3) in this work is similar to that proposed by the authors in [91]. However, we slightly modified the methodological approach to fit our interests and direction. The methodological approach consists of six steps: Step 1. Identify the simulator(s) to evaluate: The network simulators to be compared and evaluated need to be identified based on criteria to assess the simulators' various aspects. The network simulators for this purpose were selected based on five criteria: i.
The free availability of the simulator for academic and research purposes. ii. The active development of new models and protocols by the practitioners and the research community. iii. The availability of supporting documentation for the simulators. iv. The general purpose of the simulator(s) with respect to the IoT and WSNs applications. v.
The growing popularity of the simulators among academics and research communities for the simulation of LoRa/LoRaWAN network.
Based on the above criteria, we selected OMNeT++ (FLoRa), LoRaSim and NS-3 simulators for our analysis. Moreover, for the case of NS-3 LoRaWAN module, we used the NS-3 LoRaWAN Module I for our work. This is because of its excellent documentation and the most preferred module by many research communities.
Step 2: Establish the experiment setup: The platform on which the simulators are installed and run should be the same to properly compare and evaluate their performance. For this step, we installed the three simulators on Linux Ubuntu 20.04 LTS platform running on Microsoft Windows 10 version 21H1 with 19043.1466 OS build. The computer specifications are Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz 2.71 GHz with 4.00 GB of RAM (2.2 GB of disk allocated for Linux) and a 64-bit operating system x64-based processor.
Step 3: Defined the performance assessment/metrics: More precisely, we evaluate the following metrics: • Packet Delivery Ratio (PDR) : This can be defined as the total number of received packets by the network server divided by the total number of packets sent by the end nodes. The PDR can be computed per node or for the whole network. It is one of the well-known performance metrics in the sensor networks literature. For the entire network, this can be computed as shown by Equation (9): • CPU Utilization: This refers to the amount of work a Central Processing Unit (CPU) handles. It is used to estimate the system's performance. Because some tasks require a lot of CPU time while others require less, CPU utilization can vary depending on the type and amount of computing task. • Memory Usage: This is the memory requirement used by an application while the program executes. It is critical to keep track of memory usage to ensure peak performance. • Execution Time: This refers to the end-to-end time to perform one single simulation run, i.e., the interval between the start and the end time of the simulation scenario. • Collisions: With collision, we refer to the phenomenon that occurs when two or more devices or stations attempt to transmit a packet (data) simultaneously, resulting in the possible loss of transmitted data. Note that the concept of collision or how it is detected may vary depending on how the simulator defines the collision criteria.
Step 4. Design a test scenario: A test scenario needs to be designed in each simulator to evaluate their performance. For this work, we designed a small-scale IoT scenario with several sensing nodes and some actuators with LoRa communication technology. Test scenarios are defined by parameters that describe a specific use case or test case execution. For our comparison analysis, we simulated the test scenario with the support of the available LoRa frameworks/modules in these simulators. The scenario consists of a single gateway in a two-dimensional space of 100 m × 100 m and a varying number of EDs around the gateway, ranging from 50 to 400 EDs. The EDs are distributed randomly in the simulation area. The gateway, which is connected to one network server, facilitates communication in the network. To generate a realistic data traffic, we configure the EDs to transmit data packets with 51 bytes and a transmission interval of 100 s.
Step 5. Execute the designed scenario: The designed scenario is executed to obtain the needed results for the evaluation. Test scenarios often need to be executed multiple times with variations. In this work, the simulation was run several times for a given number of EDs (six times).
Step 6. Analyze and evaluate the result(s): The performance analysis of the simulators is measured based on the obtained results. Users can select the most appropriate simulator(s) according to their needs and applications. Tables 10 and 11 summarize the main simulation parameters and different versions of the simulators used, respectively.  Figure 8 shows the PDR (%) as a function of the number of nodes for SF = 7 and SF = 12. We set BW = 125 kHz and CR = 4/5 in all configurations. Moreover, the number of nodes ranges from 50 to 400. The results in Figure 8 show that a higher packet success probability is achieved with SF = 7 (dotted lines) due to shorter packet transmission. In contrast, a lower packet success probability is achieved with SF = 12 (solid lines) due to longer packet transmission. Note that shorter packets require more headers than longer packets. Hence, it is not difficult to conclude that a lower SF results in higher PDR, while a higher SF results in lower PDR.

PDR:
For the simulators, we can see that the NS-3 LoRaWAN module achieved higher PDR with SF = 7, followed by OMNeT++ (FLoRa) and LoRaSim. However, with SF = 12, we observed that for all the simulators, the PDR decreases as the number of nodes increases. In this case, OMNeT++ (FLoRa) shows a much better PDR than both NS-3 LoRaWAN module and LoRaSim. This can be attributed to the fact that OMNeT++ (FLoRa) received more packets than the NS-3 LoRaWAN module and LoRaSim. Therefore, it can be concluded that the NS-3 LoRaWAN module performs better with lower SF while OMNeT++ (FLoRa) performs better with higher SF.

CPU utilization:
The CPU utilization (%) for the simulators was measured while varying the number of nodes in the network scenario. Figure 9 shows the average percentage of CPU usage for OMNeT++ (FLoRa), NS-3 LoRaWAN module, and LoRaSim simulators along with the 95% confidence intervals on the plot. Because both NS-3 LoRaWAN module and LoRaSim have only a CLI, we also run the OMNeT++ (FLoRa) simulation using both the CLI and GUI. When the network size is larger, the CPU utilization for the three simulators does not differ much. In particular, the CPU usage at 400 EDs was approximately 76%, 78% and 80% for LoRaSim, NS-3 LoRaWAN module and OMNeT++ (FLoRa), respectively. Thus, compared to OMNeT++ (FLoRa) and NS-3 LoRaWAN module, LoRaSim had the lowest CPU usage percentage at 400 nodes. However, from about 80 to 360 nodes, we observed that OMNeT++ (FLoRa) uses less CPU while the NS-3 LoRaWAN module uses the highest CPU. However, for smaller networks (50-70 nodes), LoRaSim uses the lowest CPU usage. Moreover, the dotted line on the plot depicts the CPU usage for OMNeT++ (FLoRa) when the GUI is utilized. Additionally, we run the simulation using the express mode. We observed a high CPU usage percentage (approximately 85%) when the OMNeT++ (FLoRa) GUI is utilized. This high percentage can be due to the high CPU processing requirements for the GUI. Execution time: Figure 10 shows the average execution time in seconds versus the number of nodes for the three simulators, along with 95% confidence intervals. We observed that the execution time for the LoRaSim is considerably lower than that of the NS-3 LoRaWAN module and OMNeT++ (FLoRa) simulators. It is also evident that the NS-3 LoRaWAN module has the highest execution time, from 50 to approximately 270 nodes; i.e., the NS-3 LoRaWAN module takes much longer to execute the simulation than OMNeT++ (FLoRa) and LoRaSim. On the other hand, OMNeT++ (FLoRa) appeared to have an average execution time for the scenarios. However, for a large network size (280-400), OMNeT++ (FLoRa) requires more execution time than the NS-3 LoRaWAN module and LoRaSim. In terms of execution time, we can conclude that LoRaSim appears to be the most efficient in this context. Memory Usage: Figure 11 shows the graph of the average memory usage vs. the number of nodes for OMNeT++ (FLoRa), NS-3, and LoRaSim simulators. In the figure, the x-axis represents the number of nodes varied from 50 to 400, and the y-axis represents the memory usage in percentage (%). Again, a 95% confidence interval is shown in the figure. We observed that as the number of nodes increases, there is somewhat a linear growth in the amount of memory usage for the simulators, with minor differences. The NS-3 LoRaWAN module uses the lowest amount of memory, while OMNeT++ (FLoRa) uses the highest. LoRaSim, on the other hand, appears to use a moderate amount of memory.
Moreover, the memory usage for OMNeT++ (FLoRa) when the GUI is used is shown in the figure with a dotted line. Again, the express mode is used to obtain the memory usage in OMNeT++. We noticed a high percentage of memory usage with the OMNeT++ GUI. Of course, this high percentage of memory consumption can be attributed to the fact that GUI requires relatively more memory as it contains a lot of graphical components. In contrast, CLI does not require more memory consumption or usage. Additionally, every module requires its CPU stack, leading to more significant memory requirements for the simulation program. Overall, the NS-3 LoRaWAN module was found to be the most efficient in this regard.  Figure 12 illustrates the number of collisions occurring in the simulation as a function of the number of nodes. The figure shows that the number of collisions increases linearly when the number of nodes increases. The total number of collisions for a simulation should be minimal to achieve the highest performance. This is because an increased number of collisions lead to network performance degradation. In the figure, we can see that the number of collisions rapidly increases with higher SF. Obviously, with SF = 12, we expect more collisions due to the longer packets. LoRaSim has the highest number of collisions when SF = 12, followed by the NS-3 LoRaWAN module and OMNeT++ (FLoRa). However, with SF = 7, the NS-3 LoRaWAN module has fewer packet collisions. Thus, from a collision point of view, the number of collisions in the NS-3 LoRaWAN module is lower than in the other two simulators with a lower SF value.

Summary and Conclusions
This paper provides a detailed chronological survey of available IoT and WSNs simulation tools. Specifically, we highlight the most important works from recent studies using a systematic review approach. Next, we present an overview of LoRa/LoRaWAN technologies. We also provide a detailed background on the LoRa/LoRaWAN network, its transmission parameters, classes of its end-devices and available simulation tools. Then, we present a comparative study of three open-source simulation tools/frameworks, namely, NS-3, LoRaSim and OMNeT++ (FLoRa), for the simulation of LoRa/LoRaWAN networks. In each simulator, we equally implemented a simple IoT scenario that used the LoRa communication framework and compared their performance in terms of the Packet Delivery Ratio (PDR), CPU utilization, memory usage, execution time and the number of collisions. The simulation statistics were collected and analyzed at the end of the simulations. Despite the differences in the compared simulators and the obtained results, we would like to acknowledge that each simulator is preferable under different performance measures, depending on the primary research direction and objection.
Finally, many open issues and challenges to developing a more realistic LoRa/LoRaWAN network simulation exist. All the presented LoRa/LoRaWAN simulators have unavailable features in their frameworks that can further be implemented: for example, the incomplete implementation of the LoRaWAN specification as defined by the LoRa Alliance. Moreover, essential features such as interference between partially overlapping channels, confirmed transmission mode, support for classes B and C, duty cycle restrictions, transmission queue, and sophisticated ADR algorithms can be explored. However, because of the free availability (open-source) and the active development of these frameworks by various academic researchers and communities, we expect significant improvement of the available and newly developed simulation tools for LoRaWAN network simulation in the future.