An Experimental Evaluation of the Reliability of LoRa Long-Range Low-Power Wireless Communication

Abstract

Recent technological innovations allow compact radios to transmit over long distances with minimal energy consumption and could drastically affect the way Internet of Things (IoT) technologies communicate in the near future. By extending the communication range of links, it is indeed possible to reduce the network diameter to a point that each node can communicate with almost every other node in the network directly. This drastically simplifies communication, removing the need of routing, and significantly reduces the overhead of data collection. Long-range low-power wireless technology, however, is still at its infancy, and it is yet unclear (i) whether it is sufficiently reliable to complement existing short-range and cellular technologies and (ii) which radio settings can sustain a high delivery rate while maximizing energy-efficiency. To shed light on this matter, this paper presents an extensive experimental study of the reliability of LoRa , one of the most promising long-range low-power wireless technologies to date. We focus our evaluation on the impact of physical layer settings on the effective data rate and energy efficiency of communications. Our results show that it is often not worth tuning parameters, thereby reducing the data rate in order to maximize the probability of successful reception, especially on links at the edge of their communication range. Furthermore, we study the impact of environmental factors on the performance of LoRa, and show that higher temperatures significantly decrease the received signal strength and may drastically affect packet reception.

1. Introduction

An increasing number of radio technologies enabling low-power wireless communication over long distances has emerged in the past years. Ultra-narrowband technologies such as Sigfox (Labège, France) and Weightless-N [1] (Cambridge, UK), as well as spread-spectrum technologies such as LoRa [2] (San Ramon, CA, USA), allow for communicating up to few kilometers, and to build up low-power wide area networks (LPWANs) that do not require the construction and maintenance of complex multi-hop topologies [3,4].

A key characteristic of LPWAN technologies is indeed the ability to trade throughput for range and vice versa, i.e., one has the ability to fine-tune physical layer (PHY) settings to select a more sensitive (but slow) configuration that allows communication over a longer distance. This flexibility makes LPWAN technologies particularly appealing to developers of Internet of Things (IoT) applications requiring long-range communications with relatively low data rates. At the same time, however, the ability to fine-tune PHY settings requires a thorough understanding of their impact on network performance, especially on the reliability and energy-efficiency of communications [5].

The research community has recently devoted significant attention to the role of PHY settings in the context of LPWANs [5,6,7], especially LoRa technology. Out of the existing LPWAN technologies, LoRa has especially attracted a large body of work due to the availability of commercial off-the-shelf radio transceiver and platforms [8,9,10], as well as its ability to operate in an infrastructure-free manner and to build up ad hoc mesh networks [5]). LoRa-based networks have been deployed in several settings, ranging from indoor [4] and urban [7] environments, to maritime [11] and mountain scenarios [12]. These deployments have shown the impact of PHY settings on connectivity range and sensitivity [5,13], as well as having given a first impression of the packet reception ratio that can be achieved at different distances with different hardware platforms and physical layer configurations. Bor et al. [14] have also shown through simulation that the choice of the PHY settings affects the number of LoRa nodes that can concurrently access the channel, which has an impact on the scalability of LoRa networks. Furthermore, Bor and Roedig [4] have presented the results of systematic indoor experiments showing that the set of LoRa settings leading to the most energy-efficient operation dynamically changes over time. Based on these results, the authors proposed a protocol that periodically probes different settings and that dynamically picks the ones minimizing energy consumption at run-time.

Interplay between PHY settings and link quality. Although the aforementioned works started to shed light on how to carry out an optimal selection of LoRa’s PHY settings, they all share a common assumption: the best performance is obtained in the presence of highly reliable links. Most works, indeed, specifically target PHY settings maximizing the link quality, i.e., focus on selecting physical layer configurations that allow to sustain a packet reception ratio of 90% or higher [4,12,14] This practice is likely influenced from the behavior of non opportunistic low-power wireless data collection protocols for IEEE 802.15.4 radios, which favor high-quality links to intermediate and lossy ones [15,16]. However, adjusting the PHY settings of the radio to maximize the link quality has important implications w.r.t. energy efficiency when using long-range low-power wireless technologies such as LoRa. Maximizing the link quality, indeed, typically implies an increase in the transmission power and data overhead, and the selection of a more sensitive (and hence slow) physical layer configuration. As a result, one increases not only the likelihood to receive packets, but also the energy consumption of the radio, due to the higher transmission power, and the radio-on time, due to larger PHY layer overhead. This observation raises a yet unanswered question: is it worth selecting PHY settings to reduce the data rate in order to increase the link quality? This question is particularly relevant when two nodes are at the edge of the communication range: should one select a setting that reduces the data rate to increase the robustness of communication (and aim for a link achieving a high packet reception ratio) or rather accept having a link of intermediate quality (i.e., experiencing some packet loss), but with high data rate, and implement a re-transmission scheme on top? How this choice affects the energy-efficiency of the network still needs to be investigated.

Impact of environmental conditions on communication performance. The characteristics of LPWANs make them suitable for outdoor deployments on a large scale, and it is hence important to study in detail the impact of environmental effects such as changes in meteorological conditions, as well as variations in temperature and humidity on network performance. Unfortunately, to date, there is still little understanding about the impact of the environment on the reliability of LoRa communication, especially for links that are at the edge of their communication range. Iova et al. [12] have reported the vulnerability of LoRa communications to environmental factors such as presence of vegetation and temperature variations, but without quantifying their impact. Other works in the low-power wireless community have shown that some IEEE 802.15.4 radios are particularly vulnerable to changes in temperature, and that even the daily fluctuations recorded outdoors can render a good link useless [17,18,19,20]. However, these results are platform-specific and cannot be generalized to LoRa transceivers. Therefore, if and how much temperature affects LoRa’s communication performance is yet to be answered.

Our contributions. In this paper, we carry out an experimental evaluation of the reliability of LoRa in different settings and provide an answer to the aforementioned open questions. First, we study how PHY settings and environmental factors affect the reliability of LoRa communications through an extensive experimental campaign indoor, outdoor, and underground. In line with earlier works [7], our experiments show that PHY settings have a significant impact on packet reception rate and that indoor environments are more challenging for LoRa communications. Our results also suggest that it is better to use faster (but more fragile) settings together with a re-transmission mechanism rather then selecting resilient and slower settings, maximizing packet reception rate and link quality. A detailed study of the overhead of each PHY setting in relation to its improvement on packet reception rate indeed shows that setting a maximizing data rate and minimizing range should be preferred.

Furthermore, our experimental results show a clear correlation between temperature, humidity, packet reception rate, and received signal strength. We hence analyze in depth how environmental factors such as temperature variations affect the reliability of LoRa communications by performing a series of systematic experiments in controlled settings on different hardware platforms. These experiments show that the reliability of LoRa drastically decreases at high temperatures. On the one hand, the signal strength of received packets decreases linearly when temperature increases, as was also observed for a number of IEEE 802.15.4 radios [17,18]. On the other hand, the decrease in signal strength can significantly affect LoRa links that are at the edge of the communication range, increasing packet corruption and loss up to a point in which a link is totally compromised.

The contributions of this paper are hence threefold:

• We study how PHY settings and environmental factors affect the reliability of LoRa through an extensive experimental campaign indoor, outdoor, and underground;
• We analyze the impact of LoRa’s PHY settings on the effective data rate and energy efficiency of communications, highlighting that it is not worth selecting settings to reduce the data rate in order to increase the link quality;
• We systematically study the impact of temperature on the reliability of LoRa communications and show that high temperatures decrease the received signal strength and drastically increase packet loss and corruption for nodes at the edge of the communication range.

The paper proceeds as follows. In the next section, we introduce the reader to long-range technologies and to the LoRa physical layer settings that can be configured to fine-tune the operations of LoRa transceivers. Section 3 highlights the yet open questions with respect to LoRa’s reliability as a function of PHY settings and environmental conditions. In Section 4, we describe our experiments indoor, outdoor, and underground, highlighting the strong impact of the chosen PHY settings and environmental conditions on the reliability of communications. Thereafter, we investigate in detail the interplay between PHY settings and link quality in Section 5 and carry out experiments in controlled settings to quantify the impact of temperature on LoRa’s communication performance in Section 6. We finally summarize our contributions in Section 7, along with a discussion of future work.

2. Primer on LPWANs and LoRa

Low-power wide area networks complement short range wireless technologies such as Wi-Fi, Bluetooth Low Energy, and IEEE 802.15.4, and represent an interesting alternative to cellular technologies for urban-scale IoT applications. The success of LPWANs is due to their ability of providing long-range communication to thousands of devices at minimal cost and limited energy expenditure. Longer communication ranges allow for drastically simplifying duty cycling and networking protocol, as LPWANs can form star topologies where the low-power end devices are able to directly communicate with a more powerful orchestrator. This also allows for designing asymmetric communication schemes and to shift the load to the more powerful central device.

In order to increase the communication range, LPWAN technologies must improve the signal-to-noise ratio (SNR) at the receiver, either by narrowing down the receiver’s bandwidth (reducing the receiver’s noise-floor) or by spreading the energy of the signal over a wider freuency band (effectively reducing the spectral power density of the signal) [5]. NB-IoT [21] and Weightless-P [22], for example, encode the signal in low bandwidth (<25 kHz) to reduce the noise level and keep the transceiver design as simple and cheap as possible. Sigfox [23] and Weightless-N [24] further narrow the signal into ultra-narrow bands as narrow as 100 Hz, further reducing the perceived noise.

LoRa technology. Compared to these technologies, LoRa spreads the signal over a wider frequency band, and is more resilient to jamming and interference. LoRa is a proprietary LPWAN technology from Semtech (Camarillo, CA, USA) that recently attracted significant attention due to its ability to trade efficiently communication range against high data-rates, thus enabling IoT applications at an urban scale. The core of LoRa technology is its Chirp Spread Spectrum (CSS) modulation: the carrier signal of LoRa consists of chirps, signals whose frequency increases or decreases over time. LoRa’s chirps allow the signal to travel long distances and to be demodulated even when its power is up to 20 dB lower than the noise floor. Because of this aspect, carrier sensing in LoRa is quite challenging: LoRa radios allow carrier detection via a CAD mode, a special reception state consuming half of the energy compared to the normal reception mode. However, the signals produced by different LoRa networks operating on different settings could create interference leading to false detections [7].

LoRa’s communication performance can be fine-tuned by varying the selection of several PHY settings, including bandwidth, spreading factor, coding rate, transmission power, and carrier frequency, as summarized in

Table 1. Summary of LoRa’s configurable settings and their impact on communication performance.

Setting	Values	Effects
Bandwidth	$125\cdots 500$ kHz	Higher bandwidths allow for transmitting packets at higher data rates (1 kHz = 1 kcps), but reduce receiver sensitivity and communication range.
Spreading Factor	$2^6\cdots 2^{12}\frac{\mathrm{chips}}{\mathrm{symbol}}$	Bigger spreading factors increase the signal-to-noise ratio and hence radio sensitivity, augmenting the communication range at the cost of longer packets and hence a higher energy expenditure.
Coding Rate	$4/5\cdots 4/8$	Larger coding rates increase the resilience to interference bursts and decoding errors at the cost of longer packets and a higher energy expenditure.
Transmission Power	$-4\cdots 20$ dBm	Higher transmission powers reduce the signal-to-noise ratio at the cost of an increase in the energy consumption of the transmitter.

. We explain next in detail the impact of each PHY parameters on data rate, receiver sensitivity (including resilience to interference), transmission range, and energy-efficiency [25].

Bandwidth (BW). Varying the range of frequencies (bandwidth) over which LoRa chirp spread allows for trading radio air time against radio sensitivity, thus energy efficiency against communication range and robustness. The higher is the bandwidth, the shorter is the air time and the lower is the sensitivity. A lower bandwidth also requires a more accurate crystal in order to minimize problems related to the clock drift. Given a bandwidth BW, typically in the range of 125⋯500 kHz, LoRa’s chip-rate $R_C$ is computed as:

$$R_C=BWchips/s.$$

Spreading Factor (SF). To transmit information, LoRa “spreads” each symbol over several chips (spreading factor) to increase the receiver’s sensitivity even more. LoRa’s spreading factor SF can be selected between 6 and 12, resulting in a spreading rate ranging from $2^6$ to $2^{12}$ chips/symbol and a symbol-rate $R_S$ that can be computed as:

$$R_S=\frac{R_C}{2^{SF}}=\frac{BW}{2^{SF}}symbols/s,$$

and resulting in a modulation bit-rate that can be expressed as:

$$R_M=SF·R_S=SF·\frac{BW}{2^{SF}}bits/s.$$

Note that, in LoRa, packets transmitted with different spreading factors are orthogonal with each other and do not cause collisions if transmitted concurrently.

Coding Rate (CR). To increase the resilience to corrupted bits, LoRa supports forward error correction techniques with a variable number CR of redundant bits, ranging from 1 to 4. The resulting bit-rate BR of LoRa becomes:

$$BR=R_M·\frac{4}{4+CR}=SF·\frac{BW}{2^{SF}}·\frac{4}{4+CR}bits/s.$$

The more interference bursts are expected, the higher the coding rate that should be used to maximize the probability of successful packet reception. Note that LoRa radios with different coding rates can still communicate, since the packet header (transmitted using the maximum coding rate of 4/8) can include the code rate used for the payload.

Transmission Power (TP). As most wireless radios, LoRa transceivers also allow for adjusting the transmission power, drastically changing the energy required to transmit a packet. By switching the transmission power, for example, from −4 to +20 dBm, the power consumption increases from 66 mW to 396 mW when using the RFM95 transceiver (HopeRF, Shenzhen, China) [26]. Note also that, for transmission powers higher than +17 dBm, hardware limitations and legal regulations limit the radio duty cycle to a maximum of 1%.

Carrier Frequency (CF). LoRa transceivers use sub-GHz frequencies for their communication: among others, the 433 MHz, 868 MHz (Europe), and 915 MHz (North America) industrial, scientific and medical (ISM) radio bands. Common LoRa modules such as the Semtech SX1272 [27] and HopeRF RFM95 [26] support communication in the frequency range [860–1020] MHz and are programmable in steps of 61 Hz. Ten channels with different bandwidths can be used to communicate using LoRa in the European 868 MHz ISM band.

3. Related Work

We now summarize the body of works characterizing the performance of LoRa communications and the effects of environmental conditions on its operations.

Characterization of LoRa performance. Because LoRa technology is closed-source, only a few details about its operations are actually available—mostly derived from Semtech’s patent describing the modulation technology or from application notes written to help application designers fine-tune the performance of the transceiver to their needs. Many researchers found this information too limited and started benchmarking and reverse-engineering [28,29] the technology to better understand its mechanism and characteristics.

The first experiments focused on the range of reliable links and on the receiver sensitivity [14]—LoRa’s core characteristics. In [11], LoRa has been evaluated in urban and maritime scenarios, and a signal attenuation model was derived. In [5], instead, experiments focused on testing LoRa’s communication range on a set of diverse scenarios (from underground to overground, with and without line of sight) in order to provide a set of deployment guidelines.

Interestingly, in the evaluation process, different studies found that results were contradicting Semtech’s claims on LoRa performance. In [13], researchers were not able to observe an improved sensitivity with increasing spreading factors. Bor et al. [14] found that LoRa’s ability of penetrating buildings is rather limited compared to what was originally claimed. Similarly, the results that we present in this work show that communication in indoor scenarios with no line of sight are among the most challenging conditions for LoRa. Another challenge is represented by vegetation, as found by Iova et al. [12]. Finally, in [14,30,31], the authors model LoRa self-interference and channel utilization, concluding that LoRa’s scalability is worse than what was originally promised. Other works focus on a more detailed characterization of LoRa, in particular on packet loss [32], on the ability of receiving packets from concurrent transmissions [7], and on the energy consumption at different transmission powers [4].

Different from previous works, this paper analyzes LoRa’s PHY settings from a multi-objective perspective, with the goal of finding the best trade-off between data rate, packet reception rate, and energy efficiency.

Environmental effects on low-power radios. A large body of works has studied the impact of environmental conditions on network performance in low-power wireless radios, especially on IEEE 802.15.4-compliant radio transceivers. Several authors report the impact of meteorological conditions on packet reception, including the impact of weather conditions [20,33,34], and humidity [35], as well as the presence of vegetation [36]. One of the most comprehensive studies on wireless nodes deployed outdoors was carried out by Wennerström et al. [20], who have highlighted that packet reception ratio and received signal strength correlate the most with temperature, whereas the correlation with other factors such as absolute humidity and precipitation is less pronounced.

The strong impact of temperature on communication performance has been confirmed by several other works, also almost entirely focused on IEEE 802.15.4 transceivers. Bannister et al. [19] have shown the correlation between temperature and signal strength in a deployment in the Sonoran desert, and identified in a temperature-controlled chamber that the received signal strength of the TI CC2420 radio attenuates at high temperatures due to the impact of temperature on the radio’s low-noise and power amplifiers. Based upon this work, Boano et al. [17,18] have confirmed these findings also on other platforms such as the TI CC1020 and CC2520, and also highlighted how this can cause a complete disruption of a wireless link. The authors have also shown how the impact of temperature cannot be neglected when designing duty-cycled medium access control protocols for low-power wireless radios [37,38]. To facilitate the study of how temperature affects the operation of low-power wireless protocols on a larger scale than in a temperature-controlled chamber, several low-cost testbed infrastructures have been proposed, the most popular being TempLab and HotBox [39,40].

The impact of environmental conditions on LPWAN radios, instead, has not yet been investigated in detail. Iova et al. [12] have deployed a number of LoRa networks in urban and mountain environments, and reported that environmental factors such as the presence of vegetation and temperature variations can negatively affect communication performance. The authors, however, did not quantify the impact of these environmental factors and their work does not yet clarify whether high temperatures degrade the quality of LoRa links in a similar way as observed on several IEEE 802.15.4 transceiver platforms.

In the remainder of this paper, we conduct a number of experiments to complement the body of aforementioned related works and answer two key questions that are yet open: (i) how does the selection of LoRa’s PHY settings affect the efficiency of links, including the ones of intermediate quality? and (ii) how does temperature affect the performance of LoRa? To answer these questions, we start by carrying out experiments indoor, outdoor, and underground, and by analyzing how PHY settings and environmental factors affect LoRa’s communication performance.

4. Evaluating the Performance of LoRa

To study the reliability and energy-efficiency of LoRa communications as a function of the PHY settings described in Section 2 and as a function of environmental factors, we conduct a series of small-scale deployments.

Experimental setup. All of our experiments are carried out at the Graz University of Technology, Austria: the exact location of the nodes is shown in Figure 1. We fix the senders at three given locations (S) and place three receivers at different distances (1, 2, 3) for three scenarios: indoor with obstacles (i), outdoor with direct line of sight (o), and underground covered by a metal manhole (u). Each transmitter sends a packet with a 5-byte payload every 3 s at transmission power +20 dBm, emulating a timely report of a typical IoT sensor for urban monitoring. Every six minutes, transmitter and receivers reboot and switch to a different setting according to a set of hard-coded combinations shown in

Table 2. LoRa settings used in our experiments: spreading factor (SF), code rate (CR), bandwidth (BW), and rit-rate (BR) Note that settings are ordered by decreasing bit-rate.

Setting ID	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18
SF	7	7	7	9	7	7	9	9	7	9	9	12	9	12	12	12	12	12
CR	4/5	4/8	4/5	4/5	4/8	4/5	4/8	4/5	4/8	4/8	4/5	4/5	4/8	4/8	4/5	4/8	4/5	4/8
BW (kHz)	500	500	250	500	250	125	500	250	125	125	125	500	125	500	250	250	125	125
BR (kb/s)	21.87	13.62	10.93	7.03	6.83	5.47	4.39	3.51	3.41	2.2	1.76	1.16	1.09	0.72	0.58	0.37	0.30	0.18

. For each of the three scenarios (indoor, outdoor, and underground), we test each setting configuration sequentially every six minutes for a duration of 24 h, hence resulting in a total of 1600 packets exchanged per setting.

Hardware. The experiments are conducted using a custom-built platform (see Figure 2) based on the Moteino MEGA (LowPowerLab, Canton, MI, USA) [42]. The latter is equipped with an ATMega1284P microcontroller, and a HopeRF RFM95 LoRa transceiver operating at 868 MHz [26]. The device is powered by a 3.7 V Li-Ion battery with a capacity of 3.4 Ah that can be charged via a dedicated circuit. Without duty cycling the radio, this battery can sustain the device operation for more than 24 h, the maximum duration of our experiments. The platform we have built also embeds sensors to measure changes in the surrounding environment. In particular, temperature and humidity are read from a Bosch BME280 sensor (Gerlingen, Germany) via the I2C interface. For persistent storage, an SD card logs each received packet together with its sequence number, the sensed environmental conditions, as well as the time-stamp provided by a Maxim DS3231 real-time clock (San Jose, CA, USA). We also save the presence of cyclic redundancy check (CRC) errors in the received packets in our traces. This hardware setup was used in our experiments both for senders and receivers.

Metrics. For each 6-min experiment, we compute the packet reception ratio (prr) and the receiver sensitivity, i.e., the lowest signal strength among successfully received packets. We then check the correlation of the computed prr with the employed PHY settings, as well as with the measured temperature, humidity, and received signal strength values.

4.1. Reliability of LoRa as a Function of PHY Settings

Figure 3 shows the packet reception ratio (i.e., the percentage of packets sent that were correctly received) indoor, outdoor and underground for a number of different radio settings (see Table 2). Figure 3 plots all 6-min experiments grouped by setting ID.

Horizontal red lines represent the median, while blue boxes represent the 25th and 75th percentiles. The remaining results are enclosed by vertical dashed black lines while statistical outliers are represented by red crosses. Note that these results were previously presented in [41].

Our results show that LoRa setting ID 11 (i.e., BW = 125, SF = 9, and CR = 4/5) achieves a packet reception ratio above 95% regardless of the scenario and distance between nodes. Nevertheless, setting ID 2 also performs remarkably well: although it sustains a lower prr, it sends packets using a bit-rate that is almost eight times faster than the one used by setting ID 11. This observation will be the starting point of our analysis in Section 5 answering the question of whether it is worth selecting PHY settings that reduce the data rate in order to maximize the link quality.

Differently from previous works, in Figure 3, we also show the distribution of the experiment results rather than just the median and mean values. This allows us to make three observations. First, while the median prr is close to 1 for most settings, the quartiles and minima are not. Second, due to the lower multi-path and fading effects outdoor and underground, LoRa communications are more reliable in these scenarios rather than indoors (in line with what is observed in [7]), with packet reception ratios above 97% for almost all setting IDs. Third, the range of the LoRa radios is consistent throughout the different settings: even though the reception rate changes, all settings are able to deliver packets in similar conditions.

4.2. Factors Affecting LoRa Reliability

We explore next which environmental factors affect the reliability of LoRa. Towards this goal, we use the traces collected in the previous experiments and focus on the correlation between the packet reception rate (prr), setting ID (set), temperature (temp), humidity (hum), spreading factor (sf), coding rate (cr), bandwidth (bw), receiver sensitivity (sens), receiver signal strength (rss), and hour of the day (hour). In particular, we plot the Pearson correlation of each pair of parameters for different experimental scenarios in Figure 4: a value close to 1 (black) means that the two parameters are linearly correlated, whereas a value of 0 (white) implies that the two parameters are independent.

Figure 4 shows that, in all three scenarios (indoor, outdoor and underground), there are some obvious correlations. First (A), the setting ID depends on the bandwidth, coding rate (cr), and spreading factor (sf). This is to be expected, because the setting ID unequivocally describes a combination of these three PHY parameters. Second (B), temperature (temp) is highly correlated with humidity (hum) and both are correlated with the time of the day (hour). This is also an expected correlation, as these environmental factors are highly dependent on the sun exposure. Third (C), the radio sensitivity (sen) is correlated to the received signal strength (rss), since the former is defined as the minimum of the latter. Furthermore, one can also note in Figure 4 that the received signal strength (rss) is correlated with the packet reception ratio (prr) (D), as the LoRa radio is able to successfully decode packets that are above a certain signal-to-noise ratio.

Figure 4 also shows that temperature is tightly correlated with the received signal strength (rss) and the packet reception ratio (prr). This seem to hint that temperature variations may affect the operation of the employed LoRa radio in a similar way as observed on some IEEE 802.15.4 transceivers (see Section 3). When analyzing the figure in detail, one can actually observe a correlation cluster (E) between temperature, humidity, time of the day, packet reception ratio, and received signal strength: the strength of these correlations varies depending on the scenario and is stronger outdoors. To better understand the inter-dependency between the reliability performance (rss and prr) and environmental factors (temp), we carry out experiments in controlled settings in Section 6.

5. The Efficiency of LoRa as a Function of PHY Settings

The experimental campaign presented in the previous section has shown that it is possible to improve the reliability of LoRa by carefully choosing the PHY settings, i.e., some of the settings allow for sustaining a higher prr. In this section, we analyze the costs of such improvement in terms of energy efficiency and analyze in detail the trade-off between packet delivery rate and setting’s bandwidth, providing an answer to the question: is it more efficient to use resilient and slow settings or to use faster (but more fragile) configurations together with a re-transmission mechanism?

To answer this question, we focus on the most challenging scenario in our experimental campaign, i.e., indoor, no line of sight, and with a distance between two devices of 115 m. Figure 5a shows the distribution of packet reception ratios as a function of setting ID. Averages are represented by ’*’, while median, quartiles, and extreme values are enclosed by a blue box and two black bars (outliers are indicated with crosses). PHY settings are ordered by decreasing bit-rate, from faster and more lightweight settings on the left, to slower settings increasing the transceiver’s on-time on the right. As we can see, using the fastest setting (setting ID 1 and 2), the average prr is 80% with a worst case scenario where prr is as low as 20%. As expected, by selecting a PHY configuration that reduces the bit-rate (i.e., by decreasing the bandwidth and increasing the bit redundancy), the packet reception ratio improves, as well as its distribution.

Nevertheless, one can argue that it is more energy-efficient to re-transmit a packet using the faster settings available rather than employing PHY settings that reduce the bit-rate. To prove our point, we compute the expected number of re-transmissions (ETX) as:

$$ETX=\frac{1}{prr}$$

and compare them against each setting’s original bit-rate (BR). As we can see in Figure 5b, the expected number of re-transmissions (squares) does not directly depend on the settings’ bit-rate (triangles), suggesting that not all settings are worth their overhead.

In order to give an indication on how efficiently LoRa settings trade communication efficiency against reliability, we compute the effective bit-rate (EBR) as:

$$EBR[kb/s]=\frac{BR}{ETX}=BR·prr$$

and show both mean and distribution in Figure 5c.

The EBR shows the expected bit-rate of each setting in the case packets are re-transmitted back-to-back until one is successfully received. As we can see from Figure 5c, the mean EBR is by far the highest when the fastest LoRa setting is used (setting ID 1). Setting ID 2, on the other hand, is more consistent, showing a lower variance and the highest minimum (crosses represent outliers).

Even though these results can heavily depend on the surrounding environment, we argue that, in order to maximize the effective bit-rate, one should opt for a re-transmission mechanism and use the settings with sufficient prr (e.g., >0.2) and the highest bit-rate possible.

To test the validity of this claim, we additionally compute the EBR for 12 experiments run by independent researchers [5] and present the results in Figure 6. The experiment was run on several Libelium Waspmote LoRa motes; therefore, the setting ID shown in the figure (mode ID) is enumerated according to the Libelium application programming interface (API). As for Figure 5, we conveniently order the PHY settings starting from the one using the highest bit-rate on the left, to the one employing the lowest bit-rate on the right. As we can see, this set of experiments also confirms our observation: faster settings result in higher bit-rates, even though the quality of the link (i.e., the (prr)) is lower.

Next, we extend this analysis to different transmission power levels and explicitly also evaluate the energy efficiency of LoRa transmissions.

Energy efficiency of LoRa transmissions. We evaluate the energy efficiency of different PHY settings by repeating the indoor experiments without line of sight at a distance of 115 m. This time, we slightly vary the experimental setup as follows: we position the two nodes at the edge of transmission range and send each packet with a different power level, sequentially selected from the set {+20, +17, +15, +11, +7, +5} dBm. We first check if using the fastest PHY setting leads to the highest energy efficiency both in challenging and non challenging conditions (the lower the transmission power, the more challenging the communication). We then compute the most energy-efficient setting by computing the effective energy consumed to send a kilobit of data EKB as follows:

$$EKB[J/kb]=\frac{P}{EBR},$$

where P is the power consumption of the radio in watts and EBR is the effective bit rate.

Figure 7a shows the packet reception rate when using three different power levels: +20, +15, and +11 dBm. As expected, changing the transmission power drastically affects the prr, since the nodes are intentionally placed on the edge of the communication range. In agreement with the results presented previously, Figure 7b shows that the fastest settings are the ones with highest effective bit-rate EBR—independently of the employed transmission power. As lower transmission powers imply lower energy expenditures, we still need to answer which transmission power configuration results in the highest energy efficiency.

Figure 7c shows the energy required by each PHY setting to transmit a kilobit of data (EKB), including the cost of the re-transmissions. As we can see, the most efficient transmission power configuration (i.e., leading to the lowest EKB) is the highest, i.e., +20 dBm. Therefore, our experimental results suggest that, together with the fastest setting, the highest transmission power should be preferred: this combination provides the highest bit-rate EBR and the lowest energy consumption EKB. It is worth highlighting that, in less challenging scenarios in which several transmission powers achieve a prr = 1, the lowest one should be used, as the higher transmission powers may increase the energy consumption without any additional benefit.

6. Impact of Temperature on LoRa Transceivers

Our experimental campaign presented in Section 4 has shown that there is a strong correlation between temperature, packet reception rate, and received signal strength. To quantify this correlation and shed light on the impact of temperature on LoRa communications, we carry out a deeper investigation in controlled settings.

Experimental setup. We use the TempLab testbed [39] to expose a number of LoRa nodes to repeatable temperature variations as shown in Figure 8. The TempLab testbed available at Graz University of Technology has two different types of nodes [38]: LO nodes only heating the sensor nodes above room temperature and PE nodes having the capability to also cool down the node’s temperature below zero degrees thanks to enclosures made of hard Polystyrene foam and ATA-050-24 Peltier air-to-air assembly modules (Custom Thermoelectric, Bishopville, MD, USA). Both LO and PE nodes are remotely controlled using an Aeon Z-Wave Stick Series 2 (Aeon Labs LLC, El Cerrito, CA, USA) sending commands to (i) Vesternet EVR_AD1422 Z-Wave Everspring wireless dimmers (Smartech Holdings Ltd., Manchester, UK) connected to Philips E27 infra-red 100 W light bulbs (Philips, Eindhoven, The Netherlands) and (ii) to Vesternet EVR_AN1572 Z-Wave Everspring on–off wireless switches connected to the Peltier modules.

We place the LoRa nodes without a connected SubMiniature version A (SMA) antenna inside PE nodes and let the nodes transmit packets as fast as possible without any radio duty cycling while temperature varies in the range of 0–60 °C. In particular, we scripted TempLab to first slowly increase temperature from 0 to 60 °C and then to quickly cool down to 0 °C. Each test has a duration of five hours and was repeated for different PHY settings and hardware platforms.

Impact on received signal strength. We plot the relationship between the received signal strength indicator (RSSI) of received packets and the median temperature measured by the two nodes for different hardware platforms. Figure 9 shows the curve recorded when using a Moteino MEGA board [42] equipped with a HopeRF RFM95 LoRa radio (the same platform used for the experiments described in Section 4). Each dot represents the median of the RSSI over 40 received packets. Similarly to what was reported in [18], the RSSI decreases linearly in discrete steps for a total of about 6 dB in the temperature range of 0–60 °C. This is because, for a given voltage, a higher temperature increases the resistance of conductors, while reducing the pass-trough current. For radio transceivers, this implies that higher temperatures reduce the received signal strength and signal-to-noise ratio.

Also according to [18], we observe an hysteresis in the relationship between RSSI and temperature when comparing the curves obtained when heating and when cooling the LoRa nodes. Similarly, Figure 10 shows the relationship between RSSI and temperature recorded when exposing ST Nucleo L073RZ boards (STMicroelectronics, Geneva, Switzerland) [9] equipped with a Semtech SX1272 radio to temperature variations. Also using this hardware, we observe a linear decrease of about 6 dB in the RSSI at high temperatures. We attribute the spikes recorded on the experiment of Figure 10 (when temperature varies between 30 and 40 °C) to a temporary multi-path fading effect of the environment. Note that Figure 9 and Figure 10 refer to different experiments, both carried out using setting ID 6 (i.e., CR = 4/5, SF = 7, BW = 125 MHz).

Remarkably, the attenuation of received signal strength caused by an increase of temperature in the range [0–60] °C is comparable to the change in sensitivity that can be observed when switching from the fastest to the slowest PHY setting [14]. Therefore, in case of cold temperatures, it may even be possible to avoid using extremely slow radio settings by carefully deploying the LoRa devices in locations that are not directly exposed to sunlight.

Impact on packet reception ratio. We further analyze the effects of temperature variations on nodes that are at the edge of their communication range. We intentionally place two nodes at the limits of their communicating range and slowly change the temperature of the transmitter from 15 to 60 °C and quickly back to 15 °C. Figure 11 shows the distribution of lost, corrupted and successfully received packets for every minute in a 75-min experiment. We can observe that what was a perfect link at minute 0 (100% prr at 15 °C) slowly becomes unusable at higher temperatures. As soon as temperature (red line) starts to increase, either packets are received, but their content is corrupted, or the radio was unable to receive the packet at all. Once temperature starts decreasing again, the link is restored and sustains a high delivery rate. These experiments confirm results from previous studies on specific IEEE 802.15.4 radios [18,19], and show that temperature can drastically affect packet delivery. An important takeaway message is that LoRa nodes employing the radio transceivers used in our experiments should be deployed during the warmest time of the day or year, to ensure that network performance is sufficient throughout the system lifetime, and that nodes should be shielded from sunlight if possible.

7. Conclusions

This paper presents an analysis of the performance of LoRa as a function of different PHY settings and environmental conditions. We first study the effects of different LoRa settings on the effective bit-rate that can be achieved (i.e., on the amount of information that LoRa is able to successfully deliver during a given period). Our experimental results suggest that, when nodes are at the edge of their communication range, using the fastest PHY setting and the highest transmission power is more efficient than selecting slower settings that maximize the link quality. Even though, for example, the fastest PHY setting in our experiments yields an average packet reception rate that is 10% lower than the slowest setting, the former’s effective bitrate is 100× faster than the latter’s. Compared to the slower settings, the efficiency of the fastest PHY setting is so high that even in its worst case scenario—when the minimum prr reaches 20%—the effective bitrate is faster than twelve of the slowest PHY settings (settings 7–18, from 1× to 25× better). Second, we analyze the external factors affecting the reliability of LoRa. Our outdoor experiments show a clear correlation between temperature, humidity, packet reception rate, and received signal strength. A deeper investigation in controlled settings shows that the signal strength of received packets decreases linearly when temperature increases in two different LoRa transceivers. Different LoRa radios have shown that, over a range of 60 °C, the received signal strength is consistently reduced by 6 dBm (1 dBm/10 °C). This decrease in signal strength can significantly affect LoRa links that are at the edge of the communication range, increasing packet corruption and loss, and rendering a perfectly good link (100% prr at 15 °C) completely unusable (0% prr at 60 °C).

As a future work, we plan to quantify the impact of other environmental factors on LoRa performance, e.g., humidity and radio interference. Our ultimate goal is to design and implement an environmental-aware MAC protocol tailored to LoRa that can sustain reliable and energy-efficient operations regardless of changes in the surrounding environmental conditions.