Short-Term Field Evaluation of Low-Cost Sensors Operated by the “AirSensEUR” Platform

Abstract

Electrochemical low-cost sensors, suitable for the monitoring of different air quality parameters such as carbon monoxide or nitrogen dioxide levels, are viable tools for creating affordable handheld devices for short-term or dense air quality monitoring networks for long-term measurements and IoT applications. However, most devices that utilize such sensors are based on proprietary hardware and software and, therefore, do not offer users the ability to replace sensors or interact with the hardware, software, and data in a meaningful way. Initiatives that focus on an open framework for air quality monitoring, such as the AirSensEUR project, offer competitive open source alternatives. In this study, we examined the feasibility of the application of such devices. Five AirSensEUR units equipped with chemical sensors were placed next to a reference air quality measuring station in Vienna, Austria. During co-location, concentrations of 0.20 ± 0.06 ppm, 7.14 ± 8.66 ppb, and 17.58 ± 9.90 ppb were measured for CO, NO, and NO₂, respectively. The process of evaluating the performance of the low-cost sensors was carried out and compared to similar studies. Data analysis was carried out with the help of the basic functions in MS Excel. We investigated the linear correlation between the sensor and reference data and thus calculated the coefficient of determination, the average and maximum residuals, and the correlation coefficient. Furthermore, we discuss sensor properties in regard to selectivity and long-term stability.

1. Introduction

In the past, several studies have shown the importance of air quality for the well-being and health of humans [1,2,3] and the challenges that come with the assessment of air pollutants. Since in some areas of the world, the information on air quality is either highly sparse or non-existent due to the high cost of traditional monitoring stations [4], the monitoring of outdoor and indoor air quality in urban areas through low-cost sensor networks has received increasing interest by manufacturers, communities, and scientists alike [4,5,6]. Triggered by this development, smart home devices that make use of such sensors have entered the market, and different OEMs offer various solutions for how different types of sensors can be operated and how recorded data can be stored, accessed, and visualized [7]. On the one hand, there are commercial products that provide users with relatively easy-to-use devices and software that, most of the time, lacks the capability of hardware calibration but provides a direct output of concentration values in parts per million or similar via calibration by the manufacturer. On the other hand, open source projects, such as “AirSensEUR”, were founded to provide cost-effective, transparent, and sustainable frameworks and devices for operating various types of low-cost sensors [8], with the main drawback being that users need at least basic skills in electrical and software engineering to utilize the equipment. Aside from this, usually, calibration through co-location and the application of at least basic mathematical functions are needed to derive the correct output of concentration levels [9,10].

During the development of the AirSensEUR project, several sensors had already been reviewed and/or tested and evaluated, and mathematical functions were established to increase the accuracy and reproducibility of measurements. At the same time, the development of software and hardware that are specifically made for the application of mathematical functions and formulas to make complex data processing accessible to end users is still ongoing [11]. Since most previous evaluations have been carried out by developers and experts in the field of air quality monitoring and electrical engineering [10,12], a user’s perspective is valuable to evaluate the status of the accessibility of the framework and highlight problems that occur in the process.

We purchased five pre-built prototypes of AirSensEUR devices in late 2019 and placed the devices next to a municipal air quality reference measuring station in September 2020 in Vienna, Austria. We evaluated the performance of the selected sensors based on the software provided by the AirSensEUR project or through simple data analysis with MS Excel.

Electrochemical Sensors

Next to optical sensors that, for example, aim at the measurement of different fractions of particulate matter, electrochemical sensors for the detection of gaseous pollutants represent the most commonly used sensors in IoT devices such as “AirSensEUR”. In general, these low-cost sensors are based on the principle of gas passing through a permeable membrane (filter) and creating a reaction in an electrochemical cell that mainly consists of an electrolyte and working, counter, reference, and auxiliary electrodes (newer versions), with a cost of around EUR 150–500 per sensor.

The working electrode is the site for either the reduction or oxidation of the chosen gas species and is generally coated with a catalyst that provides a high surface area and is optimized to promote the reaction with the gas of choice. Through the reaction, the electronic charge is generated at the working electrode, which is then balanced by a reaction at the counter electrode, ultimately leading to an electric current, which is the measurable output signal of the sensor. The reference electrode is used to maintain the working electrode at a fixed potential, and the auxiliary electrode works as a second working electrode, which has no contact with the target gaseous pollutants and therefore generates a background current related to the changes in the environmental conditions, which is used to correct the working electrode that is in contact with the target pollutants [4,13].

Since the target gaseous pollutants enter the electrochemical cell only by diffusion, the sensors are designed in such a way that the rate of diffusion to the sensor is lower than the rate of reaction with the working electrode. This leads to sensor output that is directly proportional to the concentration of the target pollutant.

Table 1. Examined air pollutants.

Carbon monoxide (ppm)	CO	Alphasense CO-A4 [14]
Nitrogen monoxide (ppb)	NO	Alphasense NO-B4 [15]
Nitrogen dioxide (ppb)	NO2	Alphasense NO2-B43F [16]

shows the sensors that were examined in this study together with the parameters that were recorded by the reference measuring station.

Commonly known advantages of electrochemical sensors are their low manufacturing cost, linear output, good resolution and repeatability, low power consumption, and small form factor. The disadvantages include a narrow temperature range due to the sensitivity to temperature, cross-sensitivity to other gases, and a limited and quite short shelf life that depends on the target gas and the environment the sensors are used in [4,6].

Table 2. Impact of co-pollutants on sensor signal when measuring a pollutant mix with pollutant concentrations typical for European suburban levels, expressed in percentages. Adapted from Lewis et. al. (2016) [10].

	CO	O3	SO2	CO2	NO	NO2
NO-B4	0	−34.12	0.14	−985.32	-	−415.71
NO2-B4	0	0	0	118.94	−20.61	-
CO-A4	-	1.4	−0.01	0	0	0.40

and

Table 3. Cross-sensitivity (in ppb/ppb), estimated from Lewis et al. (2016). Adapted from [17].

	CO	O3	SO2	CO2	NO	NO2
NO-B4	0	−0.020	0.013	3.2 × 10−5	-	−1.057
NO2-B4	0	0	0.027	0.15	−0.054	-
CO-B4	-	−0.053	−0.034	0	0	0.085

show the most important and quite well-understood interfering co-pollutants in regard to ambient air in suburban areas according to Lewis et al., 2016. The observed ppb per pollutant rates were 106 ± 24 for CO; 0.2 ± 0.1 for SO₂; 1.3 ± 7.2 for NO; 23.6 ± 12.3 for O₃; 5.1 ± 0.2 for NO_2; and 389 ± 24 (ppm) for CO₂ [10,17].

To reduce the influence of meteorological parameters and co-pollutants and to increase selectivity, different approaches using the hardware and software have already been established.

In the case of meteorological parameters, mathematical methods have been developed [11,18] to correct for their influence on sensor data, which can be integrated into software for data treatment. Similar things can be done for the consideration of co-pollutants, as shown by Lewis et al. [10], with the drawback being that universal correction factors could be influenced by the concentration range of reference data, and sometimes the influence of co-pollutants could be higher than the actual sensor reading of the target gas [10]. A second approach, which is, for example, also followed by the “AirSensEUR” project, is the process of co-locating sensors to reference devices and treating the collected data with an algorithm that is designed to correct the sensor output for the influence of meteorological parameters and co-pollutants, with the drawback that co-location might not be viable for users. Other studies have also shown the feasibility of machine learning [10,19,20,21], with the machine learning method outperforming other calibration models, such as univariate linear regression and multiple linear regression [19], and the potential to overcome long-term sensor drift effects to enable repeated deployment of the sensors [20].

However, there are also significant improvements and optimizations on the hardware side. Modern electrochemical sensors include filters to protect the sensor from dust and water and prevent the access of interfering gas to the electrochemical cell to increase the selectivity of the sensor [13]. There is also ongoing development regarding improving the selectivity and long-term stability of the (catalytic) sensor material, as shown in [22,23]. According to the work of Liu et al. [22], 2D nanomaterials (graphene, MoS₂, BN, MXenes, phosphorene, etc.) as the sensing layer, (working electrode) show superior performance at room temperature in comparison to traditional metal oxide semiconductors, which would eliminate the error caused by temperature in available electrochemical sensors. Nevertheless, the manufacturing costs for 2D nanomaterials are still high due to their lack of high yield and efficient engineering processes. In addition, 2D nanosheets of metal oxides show increased robustness against temperature changes but still are not able to offer satisfying selectivity in comparison to other 2D nanomaterials.

2. Materials and Methods

2.1. AirSensEUR Devices

AirSensEUR (www.airsenseur.org, accessed on 30 June 2022) is an open framework focused on air quality monitoring at low concentrations using low-cost sensors and is composed of electronic boards (shields) and necessary hardware, firmware, and software applications. The platform, developed by the Joint Research Center (JRC; the European Commission’s science and knowledge service), among others, offers multiple possibilities in regard to sensor combination by providing the needed electronics (power supply, reference voltage, etc.) and connections on different electronic boards. Each device consists of a host that is able to control several sensor shields through a sensor bus, send commands to sensor shields, and retrieve sensor data [8]. By assuring interoperability and compliance with the INSPIRE directive, the capacity to work as a node within a network of multi-sensors is given [24].

However, since only specific shields and sensors have already been evaluated in previous studies, a standard kit of four chemical sensors, as recommended by the developers, was purchased and mounted in the ASE boxes.

The chemical sensors were plugged into the respective AirSensEUR shields that provide digital sensor signal output between 0 and 65,535, which corresponds to a 16-bit analog to digital conversion [25]. The digital values can be converted to voltages using the configuration parameters that are set by the user or given by the manufacturer and the help of Equation (1) [11].

Table 4. Parameters for configuration of chemical sensors in use.

	CO	NO	NO2
Ref-(V)	1.501	1.2	1.701
RefAD (V)	0.501	0.5	0.501
Board.Zero	1.10014	0.8315	2.151
Gain	1	1	2
Rload	50	50	50
RefAFE (V)	1.642	1.663	4.302
InternalZero	67	50	50

shows the parameters that were used in this study. Typically, these parameters and values have to be evaluated for each newly installed sensor type before measurements can be carried out. In this case, the already set and proven values by the developers were used to run the sensors, and the co-location and comparison of the sensor and reference data were used for the conversion of digital into concentration values.

$$V=\frac{\left(Ref-RefAD\right)+\left(Digital+1\right)\times 2\times RefAD}{2^{16}}$$

(1)

2.2. Sensor and Reference Data

From 14 September 2020 to 21 September 2020, five AirSensEUR units, equipped with sets of three sensors, where deployed at the reference measuring point “Taborstrasse”, managed by the “Municipal Department 22—Environmental Protection, MA22” in Vienna, Austria [26]. For data analysis and evaluation, 30-min mean values, calculated from the data recorded in intervals of 1 min, were used as the reference data. Additionally, 30-min mean values of the sensor data recorded in intervals of approximately 1 min were used.

Table 5. Equipment used at the reference measuring station.

Parameter	Product Description	Gas Flow Rate	Reproducibility	Linearity
CO	Horiba APMA-370 [28]	1.5 L/min	±1%	±1%
NO and NO2	Horiba APNA-370 [29]	0.8 L/min	±1%	±1%

shows the equipment that was used at the reference measuring station that was checked for accuracy every 24 h with the help of a multi-point ASGU-370 calibration unit by Horiba Ltd. (Kyoto, Japan) [27].

Table 6. Average concentrations, temperature, and humidity of the reference data.

			Temperature	Humidity
CO	0.11–0.51 (Ø = 0.20 ± 0.06)	ppm	11.1–30.6 (Ø = 20.6 ± 4.9) °C	29–82 (Ø = 54 ± 12.8)%
NO	0–64.93 (Ø = 7.14 ± 8.66)	ppb
NO2	2.43–50.15 (Ø = 17.58 ± 9.90)	ppb

shows the range and average of the recorded reference data set with the corresponding ranges of temperature and humidity that were present during co-location.

2.3. Software and Data Analysis

The AirSensEUR project offers developed software in the form of an application written in the programming language “R”, which can be used for the processing of recorded reference and sensor data and is available via GitHub [30]. Users have to go through the quite tedious installation process by installing the development environment for the programming language, RStudio, and additionally needed packages and have to load the code into the program to compile and run the app [11]. When complete, configuration data can be loaded and sensor data can directly be downloaded from the server, where recorded data are stored using InfluxDB, by using the graphical interface of the app. In the next step, the reference data, which are used for calibration, have to be loaded and processed. The app offers different possibilities to load these data into the program memory. We chose to provide our data as an offline * CSV file. However, we were not able to successfully load our sensor and reference data into the application and, therefore, tried to troubleshoot with the help of the developers. Unfortunately, the problems could not be resolved in the set time frame and it was decided to evaluate the data with the help of basic functions in MS Excel instead.

In succession, the digital raw data from the A/D converter were plotted against the reference data in ppb/ppm for each sensor, and the following parameters were determined: linear regression, coefficient of determination, Pearson correlation coefficient, and residuals.

2.3.1. Linear Regression and Coefficient of Determination

Linear regression analysis (method of least squares) was carried out with the help of the linear fit feature in MS Excel. The principle of the calculations is shown in Equations (2)–(4). We calculated the 30-min mean values for the sensor data and compared them to the 30-min mean values of the reference data.

$${\displaystyle \sum }_{i=1}^n{(y_i-a\times x_i-b)}^2\underset{}{\to }minimum$$

(2)

$${\displaystyle \sum }_{i=1}^n{\left(y_i-{\mathrm{ŷ}}_i\right)}^2\underset{}{\to }minimum$$

(3)

$$\mathrm{ŷ}=a\times x+b$$

(4)

To rate the linear regression for each sensor, the coefficient of determination (R²) was calculated with help of the function “=RSQ()” in MS Excel. The mathematical principle of the calculation is shown in Equation (5) and the results for R² are values between 0 and 1, where 1 is interpreted as a perfect correlation between the two input data sets and 0 is interpreted as no correlation.

$$R^2=\frac{{{\displaystyle \sum }}_{i=1}^n{\left({\mathrm{ŷ}}_i-\mathrm{Ȳ}\right)}^2}{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-\mathrm{Ŷ}\right)}^2}0\le R^2\le 1$$

(5)

2.3.2. Pearson Correlation Coefficient

The correlation between the sensor and reference data was determined with the Pearson correlation coefficient (r_xy) and the help of the function “=PEARSON()” in MS Excel. The mathematical principle of the calculation is shown in Equation (6). The resulting values vary between −1 and 1, where −1 is equivalent to a perfect negative linear correlation, 0 means no correlation, and 1 is equivalent to a perfect positive linear correlation.

$$r_{xy}=\frac{{{\displaystyle \sum }}_{i=1}^n\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{\sqrt{{{\displaystyle \sum }}_{i=1}^n{(x_i{}^2-\overline{x})}^2}\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left(y_i-\overline{y}\right)}^2}} 1\ge r_{xy}\ge -1$$

(6)

2.3.3. Average and Maximum Residuals

With the help of the equations derived from the linear fit, the residuals were calculated for each datapoint.

Example:

y = ax + b, with a = −0.0555 and b = 3437.5

For the raw sensor data, x = 61,441.989. This results in y = 27.469 ppb, compared to the recorded real value of y = 25.08 ppb.

Therefore the residual for x = 61,441.989 results in: Δy = y − y = −2.389 ppb.

Afterwards, the average and maximum absolute residuals for each sensor were found with the help of the =AVG() and =MAX() function in MS Excel, and the average deviation as a percentage (%) was calculated.

3. Results

The results derived from this study mainly consist of the calculated linear equation, the coefficient of determination, the linear correlation, residuals, and diagrams for each investigated sensor.

3.1. Nitrogen Dioxide (NO₂)

The examined NO₂ sensors showed a quite significant negative correlation to the reference data, with r_xy ranging from −0.6721 to −0.8622, but they also showed significant deviation in the form of high average residual values ranging from 4.8903 to 6.0326 ppb and maximum residuals from 30.055 to 33.313 ppb. The plotted data (Figure 1) suggest that the deviation increased with rising concentrations.

Table 7. Calculated values for data analysis of NO₂ sensors.

	290 A	291 A	2921	458 A	4583
Slope	−0.0555	−0.0498	−0.0238	−0.1069	−0.0862
Intercept (ppb)	3437.5	3074.8	1453.7	6650.8	5348.5
R2	0.6438	0.5724	0.4157	0.6734	0.7434
rxy	−0.8024	−0.7566	−0.6721	−0.8206	−0.8622
Δymax (ppb)	32.969	32.558	30.551	30.055	33.313
Δyavg (ppb)	4.8903	5.348	6.0326	4.9808	4.2677
Δyavg (%)	28.09	30.70	35.24	39.75	24.98

shows the calculated values for the NO₂ sensors.

3.2. Nitrogen Oxide (NO)

The examined NO sensors showed a weak correlation with the reference data, with r_xy ranging from 0.3342 to 0.3961, significant deviation with the high average residual values ranging from 65.184 up to 69.275%, and maximum residuals from 52.262 to 55.833 ppb. The plotted data (Figure 2) show the presence of outliers, which leads to high uncertainty when predicting the concentration values of NO.

Table 8. Calculated values for data analysis of NO sensors.

	290 A	291 A	2921	458 A	4583
Slope	0.0093	0.0051	0.0059	0.0051	0.0055
Intercept (ppb)	−79.714	−43.078	−49.794	−42.119	−46.688
R2	0.1569	0.1117	0.1186	0.1133	0.1319
rxy	0.3961	0.3342	0.3444	0.3366	0.3632
Δymax (ppb)	52.888	55.833	55.262	54.88	54.755
Δyavg (ppb)	4.7318	4.854	4.8199	5.0138	4.7744
Δyavg (%)	65.184	69.275	68.432	69.046	67.5

shows the calculated values for the NO sensors.

3.3. Carbon Monoxide (CO)

In comparison to the other sensors, the CO sensors showed the strongest linear correlation with the reference data, with r_xy ranging from 0.9315 to 0.9549, R² ranging from 0.8676 to 0.9119, a maximum residual from 0.2418 to 0.2504, and an average residual ranging from 8.0213 to 10.453%. The plotted data (Figure 3) shows the quite high correlation.

Table 9. Calculated values for data analysis of CO sensors.

	290 A	291 A	2921	458 A	4583
Slope	0.0001	0.0001	0.0002	0.0002	0.0002
Intercept (ppm)	−0.8022	−0.8623	−0.969	−0.9736	−1.0046
R2	0.9108	0.8676	0.8757	0.9104	0.9119
rxy	0.9544	0.9315	0.9358	0.9542	0.9549
Δymax (ppm)	0.2499	0.2479	0.2504	0.2418	0.245
Δyavg (ppm)	0.0165	0.0195	0.0202	0.0179	0.0176
Δyavg (%)	8.0213	9.4142	10.453	9.1793	9.0554

shows the calculated data for the CO sensors.

4. Discussion

Our results show that in the case of the NO₂ and the CO sensors, a quite high correlation (R²(NO₂) = 0.6169, R²(CO) = 0.8953) with the reference data was achieved without any data treatment. Similar studies, such as [31], indicate that our findings are in the expected range or, as in the case of NO₂, show even better performance of the sensors without further data treatment. For the NO sensors, we achieved a slightly lower R² value of ~0.13 compared to the already low 0.18 value found in [31].

Table 10. Comparison of results.

Parameter	Our Results *	E. S. Cross et al. [31] **
	R2	R2
NO2	0.62	0.21
NO	0.13	0.18
CO	0.90	0.78

* 30-min mean values, n = 325. ** 5-min mean values, n = ~30,000

shows the calculated R² values from our study in comparison to other results.

Since there are several meteorological parameters, such as temperature, humidity [32], and other pollutants such as O₃, that lead to false sensor readings and are even able to totally exceed the real influence of the measured gas, for example, as shown in [10,12,31], a relatively high deviation between different studies is expected. In addition, a quite significant drift of the sensor output over time has to be considered. Furthermore, the usage of complex calibration models or algorithms, machine learning, and complex data treatment is considered state-of-the-art, which usually leads to much higher correlations between the true reference data and the (predicted) sensor data [7,19]. To achieve the best results with these techniques, the right combination of sensors also has to be considered. If, for example, O₃ has a strong influence on the NO₂ sensor, the levels of O₃ have to be measured so that data can be fed into the respective model for data treatment. Since we were not able to utilize the algorithm offered within the AirSensEUR project yet, we were not able to analyze our data in more detail, and only a comparison to the simple linear correlation data was viable. In conclusion, it is important to not only compare the raw sensor data to the reference data but to compare data that were treated with software or models that are tied to the respective hardware framework. However, as pointed out in another study [7], even with sophisticated statistical analysis, the comparison of data models with data that aims to compare sensor to reference data is complicated and most of the time does not lead to conclusions. Factors such as the geographical location, weather situation, recording intervals, measurement range, and others have a strong impact on the collected data and the statistical characteristic of the data series [4,7,10,17].

In general, there is a need for easy-to-use, open, and standardized tools for the comparison and validation of sensor data since the impact of air quality on human health is evident, and smart homes should provide correct and validated data accordingly. In conclusion, manufacturers of smart home or IoT devices should offer transparency and cross-platform compatibility and interoperability throughout the whole process of calibration, data handling, and evaluation. The fact that interoperability seems to be a general issue with smart home devices [33,34,35] leads to the conclusion that efforts have yet to be intensified before electrochemical low-cost sensors can be applied in a meaningful way. In regard to the AirSensEUR project, we did not encounter any meaningful problems when handling the pre-assembled hardware (prototypes), but despite it being open source software, we encountered several issues with the included software application and were not able to use it accordingly. Therefore, in addition to transparency, easy-to-use software based on universally acknowledged data analysis models has to be offered to gain the interest and trust of smart home users.