On the Importance of Data Quality Assessment of Crowdsourced Meteorological Data

Round 1

Reviewer 1 Report

This manuscript uses crowdsourced weather observations (by Netatmo sensors), which is a very promising data source in urban weather/climate related fields. It is well written and it is easy to follow, although the figures are in general too small. The visualization tool seems interesting to have a fist overview on the data quality.

Netatmo data can be very useful as it provides a good coverage in urban areas. On the other hand, its relatively low level of reliability might cause problems if no quality control is previously done. As the authors mention, such problems are mainly due to the erroneous placement of the sensors, that affect both temperature and relative humidity measures. It is important, thus, to conclude about the quality of the data and what are the factors that might be deteriorating it. Or, if the idea is to use the data operationally, namely as data to assimilate in weather forecasting systems, then a fast but reliable way to filter the data is needed – a “buddy check” and/or a comparison between standard deviations can be a way of achieving the goal.

Designing methodologies to better perform a quality control of crowdsourced weather observations is a very a hot topic on weather forecast and, despites a couple of methodologies have been defined, there is room for improvements. This study presents a superficial analysis of the data, where no recommendations on how to filter/correct the data are given or on what aspects should one consider before using this kind of data. I recommend the authors to read the following papers: Nipen et al. 2020, Sgoff et at., 2022) and check the literature on TitanLib to be aware of the existent methodologies for quality control of weather data, including from Netatmo sensors.

Author Response

This manuscript uses crowdsourced weather observations (by Netatmo sensors), which is a very promising data source in urban weather/climate related fields. It is well written and it is easy to follow, although the figures are in general too small. The visualization tool seems interesting to have a fist overview on the data quality.

Answer: First, we would like to thank the Reviewer for the time and effort put into reviewing our manuscript.

We appreciate your comments and we have tried to address all the points raised to the best of our abilities. In respect to the comment raised above, in the revised version we made sure that the figures are presented in a larger format, thus hopefully resolving any legibility issues. This specifically concerns figures 2 through 7.

Netatmo data can be very useful as it provides a good coverage in urban areas. On the other hand, its relatively low level of reliability might cause problems if no quality control is previously done. As the authors mention, such problems are mainly due to the erroneous placement of the sensors, that affect both temperature and relative humidity measures. It is important, thus, to conclude about the quality of the data and what are the factors that might be deteriorating it. Or, if the idea is to use the data operationally, namely as data to assimilate in weather forecasting systems, then a fast but reliable way to filter the data is needed – a “buddy check” and/or a comparison between standard deviations can be a way of achieving the goal.

Answer: We are in an absolute agreement with the above observation. Although the potential of crowdsourced data is theoretically immense, if the data quality issues are neglected the integrity of the data is negatively affected. We equally agree with the notion of a “buddy check”, which we have tried to achieve through a comparison with near-by authoritative observations. The authoritative weather stations employed for the purpose abide to the high quality standards – from the installation principles that follow the WMO standards, to the data quality measures applied to a number of intermediate steps (from the raw data acquisition to the data storage and re-use stages). These aspects are indeed stressed at the beginning of the subsection 2.1, while further statements were added to reinforce this position.

However, to further strengthen the integrity of such an approach, we added some relevant parallels to the existing methodologies that originated from the suggested resources you have provided. We specifically found the work of Nipen et al. 2020 to be quite an interesting and highly relevant read. Please refer to the changes made in the newly added subsection 2.2.

Designing methodologies to better perform a quality control of crowdsourced weather observations is a very a hot topic on weather forecast and, despites a couple of methodologies have been defined, there is room for improvements. This study presents a superficial analysis of the data, where no recommendations on how to filter/correct the data are given or on what aspects should one consider before using this kind of data. I recommend the authors to read the following papers: Nipen et al. 2020, Sgoff et at., 2022) and check the literature on TitanLib to be aware of the existent methodologies for quality control of weather data, including from Netatmo sensors.

Answer: We appreciate your comment and recognize the benefit of providing more references to the available quality control methodologies. We equally find the suggested resources excellent and relevant to our work, so we are more than happy to reflect on these studies. Please refer to the Introduction section and the related revised content.

In contrast to the existing vast scientific body of work that thoroughly covers methodologies for quality control of weather data, in our work we aimed to highlight systematic quality issues that are integral to the Netatmo crowdsourced data and the resulting implications that arise when such data is used for empirical research and numerical simulation, specifically when data quality issues are not being resolved. We have also noted that related work oftentimes do not focus on all nuances of data quality issues, hence our intention was to deepen the discussion in this regard. Hence, we decided to look at several aspects and questions as presented at the outset of the manuscript (please refer to the last paragraph of the Introduction section). In order to bring more clarity to our intention, however, this statement is now included in the newly added subsection 2.2.

Author Response File: Author Response.pdf

Reviewer 2 Report

This study analyzes the biases in crowdsourced meteorological data collected by "unofficial" observational stations, using "authoritative" station data as the truth. This is an interesting study and a useful contribution to citizen science. In recent years, increasing amount of environmental observations have been produced by private volunteers. While the new data are potentially useful, the issue of quality control for crowdsourced data has not been systematically analyzed. This paper provides a good framework for addressing this issue. The paper is well-written and the technical details are clearly explained. I have only a few relatively minor comments, mostly regarding the interpretations of results:

 

(1) Let's not forget that the observations made by "authoritative" stations could also have biases of their own. In this study, they are regarded as truth while the difference between "crowdsourced" and "authoritative" data is attributed entirely to the bias in the former. To be more objective, we could perhaps consider the following alternative interpretation: Suppose that the level of uncertainty (i.e., the "error bar") in the authoritative data is known, then one can claim that the crowdsourced data has a significant bias only if it lies outside the error bar (say by one or two standard deviation) of the authoritative data. In this context, do we have any information on the estimated error bars for the authoritative data for a quantitative comparison?

 

(2) This study demonstrates a potentially useful framework of bias correction for the crowdsourced data. Nevertheless, with only one year of observation, I wonder if it is sufficient to establish the systematic bias in the crowdsourced data? (The common practice would be to average over multiple annual cycles in order to remove random inter-annual variability.) More generally, the bias has "systematic" and "random" components which are not yet clearly separated in this study. As such, the "bias correction" inadvertently also subtracts out the random component. (Certainly, this situation can be improved in future studies, when data from multiple years are used.)

Author Response

This study analyzes the biases in crowdsourced meteorological data collected by "unofficial" observational stations, using "authoritative" station data as the truth. This is an interesting study and a useful contribution to citizen science. In recent years, increasing amount of environmental observations have been produced by private volunteers. While the new data are potentially useful, the issue of quality control for crowdsourced data has not been systematically analyzed. This paper provides a good framework for addressing this issue. The paper is well-written and the technical details are clearly explained.

Answer: First, we would like to thank the Reviewer for the time and effort put into reviewing our manuscript. We are in an absolute agreement with the above observation.

I have only a few relatively minor comments, mostly regarding the interpretations of results:

• Let's not forget that the observations made by "authoritative" stations could also have biases of their own. In this study, they are regarded as truth while the difference between "crowdsourced" and "authoritative" data is attributed entirely to the bias in the former. To be more objective, we could perhaps consider the following alternative interpretation: Suppose that the level of uncertainty (i.e., the "error bar") in the authoritative data is known, then one can claim that the crowdsourced data has a significant bias only if it lies outside the error bar (say by one or two standard deviation) of the authoritative data. In this context, do we have any information on the estimated error bars for the authoritative data for a quantitative comparison?

Answer: We appreciate your comment and understand your concern. Indeed, we agree that we have taken the authoritative sources as the absolute possible truth, which is a very strong statement. To support such a position, we have reflected on the fact that the authoritative weather stations employed for the purpose abide to the high quality standards – from the installation principles that follow the WMO standards, to the data quality measures applied to a number of intermediate steps (from the raw data acquisition to the data storage and re-use stages). These aspects are indeed stressed at the beginning of the subsection 2.1. We have further added another statement about the certification of these products according to the ISO 9001 standard. Please refer to the subsection 2.1.

However, we equally see the benefit of introducing some flexibility to such a position. Hence, reflecting on an alternative interpretation, as suggested in the comment above, may add to the scope of scientific objectivity. We have therefore added such a statement in the subsection 3.2 (where discrepancies between PWS and authoritative data are discussed). We hope that such a statement successfully introduced some level of flexibility and a possibility that there may still be a certain level of uncertainty (i.e., the "error bar") attributed to the authoritative data. Unfortunately, the information about an actual possible estimated error bar for the authoritative data is not known at this time. However, we have equally reflected on such aspects not being taken into account in our study, due to the missing background information. This may be understood as a limitation of the current assessment to which we have raised attention, so as to achieve scientific objectivity. Nevertheless, we hope that the Reviewer can still find the merit in our contribution.

• This study demonstrates a potentially useful framework of bias correction for the crowdsourced data. Nevertheless, with only one year of observation, I wonder if it is sufficient to establish the systematic bias in the crowdsourced data? (The common practice would be to average over multiple annual cycles in order to remove random inter-annual variability.) More generally, the bias has "systematic" and "random" components which are not yet clearly separated in this study. As such, the "bias correction" inadvertently also subtracts out the random component. (Certainly, this situation can be improved in future studies, when data from multiple years are used.)

Answer: We agree that this is an important aspect to consider. We have therefore introduced another subsection in our revised manuscript that discusses a multiannual analysis for an exemplary use case. Please refer to subsection 4.2 within the Discussion section. We have reflected on respective multiannual data gaps, their irregular temporal placement over the observed years, along the computed deviations from the authoritative observations and their varying intensity and variability, making a data quality concern a much broader issue. These concerns were equally reflected upon within the Conclusion section.

Author Response File: Author Response.pdf

Reviewer 3 Report

This article deals with an interesting topic, however, the methods and conclusion sections should be improved. Regarding figure no.1, it would be advisable to show LCZ maps and then quantify the LCZ class distribution. In the discussion and conclusion sections, please provide an explanation of how this study can be useful and how the developed methods can be applied to further case studies in other cities.

Author Response

This article deals with an interesting topic, however, the methods and conclusion sections should be improved. Regarding figure no.1, it would be advisable to show LCZ maps and then quantify the LCZ class distribution. In the discussion and conclusion sections, please provide an explanation of how this study can be useful and how the developed methods can be applied to further case studies in other cities.

Answer: We would like to thank the Reviewer for the time and effort put into reviewing our manuscript. We are in an absolute agreement with the above observations.

Firstly, we recognize the benefit of providing more references on the existing body of work illustrating data quality control approaches that tackle crowdsourced data. We have thus reflected on a number of relevant resources and their work within the Introduction section. Please refer to the revised manuscript.

We have also included additional relevant studies in the methodology section to further strengthen the integrity of our approach. We have added some relevant parallels to the existing methodologies that originated from prior work. Please refer to the changes made in the manuscript, specifically the newly added subsection 2.2.

We equally recognize the benefit of introducing more information that may complement Figure 1, such as the LCZ map. Hence the respective figure LCZ depiction has been introduced as a complementary Figure 2, as per Reviewer’s suggestion. Indeed, a visual representation of respective LCZ distribution in the concerned urban domain brings more clarity to the representative LCZ class of study domains.

Within the discussion section, we have introduced a new subsection (4.4) that discusses the replication potential of the introduced methodology to other cities, along the potential factors limiting direct transferability. We have equally introduced another subsection (4.2) in our revised manuscript that discusses a multiannual analysis for an exemplary use case.

Within the conclusion section, we have reflected on some aspects that may be understood as a limitation of the current assessment to which we have raised attention, so as to achieve scientific objectivity. This includes the discussion of the replication potential, among other things, along the data quality concerns being a much broader issue than a single annual incidence.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The manuscript was slighly improved. Nevertheless, the changes were not enough to change my opinion.

It is stated that this study "aims to highlight systematic quality issues...". I agree: the most evident quality issues are highlighted, indeed:  missing data and deviations. Nevertheless, this is not new and the PWS data users are already aware of such limiations. I recommend the authors to add any novalty to this study, for instance contribute to define quality control measures that can be easily applied/verified. 

Author Response

The manuscript was slightly improved. Nevertheless, the changes were not enough to change my opinion.

It is stated that this study "aims to highlight systematic quality issues...". I agree: the most evident quality issues are highlighted, indeed:  missing data and deviations. Nevertheless, this is not new and the PWS data users are already aware of such limitations.

Answer: We would like to thank the Reviewer for the time and effort put into reviewing our revised manuscript.

We appreciate your comments and we have tried to address the latest points raised to the best of our abilities. In an effort to avoid any potentially misleading statements, we have decided to delete a potential problematic notion of “systematic” in the revised manuscript. By doing so, we are now stating that “we aimed to highlight quality issues that are integral to the crowdsourced PWS data and the resulting implications that arise when such data is used for empirical research and numerical simulation”. We feel that this allows us to avoid any potential falsehood regarding the scope of our work presented in this manuscript.

I recommend the authors to add any novelty to this study, for instance contribute to define quality control measures that can be easily applied/verified.

Answer: We appreciate your comment and we see the benefit of including the suggested aspects. We have added a new subsection entitled “4.5 Future prospects for correcting faulty data” where we discussed an application of a “buddy check” approach and the resulting identification of faulty signals. We aimed to illustrate how one may identify those faulty signals based on the neighboring station and eventually derive a more reliable dataset.

Reviewer 3 Report

The authors have completed what was required. No suggestions for authors.

Author Response

We would like to thank the Reviewer for the time and effort put into reviewing our manuscript. We are pleased to know that our revised manuscript successfully addressed all the points raised by the Reviewer.

Round 3

Reviewer 1 Report

Congratulations!