The Potential for Citizen Science to Improve the Reach of Sanitary Inspections

Round 1

Reviewer 1 Report

The manuscript "The potential for citizen science to improve the reach of Sanitary Inspections" presents the results of study conducted in order to investigate the eficiency of citizen science used as hazard-identification which may compromise the quality ofl drinking-water supplies.

Used method is presented as cheap, easy and useable by uneducated persons. The authors emphasize the challenges and advantages of used method.

They also noted that citizen science could be a good option for extending the reach of Sanitary Inspections in a bid to reach the Sustainable Development Goals.

Before acceptance, I recommend to authors to improve the quality of all figures presented in paper.

Author Response

Many thanks for your comments and feedback on our submitted manuscript. Please see below amendments in response to these comments.

Comment 1: Before acceptance, I recommend to authors to improve the quality of all figures presented in paper.

Response 1: All figures have been amended to improve legibility and quality.

Reviewer 2 Report

The potential for citizen science to improve the reach of Sanitary Inspections
Authors: Jo Herschan, Richard King, Theresa Mkandawire, Kenan Okurut, Dan J Lapworth, Rosalind Malcolm, Katherine Pond
Drinking Water Safety Management

The aim of this paper is to investigate the potential role of citizen science in the field of SI with remote validation of results by experts in SI. A mixed methods approach was used to answer the following questions:

i) Is there consistency between citizen’s perception of risk to small drinking-water supplies?

ii) Do citizen’s perception of risk change with the number of inspections conducted?

iii) Do citizens require additional information to survey supplies effectively?

iv) Do citizens assign comparative risk scores when surveying supplies on-site and remotely using photographic images?

v) Is it possible to verify responses remotely using expert judgement?

It is anticipated that the findings of this study will contribute to achieving SDG Target 6.1 by suggesting an approach which will increase the reach of SIs in small drinking-water supplies through the use of citizens.

Major comments

The questions posed in the article and the purpose of the article are relevant and have practical significance. However, it seems that in fact the article assesses the subjective natural abilities of students, and not the ability of students to assess objective technogenic risks.

Is the fencing around the borehole absent? Is the apron around the borehole absent? Is the sources of pollution within 10 m of the borehole? - these and other questions from the borehole SI form and the spring SI form require unambiguous answers (yes-no). There can be no different options here. Therefore, along with the results of the assessments made by the students at 18 wells and 12 sources, it would be necessary to present the estimates made at the same wells and sources by professional experts. Then it would be necessary to show on all graphs how the results of students and experts correlate, what are the typical mistakes of students: in order to take this experience into account in the future. In general, it is necessary to study students' learning ability to correctly assess risks. To do this, professional experts need to train them in short-term (3-4 lessons at least) courses of public health volunteers with the issuance of certificates for the right to conduct surveys and risk assessment.

Minor comments

Graphs are hard to read and difficult to analyze due to blurred lines and fonts.

Fig. 1. You need to show the numbers of wells 1-13 and 14-18.

Page 4, lines 155-157

You write: The citizens travelled to the supplies under inspection as one group and were provided the same length of time to inspect each supply. All data was collected during the same week in which weather conditions were warm and dry, so corrections for seasonal variation were not required.

It raises the question of conducting inspections by one group. It seems that inspections should be strictly individual in order to exclude "cheating", which is typical for students.

Page 6, lines 229-231

You write: Figure 3 provides the total number of risks assigned for all citizens for all supplies per question number. Question 5 was the most frequently identified risk, followed by questions 8, 3, 4 and 7. Questions 1, 2, 6 and 9 were answered “Yes” noticeably less frequently.

Questions should be briefly identified, for example 5 (fencing), 8 (sources of pollution 10 m), 3 (apron), 4 (drainage), 7 (sanitation 30 m)…

Comments for author File: Comments.pdf

Author Response

Many thanks for your comments and feedback on our submitted manuscript. Please see below amendments in response to these comments.

Comment 1: The questions posed in the article and the purpose of the article are relevant and have practical significance. However, it seems that in fact the article assesses the subjective natural abilities of students, and not the ability of students to assess objective technogenic risks.

Is the fencing around the borehole absent? Is the apron around the borehole absent? Is the sources of pollution within 10 m of the borehole? - these and other questions from the borehole SI form and the spring SI form require unambiguous answers (yes-no). There can be no different options here. Therefore, along with the results of the assessments made by the students at 18 wells and 12 sources, it would be necessary to present the estimates made at the same wells and sources by professional experts. Then it would be necessary to show on all graphs how the results of students and experts correlate, what are the typical mistakes of students: in order to take this experience into account in the future. In general, it is necessary to study students' learning ability to correctly assess risks. To do this, professional experts need to train them in short-term (3-4 lessons at least) courses of public health volunteers with the issuance of certificates for the right to conduct surveys and risk assessment.

Response 1: The authors agree that the next step to investigate the potential for SIs to be used by citizens is for experts to carry out on-site inspections with citizens to highlight problem areas or questions and correlate responses between experts and citizens. It is however logistically challenging to achieve, due to the remoteness of some of the supplies, as it would be beneficial to do so simultaneously to ensure no variations in on-site conditions. The authors agree that this could be clarified within the article and have included this in a further work section (lines 524 - 531) and have added the challenges to the limitations section (lines 541– 549).

The authors discussed the inclusion of the comparison of their abilities to assess risk when preparing the article. We decided that it was important to identify and present trends in the variation in student responses in a first step to confirming whether SIs could be used by non-experts or if students gave wildly differing responses in risk based on their degree subjects.

The authors agree, as discussed in the article, that further guidance and support may be required however to maintain the citizen science nature, and to limit the strain on resources often experienced in small supplies, considered this to be more in the form of documentation as opposed to training packages. Conducting courses on surveys and risk assessment would negate the citizen science aspect of the work as the aim is not to increase the number of skilled sanitary inspectors but to test whether the forms could be used by non-expert/lay people.

Comment 2: Graphs are hard to read and difficult to analyze due to blurred lines and fonts.

Response 2: All figures have been amended to improve legibility and quality.

Comment 3: Fig. 1. You need to show the numbers of wells 1-13 and 14-18.

Response 3: Supply numbers have been added to Figure 1.

Comment 4: Page 4, lines 155-157

You write: The citizens travelled to the supplies under inspection as one group and were provided the same length of time to inspect each supply. All data was collected during the same week in which weather conditions were warm and dry, so corrections for seasonal variation were not required.

It raises the question of conducting inspections by one group. It seems that inspections should be strictly individual in order to exclude "cheating", which is typical for students.

Response 4: Many thanks for this comment. You raise a good point however is a logistically challenging one to overcome due to the remoteness of the supplies visited, especially the spring sources. Surveying all at the same time provides a comparison of risk in the same conditions i.e. surveys carried out during different times of the day or during different seasons may yield variations in results which may influence risk perception. Additional wording has been added to the limitations section to highlight this (lines 541 – 549).

From the outset of the study and throughout the study students were instructed and reminded not to confer with eachother. They were told several times this was not a test and there were no correct or incorrect answers. Therefore, a certain element of trust in the students’ honesty and integrity is required.

Comment 5: Page 6, lines 229-231

You write: Figure 3 provides the total number of risks assigned for all citizens for all supplies per question number. Question 5 was the most frequently identified risk, followed by questions 8, 3, 4 and 7. Questions 1, 2, 6 and 9 were answered “Yes” noticeably less frequently.

Questions should be briefly identified, for example 5 (fencing), 8 (sources of pollution 10 m), 3 (apron), 4 (drainage), 7 (sanitation 30 m)…

Response 5: A table with identification has been added (new Table 3). Identifiers were trialled in text however it confused the flow of the sentences.

Reviewer 3 Report

Dear Authors,

thank you very much for this interesting and relevant paper. I believe using citizen science can bring relevant improvements to the implementation of WSPs in many regions of the world.

I only have one major methodological comment:  

The different groups of participants, distinguished by their degree subject have significantly varying size (1 to 6). In addition, the two subjects represented only by one person (auditing and physical planning) show lowest and highest risk scores. I would therefore refrain from discussing the results by degree subject in detail (this should be shown in one figure only), but rather concentrate on the variations within the total group of citizens. You could group by supply, by question or by setting. Figure 5 is for me the most relevant one of the complete study.

The major outcome for me is, that SIs completed by citizens can bring so benefit, but in order to know, how sound the results are, it would be important to find out, how these results relate to expert assessment. It would also be interesting to know how a group of experts would vary in their risk assessment.

Some minor comments in the pdf attached.

I hope this can contribute to further improvement of this relevant paper..

All the best!

Comments for author File: Comments.pdf

Author Response

Many thanks for your comments and feedback on our submitted manuscript. Please see below amendments in response to these comments.

Comment 1: The different groups of participants, distinguished by their degree subject have significantly varying size (1 to 6). In addition, the two subjects represented only by one person (auditing and physical planning) show lowest and highest risk scores. I would therefore refrain from discussing the results by degree subject in detail (this should be shown in one figure only), but rather concentrate on the variations within the total group of citizens. You could group by supply, by question or by setting. Figure 5 is for me the most relevant one of the complete study.

Response 1: Thank you for the comment. The authors did discuss this and decided to investigate via subject because it allows for the discussion around an individual’s experiences and perception of risk, hence why lines 433 – 442 in the discussion have been left in. Figure 2 has been removed because the authors agree that on balance this figure does not add further to what is provided in the figure now labelled as Figure 3.

The major outcome for me is, that SIs completed by citizens can bring so benefit, but in order to know, how sound the results are, it would be important to find out, how these results relate to expert assessment. It would also be interesting to know how a group of experts would vary in their risk assessment.

Response 2: The authors acknowledge that this is a small sample size and that further testing is required. Comparisons between the expert and citizen responses were carried out as part of the remote inspection. On-site inspections by experts and citizens have been added to the further work section (lines 524– 531).

Comments from the attached pdf:

Comment 1: Line 363, change citizen’s to citizens

Response 1: Completed

Comment 2: Figure 9 caption should say inspection at the end

Response 2: Updated

Comment 3: Line 386 – ‘Sure this is the right questionnaire? (spring not borehole?)

Response 3: Thank you for picking up on this. Line 386 updated from ‘question 8’ to ‘question 6’.

Comment 4: Line 392 – In the Methods you state that 7 and 9 were not part of this questionnaire. Is that correct?

Response 4: Line 384 updated to include additional wording for clarification.

Reviewer 4 Report

It is a good initiative to employ citizen science for SI. However, all participants were university students so that there are not difficulties for them to learn and master the check list, and conduct the survey. A true citizen group may include people with different ages, different educational levels and different life experiences.  Therefore, the design of this study is somehow flawed although the results are worth publishing. 

It selected eighteen borehole supplies and considered them as representative in terms of design and condition. Could details of the design and conditions be provided?

Author Response

Many thanks for your comments and feedback on our submitted manuscript. Please see below amendments in response to these comments.

Comment 1: It is a good initiative to employ citizen science for SI. However, all participants were university students so that there are not difficulties for them to learn and master the check list, and conduct the survey. A true citizen group may include people with different ages, different educational levels and different life experiences.  Therefore, the design of this study is somehow flawed although the results are worth publishing. 

Response 1: Many thanks for your comment. The students are all citizens of Malawi but the authors agree that it would be beneficial to test SIs using individuals from a breadth of backgrounds and ages etc.. This was noted in the limitations section however on reflection could be expanded upon. Further wording has been added to lines 539-540 for clarity. Comments have also been incorporated into a further work section.

Comment 2: It selected eighteen borehole supplies and considered them as representative in terms of design and condition. Could details of the design and conditions be provided?

Response 2: Wording has been amended to include that all boreholes were operational, handpump boreholes.