A Survey Bias Index Based on Unmanned Aerial Vehicle Imagery to Review the Accuracy of Rural Surveys
Round 1
Reviewer 1 Report
This paper focuses on the comparison of data on the area of homesteads in the rural area obtained by a questionnaire and by a drone. The main strength of the article lies in the methodology which is processed instructively. The aim of the article is not entirely clear. The title implies that it should be a matter of detecting data distortion by respondents. Although this goal has been achieved, its explanatory power is limited to this specific case. As rightly pointed out in the discussion, drones can only be used in this direction for objects observable from the air. In addition, it was probably a one-off observation, so it was a static analysis- hardly acceptable for dynamic research. The example is not chosen very well, because it is better to use real estate records to find out the data. If real estate registration is not available in China, the main conclusion should be that it is necessary to digitize cadastral areas.
It goes without saying that data from sociological surveys are fraught with errors. There can be several reasons: the respondent does not know the exact data and does not want to admit it, or the respondents want to introduce themselves better (exceptionally worse) than they are, or try to hide some facts which they consider for confidential or not legal and the like.
Inaccurate facts are of a various nature. Some are completely accurate (such as gender or age data), others may be slightly skewed, others are entirely subjective. There are a number of methods for detecting differences, but only a few can be detected by remote sensing. So the paper presents an interesting methodology, but its use is not obvious. Is the result the fact that the owners of larger properties distort the data more or that the owners of properties in the urban fringe distort the facts more than the owners in the purely rural zone? To what extent are these conclusions generally valid?
The statement on the creation of the indicator is somewhat exaggerated. Are the findings from the article generally valid? What types of problems can they cover? I propose either rejecting the article or reworking it in order to document its real contribution - that is the possibility of monitoring certain facts from above (from roofs, aircraft models, aircraft, satellites) and a sample of a case study.
Author Response
Reviewer #1:
Point 1:
This paper focuses on the comparison of data on the area of homesteads in the rural area obtained by a questionnaire and by a drone. The main strength of the article lies in the methodology which is processed instructively. The aim of the article is not entirely clear. The title implies that it should be a matter of detecting data distortion by respondents. Although this goal has been achieved, its explanatory power is limited to this specific case. As rightly pointed out in the discussion, drones can only be used in this direction for objects observable from the air. In addition, it was probably a one-off observation, so it was a static analysis- hardly acceptable for dynamic research. The example is not chosen very well, because it is better to use real estate records to find out the data. If real estate registration is not available in China, the main conclusion should be that it is necessary to digitize cadastral areas.
Response 1: Thank you for your comment.
The main aim of the article is to develop cost-effective and efficient methods based on UAVs to validate the accuracy of self-reported data in questionnaires of field surveys. Just as you said, as an auxiliary data acquisition means, UAV can only obtain data of some objects that can be observed directly. However, my suggestion is to use this part of the data to verify the reliability of the questionnaire data. From this point of view, UAV assisted survey is still very necessary. Furthermore, the duration of a UAV survey is limited. Periodic UAV data acquisition should be used in the construction of a long-timescale continuous rural panel data set. China has established real estate registration. However, registration networking and real-time update have not been completed. There are many rural residential reconstructions and expansions in the study area, and there are some differences between the homestead area and the registered area. This study chooses to record the self-reported homestead area data.
I have presented this in the discussion section (lines 291–299) as follows:
“The SBI method based on UAV has some limitations. First, it can only be used to verify specific objects in rural surveys, rather than abstract objects, such as income, employment, and education. It cannot be used for objects invisible to UAVs, such as indoor assets. However, in this case, this part of the data can be used to verify the reliability of the questionnaire data. Second, the duration of a UAV survey is limited. Periodic UAV data acquisition should be used to construct a long-timescale continuous rural panel data set [30]. Third, the scope and scale of UAV surveys are limited. When surveys are conducted in large villages, small drones may not be able to obtain data effectively.”
Point 2:
It goes without saying that data from sociological surveys are fraught with errors. There can be several reasons: the respondent does not know the exact data and does not want to admit it, or the respondents want to introduce themselves better (exceptionally worse) than they are, or try to hide some facts which they consider for confidential or not legal and the like.
Inaccurate facts are of a various nature. Some are completely accurate (such as gender or age data), others may be slightly skewed, others are entirely subjective. There are a number of methods for detecting differences, but only a few can be detected by remote sensing. So the paper presents an interesting methodology, but its use is not obvious. Is the result the fact that the owners of larger properties distort the data more or that the owners of properties in the urban fringe distort the facts more than the owners in the purely rural zone? To what extent are these conclusions generally valid?
Response 2: I agree with you that there are many methods to verify the accuracy of social survey data. UAV verification is only one of them. This study aimed to verify the feasibility of this method and clarify its limitations. In the future, the social survey should still focus on the household survey, but more technical methods may be introduced to improve the accuracy of the survey. Further, my results suggest that homestead areas in the pre-urban zone showed more survey bias than those in the pure rural zone. In addition to the three reasons mentioned above, the homesteads in pre-urban zone have greater possibilities and higher frequency of reconstruction and building extension for rental and other businesses. Therefore, the homestead areas in the pre-urban zone showed greater SBI than the pure rural regions, where there is a lack of such motivation to reconstruct, and the degree of survey bias was significantly lower. The larger homesteads showed greater data bias than others. One possible explanation is that the rebuilt houses may expand the area, resulting in a great survey bias in the investigation of a large area of the homestead. Another rural homestead survey also reported that the evolution and upgrading of the structure and function of homesteads are influenced by factors such as expanding production, improved living conditions and promotion of regional economic development, and mutual comparison between neighbors.
I have presented this in the discussion section (lines 264–281) as follows:
“Survey bias has three possible sources: 1) Different measurement methods lead to different measurement areas; 2) The measurement methods are the same, but accuracy errors exist; 3) Inconsistent area due to house reconstruction and expansion [28].
SBI obtained in the pre-urban zone (0.515) was higher than that in the pure rural zone (0.258). The homestead areas in the pre-urban zone showed more survey bias than those in the pure rural zone (Table 3). In addition to the three reasons mentioned above, the homesteads in the pre-urban zone have greater possibilities and higher frequency of reconstruction and building extension for rental and other businesses [26]. Therefore, the homestead areas in the pre-urban zone showed greater SBI than the pure rural regions, where there is a lack of such motivation to reconstruct, and the degree of survey bias was significantly lower. As shown in Fig. 5, the larger homesteads showed greater data bias than others. One possible explanation is that the rebuilt houses may expand the area, resulting in a great survey bias in the investigation of a large area of the homestead. Another rural homestead survey also reported that the evolution and upgrading of the structure and function of homesteads are influenced by factors such as expanding production, improved living conditions, promotion of regional economic development, and mutual comparison between neighbors [29].”
Point 3:
The statement on the creation of the indicator is somewhat exaggerated. Are the findings from the article generally valid? What types of problems can they cover? I propose either rejecting the article or reworking it in order to document its real contribution - that is the possibility of monitoring certain facts from above (from roofs, aircraft models, aircraft, satellites) and a sample of a case study.
Response 3: I agree with your comment. The social desirability bias index has been revised. My results focused on determining the survey bias by UAV through a case study focusing on survey bias detection other than social desirability bias.
I have revised the entire section 2.5 (lines 165 – 183) as follows:
“2.5. Survey bias index
Values obtained from the UAV were set as the baseline. Survey bias was then estimated using the degree of shift from the baseline. Therefore, in the absence of survey bias, the homestead areas reported by the respondents should closely match the information extrapolated from UAV data. Any discrepancy between the datasets indicates some degree of survey bias in the household survey data. To minimize error from the respondents’ inaccurate memory, the measurement at the main building area was selected for comparison between the survey data and UAV data (as respondents were likely to have the greatest familiarity with the dimensions of this facility).
To evaluate survey bias during surveys on rural homesteads and analyze the interspecific differences in the two survey methods, the index of survey bias for rural surveys was calculated using the least squares method:
(1)
where is the survey bias index, is the value associated with the rural homestead obtained by UAV (m2), is the difference between homestead areas obtained from self-reported questionnaires and UAV (m2), is the number of rural homesteads surveyed using the two methods, and is the i-th zone surveyed.”
Author Response File: 
Author Response.docx
Reviewer 2 Report
The manuscript „Social Desirability Bias Index Based on Unmanned Aerial Vehicle Imagery To Review the Accuracy of Rural Surveys” has in attention the comparison of the data recorded in questionnaires with those from portable unmanned aerial vehicles (UAVs).
The authors should consider the following aspects to be implemented in the manuscript:
- The Introduction should be extended to comprise the most recent published studies on the advantages and disadvantages of the considered indexes.
- Results: organize this section (results on UAV, results on questionnaires).
- Discussions: consider including comparisons to previously published studies.
Author Response
Reviewer #2
Point 1:
The manuscript „Social Desirability Bias Index Based on Unmanned Aerial Vehicle Imagery To Review the Accuracy of Rural Surveys” has in attention the comparison of the data recorded in questionnaires with those from portable unmanned aerial vehicles (UAVs).
The authors should consider the following aspects to be implemented in the manuscript:
1)The Introduction should be extended to comprise the most recent published studies on the advantages and disadvantages of the considered indexes.
2)Results: organize this section (results on UAV, results on questionnaires).
3)Discussions: consider including comparisons to previously published studies.
Response 1: Thank you for your comment and suggestion.
- In the introduction section, some research progress was presented, demonstrating that third-party data were collected to check the accuracy of self-reported data from the field survey. However, there is no uniform approach for representing the degree of bias. Therefore, this study aimed to establish an index to assess the degree of survey bias based on third-party data from UAVs.
- I have revised section 3.2 according to your suggestion, (lines 199–208), as follows:
- “Twelve rural homesteads were surveyed using a questionnaire in Zone A, a typical pre-urban zone (Fig. 3a). The areas of rural homesteads in Zone A range from 65.12 to 243.11 m2, with a mean of 138.57 ± 58.69 m2 (Fig. 3a). Existing buildings in Zone A form the original neighborhood. Most buildings are 2–4 stories and self-built by farmers for rental purposes. Figure 3b shows all areas of the homesteads in Zone A as a visual interpretation based on UAV data. Nineteen rural homesteads were surveyed using a questionnaire in Zone B (Fig. 3c). Their areas range from 50.16 to 163.88 m2, with a mean of 105.98 ± 32.11 m2 (Fig. 3c). The buildings in Area B are distributed along the riverbank. There are 1–2 story buildings occupied by farmers for personal use. Whole homesteads in Zone B were interpreted based on orthographic and 3D UAV data (Fig. 3d).”
- I have added some previously published studies in the discussion section (lines 264–281) and (lines 291–299) as follows:
- “Survey bias has three possible sources: 1) Different measurement methods lead to different measurement areas; 2) The measurement methods are the same, but accuracy errors exist; 3) Inconsistent area due to house reconstruction and expansion [28].
- SBI obtained in the pre-urban zone (0.515) was higher than that in the pure rural zone (0.258). The homestead areas in the pre-urban zone showed more survey bias than those in the pure rural zone (Table 3). In addition to the three reasons mentioned above, the homesteads in the pre-urban zone have greater possibilities and higher frequency of reconstruction and building extension for rental and other businesses [26]. Therefore, the homestead areas in the pre-urban zone showed greater SBI than the pure rural regions, where there is a lack of such motivation to reconstruct, and the degree of survey bias was significantly lower. As shown in Fig. 5, the larger homesteads showed greater data bias than others. One possible explanation is that the rebuilt houses may expand the area, resulting in a great survey bias in the investigation of a large area of the homestead. Another rural homestead survey also reported that the evolution and upgrading of the structure and function of homesteads are influenced by factors such as expanding production, improved living conditions, promotion of regional economic development, and mutual comparison between neighbors [29].”
- “The SBI method based on UAV has some limitations. First, it can only be used to verify specific objects in rural surveys, rather than abstract objects, such as income, employment, and education. It cannot be used for objects invisible to UAVs, such as indoor assets. However, in this case, this part of the data can be used to verify the reliability of the questionnaire data. Second, the duration of a UAV survey is limited. Periodic UAV data acquisition should be used to construct a long-timescale continuous rural panel data set [30]. Third, the scope and scale of UAV surveys are limited. When surveys are conducted in large villages, small drones may not be able to obtain data effectively.”
Author Response File: Author Response.docx
Reviewer 3 Report
This is a very intriguing paper. It is well designed and described in straight forward language. I do, however, have a question about whether it would be possible to extend the UAV model to respondents' descriptions of their attitudes and behaviors. Would there, for example, be ways to monitor responses with some type of electronic assessment of breathing/anxiety, etc?
Author Response
Reviewer #3
Point 1:
This is a very intriguing paper. It is well designed and described in straight forward language. I do, however, have a question about whether it would be possible to extend the UAV model to respondents' descriptions of their attitudes and behaviors. Would there, for example, be ways to monitor responses with some type of electronic assessment of breathing/anxiety, etc?
Response 1: Thank you for your novel and constructive suggestion. I would be interested in exploring this suggestion further and potentially including it in my future field survey work.
Author Response File: Author Response.docx
Reviewer 4 Report
Although the research is clearly interesting, it cannot be published because of ethical concerns. There is no indication that respondents were informed that their responses were going to be compared with UAV data. There is no indication that there was an ethical committee that allowed and overviewed the research. Results clearly indicate that survey responses were compared to SAV data to see if their answers were true. If the researcher wants to prove the accuracy of the respondents´ answers, respondents first need to be informed that this will be done. Because of the low educational level of respondents, there needs to be oversight that they understood that their answers were going to be compared with actual measurements. But even with oversight and approval from an ethics committee, this research should not be published in a reputable journal because it portrays rural inhabitants as liars. Finally, even if respondents are concealing the truth with their answers, they should have the right to do so, if that is their decision.
Author Response
Reviewer #4
Point 1:
Although the research is clearly interesting, it cannot be published because of ethical concerns. There is no indication that respondents were informed that their responses were going to be compared with UAV data. There is no indication that there was an ethical committee that allowed and overviewed the research. Results clearly indicate that survey responses were compared to SAV data to see if their answers were true. If the researcher wants to prove the accuracy of the respondents´ answers, respondents first need to be informed that this will be done. Because of the low educational level of respondents, there needs to be oversight that they understood that their answers were going to be compared with actual measurements. But even with oversight and approval from an ethics committee, this research should not be published in a reputable journal because it portrays rural inhabitants as liars. Finally, even if respondents are concealing the truth with their answers, they should have the right to do so, if that is their decision.
Response 1:
Thank you for your comment. I have added further details to section 2.2 Questionnaire and survey design. Additionally, in the discussion and conclusion sections, I have revised the explanation of the source of the difference between the homestead area from the questionnaire and the UAV. Some potential differences in measurement methods, accuracy, and reconstruction were included. The explanation of overemphasizing subjective bias in the original discussion is not comprehensive. The evolution and upgrading of the structure and function of homesteads are influenced by factors such as expanding production, improved living conditions and promotion of regional economic development, and mutual comparison between neighbors.
I have added details to section 2.2. Questionnaire and survey design (lines 107–111) as follows:
“The investigator explained that the purpose of the survey is to evaluate the accuracy of the homestead area, and the survey data will be compared with the UAV remote sensing data. Because the study area has many rural residential reconstructions and expansions in the study area, and differences exist between the homestead area and the registered area, in this study, we chose to record the homestead area data as self-reported.”
Author Response File: Author Response.docx
