Citizens’ Satisfaction with Air Quality and Key Factors in China—Using the Anchoring Vignettes Method
Round 1
Reviewer 1 Report
This paper represents an innovative approach to studying public perceptions of air quality, and potentially has merit. However, I have some concerns that are outlined below. I have organized my comments around the reviewer questions posed above.
Does the introduction provide sufficient background and include all relevant references?
The introduction could explain better what the study is actually doing, and it is a little thin on references. The focus seems to be on studies of China--not without good reason--but there is a considerable body of research on this topic outside China that is not addressed.
The literature review/hypothesis section reads more like a methods section, with its focus on measurement. And again, there is a wider body of research in this area that could be addressed.
Is the research design appropriate?
I consider the design to be appropriate, and even somewhat innovative. However, I do have some issues:
The anchoring vignettes seem to conflate weather conditions with air quality, which could potentially limit their usefulness. Using myself as an example, I live in a state that gets 300 days of sunshine per year--"blue sky and white clouds" according to the vignette. My experience is very different from someone who lives in, for example, Seattle, where they have clouds and rain all the time, but has nothing to do with air quality.
The questions about satisfaction with government services are somewhat generic, and it is not clear how they would relate specifically to air quality. While we can to some extent assume that the government is (at least partially) responsible for air quality, those questions do not really get at that issue.
Why are the media usage (both social and official) and "personal experience" operationalized as dummy variables? It would make more sense to look at time spent on these platforms and/or what people rely on as their primary source of information. Also, as far as I can tell "personal experience" is never actually defined, which relates to the next question.
Are the methods adequately described?
The short answer is no. While the anchoring vignette method is innovative, it is not clear how these vignettes reduce measurement error. Given that this is the central focus of the paper it should be much better explained.
Beyond that, there are a number of independent variables where how they are specifically measured is unclear.
On a more minor note, Table 3 kind of comes out of nowhere. It could benefit from some setup and/or discussion.
Are the results clearly presented?
In the results section the equations presented seem like they would fit better in the methods section--and could also be better explained as noted above.
It is not clear what the maps are intended to show, or what the differences between the two maps are. I assume they are showing city level differences in satisfaction, but they do not look very different and whatever differences there are are not very well explained.
Are the conclusions supported by the results?
The paper concludes that anchored vignettes are a better way to measure public satisfaction with air quality. Although I do not question this conclusion it needs to be much better explained why, and what the vignettes actually do.
It is not clear why greater sensitivity to PM2.5 relative to SO2 and dust indicates that "sustainable citizenship is becoming increasingly important in China." This claim needs much more elaboration.
The conclusion regarding public services also could use some more elaboration, especially since the survey questions on that issue do not directly address government responsibility for air quality.
The policy implications do not necessarily relate to the results. If they do, further elaboration is needed here as well.
Author Response
Response to Reviewer 1 Comments
This paper represents an innovative approach to studying public perceptions of air quality, and potentially has merit. However, I have some concerns that are outlined below. I have organized my comments around the reviewer questions posed above.
Does the introduction provide sufficient background and include all relevant references?
Point 1: The introduction could explain better what the study is actually doing, and it is a little thin on references. The focus seems to be on studies of China--not without good reason--but there is a considerable body of research on this topic outside China that is not addressed.
Response 1: Thank for your suggestions. With your advice accepted, we adjust our writing in first paragraph to illustrate why we focus on studies of China. And also we also add researches done outside China, eg, the USA, United Kingdoms, to make the issue more important.
Point 2：The literature review/hypothesis section reads more like a methods section, with its focus on measurement. And again, there is a wider body of research in this area that could be addressed.
Response2: We really appreciate your advice. One of our research motivations is to confirm whether anchoring vignettes method works for air quality perception, so that the first research hypotheses is on the method, making the literature review like a method section. However, more researches related to this paper have been added into the literature review part, please see in the updated version of this article.
Is the research design appropriate?
I consider the design to be appropriate, and even somewhat innovative. However, I do have some issues:
Point 3: The anchoring vignettes seem to conflate weather conditions with air quality, which could potentially limit their usefulness. Using myself as an example, I live in a state that gets 300 days of sunshine per year--"blue sky and white clouds" according to the vignette. My experience is very different from someone who lives in, for example, Seattle, where they have clouds and rain all the time, but has nothing to do with air quality.
Response3: Thanks for your question on the anchoring vignettes design. At the very beginning, we think that good air quality, to a large extent, is “blue sky and white clouds”. To be honest, we didn’t think of the extreme case like Seattle. However, in 17 cities from Shandong Province, there is no one city like Seattle in terms of climate type. But still, it is a good point of view when designing anchoring vignettes in the future study.
Point 4: The questions about satisfaction with government services are somewhat generic, and it is not clear how they would relate specifically to air quality. While we can to some extent assume that the government is (at least partially) responsible for air quality, those questions do not really get at that issue.
Response 4: Thank you very much for pointing that out. The question of satisfaction on public service is to measure the citizen’s expectation towards public service instead of satisfaction itself. The measuring method is from prior researches, for example, Van Ryzin (2004). It seems that there is no direct relationship between what we asked and air quality perception. However, we think that air quality perception is influenced by many factors, for instance, psychological and cultural ones. Therefore, these questions are to measure the citizen’s expectations which might influence their perception to air quality. Yes, of course. It would be better if we could design questions which can measure citizen’s expectation in terms of air quality. we think it would be research to be done in the near future.
Point 5: Why are the media usage (both social and official) and "personal experience" operationalized as dummy variables? It would make more sense to look at time spent on these platforms and/or what people rely on as their primary source of information. Also, as far as I can tell "personal experience" is never actually defined, which relates to the next question.
Response 5: Thanks for your advice. We have to admit that we measure the information sources only by one multiple choice question. It is a multiple choice question so that we have to make them dummy variables. In addition, personal experience refers to evaluating the air quality based on your experience, which can be easily understood in Chinese. And the specifications for other variables are also explained in the revised version.
Are the methods adequately described?
Point 6: The short answer is no. While the anchoring vignette method is innovative, it is not clear how these vignettes reduce measurement error. Given that this is the central focus of the paper it should be much better explained.
Response 6: We accept your advice and make more elaboration on anchoring vignettes, see the details in the updated version. [ According to prior research, the anchors that can be used to attach the answers of different individuals should be the same standard scale [11,12]. Although vignettes have been designed in political efficacy, self-reported health, and corruption perception, there are no vignettes that can be referred to in air pollution domain.
Also, using the anchor package in R Language, we tested the order of the two vignettes according to the assumption that on average the level of satisfaction in vignette 1 is higher than that in vignette 2, indicating that the two vignettes we designed were appropriate for correcting the DIF problem in measuring the air quality satisfaction.]
Point 7: Beyond that, there are a number of independent variables where how they are specifically measured is unclear.
Response 7: Thanks very much. We make more explanation on variables specification, see the details in the revised version.
Point 8: On a more minor note, Table 3 kind of comes out of nowhere. It could benefit from some setup and/or discussion.
Response 8: Thanks a lot. We give a note on table 3. [Table 3 gives the summary statistics of the dependent and independent variables both at individual and city level.]
Are the results clearly presented?
In the results section the equations presented seem like they would fit better in the methods section--and could also be better explained as noted above.
Point 9: It is not clear what the maps are intended to show, or what the differences between the two maps are. I assume they are showing city level differences in satisfaction, but they do not look very different and whatever differences there are are not very well explained.
Response 9: Thank you very much for the kind suggestion. We take your advice and remake the map only to show the difference of satisfaction with air quality before and after anchoring vignettes so that it can indicate clearly the variation among 17 cities in Shandong Province.
Are the conclusions supported by the results?
Point 10: The paper concludes that anchored vignettes are a better way to measure public satisfaction with air quality. Although I do not question this conclusion it needs to be much better explained why, and what the vignettes actually do.
Response 10: Thank you for your advice. We make more elaboration on the mechanism why anchoring vignettes can overcome the measurement error, see the details in the revised version in conclusion section. [Estimating the air quality satisfaction combining the two vignettes instead of directly asking is preferred since citizens’ view of air quality satisfaction can be varied a lot. Therefore, when respondents are put into the same air quality context where each one can have same understanding to what is clean air, it is reasonable that the satisfaction level can be compared with.]
Point 11: It is not clear why greater sensitivity to PM2.5 relative to SO2 and dust indicates that "sustainable citizenship is becoming increasingly important in China." This claim needs much more elaboration.
Response 11: We changed our saying of this sentence into [Compared to SO2 and dust emission concentrations, PM2.5 has an impact on citizen’s air quality which perception to a larger extent, which makes it clear that sustainable citizenship is becoming increasingly important in China. In other words, citizens are paying more attention to the effects of air quality, especially measured by PM2.5, on public health. Therefore, the air quality can be perceived by citizens accurately. However, figure 6 shows that citizens’ satisfaction with air quality won’t decrease when the value of PM2.5 arrives at 100 and above.]
Point 12: The conclusion regarding public services also could use some more elaboration, especially since the survey questions on that issue do not directly address government responsibility for air quality.
Response 12: Accept your advice, we add some elaboration on why we made that conclusion on public service. [Because air quality is one of the key indicators for public environment, in ……]
Point 13: The policy implications do not necessarily relate to the results. If they do, further elaboration is needed here as well.
Response 13: Thank you very much. We add much more elaboration to the policy implication section, see the details in the new version. [First, citizens accurately perceive the quality of the air quality, which is something to which government at all levels of China should pay close attention, in other words, citizens have the capability to participate in the governance of public affairs, especially the air pollution control. Second, information on the public environment, especially regarding air quality, should be released in a timely manner, as this is something to by which citizens are getting more and more influenced. Third, and most importantly, the government should pay direct attention to citizens’ requests and expectations for clean air and a healthy environment instead of only economy development.]
Author Response File: 
Author Response.docx
Reviewer 2 Report
Overall
There are some language issues. Recommend professional language check.
Inconsistent referencing style, for example Atari, Luginaah and Fung (2009) and Oglesby et al. (2000). Check the journal referencing guideline.
There is a need for more thorough explanation about DIF issues and anchoring vignettes.
There are missing references in many places, for example, mechanism for estimating satisfaction regarding anchoring vignettes. There are also some statements lack of references.
Introduction
Background and problem discussion must be improved. The main issue is the writing flow.
DIF issues must be clarified. How and why five-level Likert scale causes such an issue? This would strengthen the problem discussion and the use of anchoring vignettes.
Good explanation about the research design and methods.
Literature review
DIF issues and anchoring vignettes must be clarified more thoroughly since it is one of the main contribution of your study that you propose this method in measuring satisfaction.
Missing references regarding anchoring vignettes, especially in relation to figure 1.
You used two vignettes, but in figure 1, you presented the example of four vignettes. Also, the logic you try to show with the figure is not correct (or at least not in line with what you really do in your study).
The relationship between actual and perceived air quality must be elaborated more before coming down to hypothesis.
Data collection and variable specification
It is unclear on how you developed the vignettes. You must explain vignette development. What are the constructs in both vignettes? Have the two vignettes been tested to ensure the satisfaction level?
You must be more specific about perception of air quality. This can be linked to the constructs of the vignettes. It is better if you can elaborate the constructs of perceived air quality in relation to the actual air quality (PM2.5, SO2, and dust emission concentrations). How do those of the actual air quality relate to the constructs of perceived air quality (i.e. period of blue sky, white clouds, and visibility)?
Results
The mechanism for estimating satisfaction should be explained in terms of two vignettes precisely as you use in this study, not just the general one from King and Wand.
The conclusion regarding PM2.5 (in connection with hypothesis 2) is somehow too naive. Together with the missing link between the constructs of actual air quality and the constructs of perceived air quality, it makes this conclusion very questionable. This sentence is too strong. "Citizens in china are more sensitive to PM2.5 than to other indicators...."
The CHOPIT model looks convincing. However, you should explain more about the differences between model 1 and 2.
Author Response
Response to Reviewer 2 Comments
Overall
Point 1: There are some language issues. Recommend professional language check.
Response 1: Thank you for your suggestions with great details. We already did the proofreading work. And we accept your advice to revise the article as best as we can.
Point 2: Inconsistent referencing style, for example Atari, Luginaah and Fung (2009) and Oglesby et al. (2000). Check the journal referencing guideline.
Response 2: Thank you for your suggestion with great details. Accept your advice, we unify the reference style according to sustainability.
Point 3: There is a need for more thorough explanation about DIF issues and anchoring vignettes.
Response 3: We add more explanation on DIF and anchoring vignettes.
Point 4: There are missing references in many places, for example, mechanism for estimating satisfaction regarding anchoring vignettes. There are also some statements lack of references.
Response 4: Thanks again. We check all the similar problems that might appear in the paper.
Introduction
Background and problem discussion must be improved. The main issue is the writing flow.
Point 5: DIF issues must be clarified. How and why five-level Likert scale causes such an issue? This would strengthen the problem discussion and the use of anchoring vignettes.
Good explanation about the research design and methods.
Response 5: Thanks for your great suggestions on introduction section. We really appreciate it and accept your advice. We add elaboration on “[How and why five-level Likert scale causes such an issue?” by holding the view that five-level Likert table to measure air pollution by assuming that each respondent is in the same context. Also we explain more on DIF appeared in the introduction section by adding sentence” defined as “the circumstance in which two individuals of similar ability do not have the same probability of answering a question in a particular way “to make it easy to understand]”.
Literature review
Point 6: DIF issues and anchoring vignettes must be clarified more thoroughly since it is one of the main contribution of your study that you propose this method in measuring satisfaction.
Missing references regarding anchoring vignettes, especially in relation to figure 1.
You used two vignettes, but in figure 1, you presented the example of four vignettes. Also, the logic you try to show with the figure is not correct (or at least not in line with what you really do in your study).
The relationship between actual and perceived air quality must be elaborated more before coming down to hypothesis.
Response 6: Thank you for your kind suggestions on literature review. We accept your advice and add more elaboration on DIF issues and anchoring vignettes method. In addition, figure 1 is made by authors themselves so reference is not needed. And we changed the figure 1 into one with only two vignettes.
Also, we elaborate more on the mechanism for the relationship between the actual and perceived air quality. Please see the revised version.
Data collection and variable specification
Point 7: It is unclear on how you developed the vignettes. You must explain vignette development. What are the constructs in both vignettes? Have the two vignettes been tested to ensure the satisfaction level?
Response 7: We accept your advice and add more elaboration on how we develop the two vignettes. What has been added are [ According to prior research, the anchors that can be used to attach the answers of different individuals should be the same standard scale. Although vignettes have been designed in political efficacy, self-reported health, and corruption perception, there are no vignettes that can be referred to in air pollution domain. Following the rule that anchors should create a context that can be understood without confusion, we designed two vignettes to anchor the respondents to a particular situation in order to correct their self-reported answers.]
In addition, we also tested the validity of the two vignettes using anchor package in R and the result is valid and strong indicating that the two vignettes we designed were appropriate for correcting the DIF problem in measuring the air quality satisfaction.
Point 8: You must be more specific about perception of air quality. This can be linked to the constructs of the vignettes. It is better if you can elaborate the constructs of perceived air quality in relation to the actual air quality (PM2.5, SO2, and dust emission concentrations). How do those of the actual air quality relate to the constructs of perceived air quality (i.e. period of blue sky, white clouds, and visibility)?
Response 8: Thanks for your kind suggestions. The two vignettes that we designed are based on the actual weather situation among the 17 cities in Shandong province. As we all know, some of the cities are close to the sea such as Qingdao, Weihai and Rizhao, whereas some cities far in the land, such as Jinan, Dezhou, there is significant variation in terms of weather condition. And the “blue sky, white cloud” is the general term that can be understand by each citizen despite of their age, gender, education level, et ct. therefore, the two vignettes construct the vignettes. And we also add the mechanism for the relationship between the actual and perceived air quality. [In other words, citizens are paying more attention to the effects of air quality, especially measured by PM2.5, on public health. Therefore, the air quality can be perceived by citizens accurately. However, figure 6 shows that citizens’ satisfaction with air quality won’t decrease when the value of PM2.5 arrives at 100 and above.]
Results
Point 9: The mechanism for estimating satisfaction should be explained in terms of two vignettes precisely as you use in this study, not just the general one from King and Wand.
Response 9: Estimating the air quality satisfaction by combining the two vignettes instead of directly asking is preferred since citizens’ view of air quality satisfaction can be varied a lot. Therefore, when respondents are put into the same air quality context where each one can have same understanding to what is clean air, it is reasonable that the satisfaction level can be compared with.
Point 10: The conclusion regarding PM2.5 (in connection with hypothesis 2) is somehow too naive. Together with the missing link between the constructs of actual air quality and the constructs of perceived air quality, it makes this conclusion very questionable. This sentence is too strong. "Citizens in china are more sensitive to PM2.5 than to other indicators...."
Response 10: Thank you very much for your advice. We add the link between actual and perceived air quality. [ In recent years, PM2.5, as a salient indicator of air quality, has been reported both by official and social media so that it is getting more and more attention by the public. Compared to SO2 and dust emission concentrations, PM2.5 has an impact on citizen’s air quality perception to a larger extent, which makes it clear that sustainable citizenship is becoming increasingly important in China. In other words, citizens are paying more attention to the effects of air quality, especially measured by PM2.5, on public health. Therefore, the air quality can be perceived by citizens accurately. However, figure 6 shows that citizens’ satisfaction with air quality won’t decrease when the value of PM2.5 arrives at 100 and above.]
Point 11: The CHOPIT model looks convincing. However, you should explain more about the differences between model 1 and 2.
Response 11: Thanks for your kind suggestions on model section. In this section, all of our findings are based on model1 and model2 in table 4. The very difference between the two models is that model 2 controlled the fixed effects at city level, whereas the model 1 wasn’t; in order to make it clear, we make more explanation on findings based on model1 and model2, see details from the revised version.
Author Response File: Author Response.docx
Reviewer 3 Report
Thank you for the opportunity to review this manuscript and learn about the relation between actual and perceived air quality and the role of public’s expectations of public service. The authors address a very relevant and interesting topic and the recommendations drawn from the research results are clear and straightforward. I only have some minor comments and suggestions:
First, the theoretical derivation of Hypothesis 2 relies on the two main perspectives, media framing and social construction. While I can clearly follow the media-framing-rationale, I am not quite convinced by the second one. Maybe the authors could describe this perspective a little clearer by adding two or three more sentences.
Second, even though I am a huge fan of figures, I am not quite sure if Figure 2 is suitable: One reason is that the authors only partially explain the model—leaving out the explanation for disconfirmation. I do understand that this part of the model is not the focus of the paper. However, in my opinion, the figure is misleading the reader. Not only because disconfirmation is displayed but not central to the paper. Also, because in the figure a mixed relationship between expectations and satisfaction is displayed, but the authors assume a negative relation between public’s expectation and satisfaction with air quality. I would recommend displaying only those parts of the model that are relevant to the present paper. Because these are the paths/relations that can be and are actually tested empirically.
Lastly, the authors cite Hopkins and King (2010) who demonstrated that “switching the question order so that self-assessments follow the vignettes primes respondents to define the response scale in a common way. In this case, priming is … a means of better communicating the question’s meaning.” I think the authors should discuss why they chose to do the opposite, i.e. asking the self-reported question first, followed by the vignette questions. I would guess that if authors had followed the recommendations by Hopkins and King, more measurement error could have been controlled for.
Author Response
Response to Reviewer 3 Comments
Thank you for the opportunity to review this manuscript and learn about the relation between actual and perceived air quality and the role of public’s expectations of public service. The authors address a very relevant and interesting topic and the recommendations drawn from the research results are clear and straightforward. I only have some minor comments and suggestions:
Point 1: First, the theoretical derivation of Hypothesis 2 relies on the two main perspectives, media framing and social construction. While I can clearly follow the media-framing-rationale, I am not quite convinced by the second one. Maybe the authors could describe this perspective a little clearer by adding two or three more sentences.
Response 1: Thanks for your advice. We add more elaboration on why social construction perspective can work for hypothesis 2. [In other words, subjective evaluations of air quality are the combined outcome of direct pollution-related experiences and a series of individual or localized contextual factors, such as physical health conditions and sensitivity, the neighborhood environment, and cultural and social senses of belongs]
Point 2: Second, even though I am a huge fan of figures, I am not quite sure if Figure 2 is suitable: One reason is that the authors only partially explain the model—leaving out the explanation for disconfirmation. I do understand that this part of the model is not the focus of the paper. However, in my opinion, the figure is misleading the reader. Not only because disconfirmation is displayed but not central to the paper. Also, because in the figure a mixed relationship between expectations and satisfaction is displayed, but the authors assume a negative relation between public’s expectation and satisfaction with air quality. I would recommend displaying only those parts of the model that are relevant to the present paper. Because these are the paths/relations that can be and are actually tested empirically.
Response 2: Thank you very much for the kind suggestion. We take your advice and remake the map only to show the difference of satisfaction with air quality before and after anchoring vignettes so that it can indicate clearly the variation among 17 cities in Shandong Province.
And we also make the meanings of figures as clear as we can by adding more elaboration on the relationship between the expectation and satisfaction with air quality.
Point 3: Lastly, the authors cite Hopkins and King (2010) who demonstrated that “switching the question order so that self-assessments follow the vignettes primes respondents to define the response scale in a common way. In this case, priming is … a means of better communicating the question’s meaning.” I think the authors should discuss why they chose to do the opposite, i.e. asking the self-reported question first, followed by the vignette questions. I would guess that if authors had followed the recommendations by Hopkins and King, more measurement error could have been controlled for.
Response 3: Thank you for your advice on vignettes method. Actually, we do follow what King and his authors did, namely ‘asking self-assessment first, then vignettes question’. The reason Hopkins and King (2010) would do the opposite is the need to demonstrate the method, trying to tell the readers the very difference that the order of questions asked can cause.
Thank you very much.
Author Response File: Author Response.docx
Reviewer 4 Report
In general the article is very well put together. I would suggest only minor changes:
1) Uniformization of all figures in the article, as they are not consistent.
2) Use of non colloured figures in the article, except, perhaps, for the maps.
3) It wouldn't hurt if the Discussion and Conclusion section would be increased in order to improve the overall balance of the article's structure.
Author Response
Response to Reviewer 4 Comments
in general the article is very well put together. I would suggest only minor changes:
Thank you very much.
Point 1: Uniformization of all figures in the article, as they are not consistent.
Response 1: According to your advice and requests from the journal, we unify all the figures.
Point 2: Use of non colloured figures in the article, except, perhaps, for the maps.
Response 2: Thanks again for your attention to the great details. According to your advice, we revised the article, see the details in the new version.
Point 3: It wouldn't hurt if the Discussion and Conclusion section would be increased in order to improve the overall balance of the article's structure.
Response 3: Thanks again. We add more elaboration to the discussion and conclusion section.
Author Response File: Author Response.docx
Round 2
Reviewer 1 Report
The revisions to this manuscript have sufficiently addressed my previous concerns, and I recommend accepting it.