Improving the Sustainability of Open Government Data
Round 1
Reviewer 1 Report
Thank you authors for the interesting research with a title "Improving the Sustainability of Open Government Data". Please find the several remarks for the improvement of the paper:
1. Newly literature source must be included in research literature review from the 2019 year;
2. In the research the sustainability aspects can be have more dimensions (Table 1, 2) representing analysed formal activity, macro and micro environments.
3. Please note that the several correction for the presented must be made before publishing, like in appendix file page 5 line 575; 576 and ect.
Author Response
1. Newly literature source must be included in research literature review from the 2019 year;
Response:
Thank you for your recommendation. The newly literature source from the 2019 had been added to this revision. They are reference [54] and [65].
2. In the research the sustainability aspects can be have more dimensions (Table 1, 2) representing analysed formal activity, macro and micro environments.
Response:
Thank you for your suggestions. This study adopts the OGD assessment draft framework proposed by ODI and New York University, and our research framework includes the environment, data, use, and impact dimensions. Environment is the external condition and data is the basis resource of sustainable OGD. By using data resources, the impacts on the politics, economy, society, will be created. The impacts that is the value of OGD will promote sustainable OGD.
As you said, there are other dimensions aspect sustainability of OGD, such as data security, privacy protection, and so on. Basing on this paper, the other dimensions influenced on sustainability of OGD could be considered in future work. These are discussed in conclusion section and future work.
3. Please note that the several correction for the presented must be made before publishing, like in appendix file page 5 line 575; 576 and ect.
Response:
Thank you for your recommendation. We have corrected these problems in the revision.
Reviewer 2 Report
COMMENTS AND SUGGESTIONS
The goal of the study is to construct a new evaluation model for the sustainability of OGD. The researchers use an interesting method with which they construct the evaluation model and explore the main causal factors (DANP). The authors do not claim that this method is new in itself.
A previous version of this article was reviewed by me (412800). The authors made several improvements to the article by a.o. adding a definition of sustainability, sustainable and sustainable development goals with the OGD and adding the questionnaire in the appendix. However, in my opinion the article, in its present form, still has issues, which lead to the decision to a major revision.
In the first place, it is somewhat surprising to find that the earlier version of the article lacked a definition of sustainable, while the central aim of the study is to provide an OGD sustainability evaluation model.
Secondly, the added explanation of sustainable in the introduction is difficult to follow, partly because the English formulation needs to be improved. The sustainability of OGD is that different from sustainable OGD? Is it the same? And the formulation is not always clear. For example, R90-R95 is a long and difficult to understand sentence. And what is meant by R95-R96 "So, sustainable ...". Is meant "So, sustainable OGD ..."? What exactly is meant by this? Sentence R97 “Data are the life-blood….” : another term for life-blood? See also R143: “that” should be removed? R90-91 "... is applied to ... that has been ...."? It is not clear from the passage R88-R109 what the difference is between OGD and Sustainable OGD. R98 - R104 are not about sustainable OGD but about OGD. Another example is R105-R106: What is meant by "OGD is not only a sustainable development goals [goal?] .."? R106: “Ways to improve sustainable OGD” or “Ways to improve sustainability or OGD” (like in the title).
Thirdly, the literature study does not sufficiently address existing literature. I advise the researchers, for example, to take the literature on OGD ecosystems more into consideration. See below some examples of possible additional literature. A question that arises is whether the evaluation factors are sufficiently complete for this exercise.
The numbering of the references is missing in the .pdf, so that the literature that was included in the article could not be checked against the reference list. The older article by Dawes, S. S. (2010) is quoted, but not the recent one by Dawes et al. (2016).
Literature examples
- International Working Conference on Transfer and Diffusion of IT TDIT 2014: Creating Value for All Through IT pp 62-82| Generating Sustainable Value from Open Data in a Sharing Society (Thorhildur Jetzek; Michel Avital; Niels Bjørn-Andersen).
- Future Internet 2012, 4(4), 900-928; https://doi.org/10.3390/fi4040900 Creating Open Government Ecosystems: A Research and Development Agenda (Teresa M. Harrison , Theresa A. Pardo and Meghan Cook).
- Zuiderwijk, A., Janssen, M., Choenni, S., Meijer, R., & Sheikh Alibaks, R. (2012). Sociotechnical impediments of open data. Electronic Journal of E-Government, 10(2), 156–172.
- Journal of Theoretical and Applied Electronic Commerce Research; ISSN 0718–1876 Electronic Version; VOL 9 / ISSUE 3 / SEPTEMBER 2014 / 45-58; © 2014 Universidad de Talca – Chile; Reconciling Contradictions of Open Data Regarding Transparency, Privacy, Security and Trust (Ronald Meijer, Peter Conradie and Sunil Choenni)
Author Response
The goal of the study is to construct a new evaluation model for the sustainability of OGD. The researchers use an interesting method with which they construct the evaluation model and explore the main causal factors (DANP). The authors do not claim that this method is new in itself.
A previous version of this article was reviewed by me (412800). The authors made several improvements to the article by a.o. adding a definition of sustainability, sustainable and sustainable development goals with the OGD and adding the questionnaire in the appendix. However, in my opinion the article, in its present form, still has issues, which lead to the decision to a major revision.
1.In the first place, it is somewhat surprising to find that the earlier version of the article lacked a definition of sustainable, while the central aim of the study is to provide an OGD sustainability evaluation model.
Response:
Thank you for your recommendation. We have added “2.1 Definition of sustainability of OGD” to section 2.
2.Secondly, the added explanation of sustainable in the introduction is difficult to follow, partly because the English formulation needs to be improved. The sustainability of OGD is that different from sustainable OGD? Is it the same? And the formulation is not always clear. For example, R90-R95 is a long and difficult to understand sentence. And what is meant by R95-R96 "So, sustainable ...". Is meant "So, sustainable OGD ..."? What exactly is meant by this? Sentence R97 “Data are the life-blood….” : another term for life-blood? See also R143: “that” should be removed? R90-91 "... is applied to ... that has been ...."? It is not clear from the passage R88-R109 what the difference is between OGD and Sustainable OGD. R98 - R104 are not about sustainable OGD but about OGD. Another example is R105-R106: What is meant by "OGD is not only a sustainable development goals [goal?] .."? R106: “Ways to improve sustainable OGD” or “Ways to improve sustainability or OGD” (like in the title).
Response:
Thank you for your valuable suggestions. In this paper, the sustainability of OGD is the same as sustainable OGD. We have modified these problems in R91-R104.
3.Thirdly, the literature study does not sufficiently address existing literature. I advise the researchers, for example, to take the literature on OGD ecosystems more into consideration. See below some examples of possible additional literature. A question that arises is whether the evaluation factors are sufficiently complete for this exercise.
The numbering of the references is missing in the .pdf, so that the literature that was included in the article could not be checked against the reference list. The older article by Dawes, S. S. (2010) is quoted, but not the recent one by Dawes et al. (2016).
Literature examples
- International Working Conference on Transfer and Diffusion of IT TDIT 2014: Creating Value for All Through IT pp 62-82| Generating Sustainable Value from Open Data in a Sharing Society (Thorhildur Jetzek; Michel Avital; Niels Bjørn-Andersen).
Response:
Thank you very much for your recommendations and the provided literature. We have used the literature on OGD ecosystems to explain our evaluation model. The details were described in R134-R162.
Reviewer 3 Report
Prior to the evaluation of OGD, the authors first needs to elaborate the concept of "the sustainability of OGD" with its importance, conceptual composition, and the relationships with other related concepts.It is very hard to link the current literature review on the sustainability of OGD with its framework.
Over the manuscript, the authors uses the different words to describe the purpose of the article: causal, influencing factors, determinants, and evaluation. Authors need to make clear on their purpose.
The authors need to specify the sample, the 10 experts, about their representation and the validation of their responses. Also, they need to explain how the method can work with the small number of the sample.
It is not clear how the authors draw the conclusion which is represented in Fg. 2 by their analysis. Weighting factors is different from building a model. The authors need more explanation on the relationships represented by the arrows in Fg. 2 and how the arrows can be drawn from the analysis.
Author Response
1. Prior to the evaluation of OGD, the authors first needs to elaborate the concept of "the sustainability of OGD" with its importance, conceptual composition, and the relationships with other related concepts. It is very hard to link the current literature review on the sustainability of OGD with its framework.
Response:
Thank you for your suggestions. We have added “2.1 Definition of sustainability of OGD” to section 2. The sustainability of OGD with its importance, conceptual composition, and the relationships with other related concepts has been strengthened in the revision. We hope this revision can clarify your concerns.
2. Over the manuscript, the authors uses the different words to describe the purpose of the article: causal, influencing factors, determinants, and evaluation. Authors need to make clear on their purpose.
Response:
Thank you for your recommendation. We have corrected these ambiguities in the revision.
3.The authors need to specify the sample, the 10 experts, about their representation and the validation of their responses. Also, they need to explain how the method can work with the small number of the sample.
Response:
Thank you for your recommendation. We interviewed 10 experts from different departments, which helps us to reflect the opening government data issues from different backgrounds. We verified the compatibility of their opinions and had concluded that they were mutually compatible. In addition, it is easy to lead to difficult convergence of opinions and failure to achieve consensus if too many experts are interviewed, and too few experts may result in unrepresentative results. We referred to some literatures, in which the number of experts interviewed ranged from 6 to 10. (Rezaei, et al., Lo, et al.,)
Rezaei, J., Hemmes, A., Tavasszy, L.Multi-criteria decision-making for complex bundling configurations in surface transportation of air freight. Journal of Air Transport Management.2016, 2, 1-11.
Lo, H.W., Liou, James J.H., Wang, H.S, Tsai, Y.S., An integrated model for solving problems in green supplier selection and order allocation. Journal of Cleaner Production 2018,190,339-352.
4.It is not clear how the authors draw the conclusion which is represented in Fg. 2 by their analysis. Weighting factors is different from building a model. The authors need more explanation on the relationships represented by the arrows in Fg. 2 and how the arrows can be drawn from the analysis.
Response:
Thank you for your remarks. Using this method introducing in reference 35, we established the major influence route between dimensions (showed in Figure 2). Figure 2 was drawn from the calculated results based on Table A3 and Table A4. For example, the arrow from Environment (D1) to Data (D4) because Environment (D1) has higher net influence degree (ri – ci) than that of Data (D4).
The major influence route between dimensions shows that “environment” (D1) is the primary influencing factor and the “impact” (D4) is the important affected factor. We have explained on R420-R425.
Round 2
Reviewer 2 Report
On the previous comment that the literature study does not sufficiently address existing literature, the article was improved by the authors. However the literature based explanation of the evaluation model, needs still to be improved. Insufficient attention is given to the possible impediments of open data (e.g. see reference A below). Moreover the description of the evaluation model does not take into account the possibility of contradictions between values, such as trust, transparency, privacy, and security, and Open Data policies. See for instance R196-197 “The richer the open data, the more values can be generated by reusing data, while protecting national security, trade secrets, and personal privacy [19,23-24,56-58]. For instance, in the article mentioned reference B below it is demonstrated that there is not such a straightforward effect – as R196-197 suggest – between for example richness of data and value creation. More in general, also article mentioned in reference C below demonstrates that while expectations high, there are some fundamental objections on the straightforward use of Big Data. This also holds for Open Data.
NB. All references mentioned were also included in my previous comment.
References:
A
Zuiderwijk, A., Janssen, M., Choenni, S., Meijer, R., & Sheikh Alibaks, R. (2012). Sociotechnical impediments of open data. Electronic Journal of E-Government, 10(2), 156–172.
B
Journal of Theoretical and Applied Electronic Commerce Research; ISSN 0718–1876 Electronic Version; VOL 9 / ISSUE 3 / SEPTEMBER 2014 / 45-58; © 2014 Universidad de Talca – Chile; Reconciling Contradictions of Open Data Regarding Transparency, Privacy, Security and Trust (Ronald Meijer, Peter Conradie and Sunil Choenni)
C
2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiquitous Computing & Communications, Big Data & Cloud Computing, Social Computing & Networking, Sustainable Computing & Communications. On the Usability of Big (Social) Data; (Sunil Choenni, Niels Netten, Mortaza S-Bargh, and Rochelle Choenni).
Author Response
Response:
Thank you for your recommendations. Opening and using data encounters numerous impediments such as availability and access, find ability, usability, understand ability, quality, metadata, and so on. We have modified the revised in R152-153 and included those references in the paper. Many thanks again.
Reference B mentions that transparency, trust, privacy, security as public values. In our study we defined that social impact, political impact, economic impact as OGD value. OGD value includes the public values. The “more value” in R165 means OGD value. We take into account the possibility of contradictions between values, such as trust, transparency, privacy, and security, and Open Data policies. We have modified the revised in R197-199.
Reference C distinguishes three major building blocks in big data, the data as input for analyses, the algorithms to analyze the data, and the models as output of the analyses. Only the outcomes of big data are correctly interpreted, usability of big data in the social domain can be created. The details are in R201-202.
Author Response File: 
Author Response.docx
This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.
Round 1
Reviewer 1 Report
Thank you for this study.
You use a lot of techniques and methodology. But it is not clear what you are actually investigating with all these methods?
I would suggest, that you include a real case study and show how your method is applied, so it will get clear what your evaluation system can be used for. And readers can see if your model makes sense.
You show very detailed results, but to me they make no sense, because I do not know the input data. It would be good, if you also present the input data of your study in the same detail. The questions of the questionnaire and the according (anonymized) answers would help very much in this case.
Reviewer 2 Report
COMMENTS AND SUGGESTIONS
The authors state that ways to improve the sustainability of OGD are an important and challenging issue. The goal of their study is to construct a new evaluation model for the sustainability of OGD The authors use a combination of Decision-Making Trial and Evaluation Laboratory method and Analytical Network Process method to construct an influential network relationship map.
The article seems to be linked to two thoughts: a story about a method and a story about a model intended for the evaluation of (sustainable) OGD and its results.
The method receives a lot of attention. If the article is regarded as a story about method, then it is not clear what is new and there is virtually no critical reflection, for example on the approach, the results and the like. In the section on methods, the authors do not make clear what is new in a scientific sense. This part does not seem to be more than a rather extensive description of an existing method. A discussion of the method used in the light of the application and the results found is missing. In this article it is not made clear what the added value would be of applying the method to a more qualitative method. With all due respect for the interviews: ten people are interviewed. Then why do we need a mathematical model approach? What is the added value? This is, in my opinion, not convincingly explained.
The authors state that this study is the first in the field of sustainable OGD (R85-87): The research has mainly focused on the evaluation of OGD projects, but few studies have focused on sustainable OGD. We have not found any research on finding the relationships among factors for the improvement of OGD sustainability.” However, the article does not sufficiently elaborate what is meant by sustainable. A definition of sustainability, sustainable and sustainable development goal etc. is lacking. And a justification of the literature review is missing.
There is no justification for the approach taken to the questionnaires and their implementation. What is meant by the "communication with experts" (R109). How are the dimensions and the criteria in the questionnaires operationalized?
See for instance R109-110: “According to the literature and communication with experts, improving OGD sustainability has four dimensions and criteria”. It is not justified how this finding has been derived. In particular the concept of sustainability is not defined.
A basic element of the study is the interview part with a total of ten respondents. However, in the article the authors do not adequately justify whether that group is sufficiently representative to come up with reliable substantive results. It can not be assessed whether the questionnaire has been set up in a valid way, because the accountability of its creation and structure is insufficient. There is no critical consideration. To what extent are the results sensitive to the composition of the group of respondents? Suppose, for example, that fewer or more Government respondents would have been present? Or more users? In other words, the question arises as to how representative the selection of the respondents is. This question is not covered in the article. The substantive results are now too much presented as absolute outcomes. In terms of content, the results are not sufficiently innovative in themselves: the results do not add enough to the insights of the existing literature on OGD.
SOME DETAILED COMMENT
R49-58. I do not fully understand the reasoning: the 2016 Global Open data Barometer Report places China in place 71 out of 114. The (Chinese) government promotes OGD, but apparently with a level of OGD that for China or for OGD in general as insufficient is considered to. And then a sentence further: "Thus, the aim of this study is to reveal the influencing factors and sustainable OGD, as well as propose strategies for improvement." This doesn’t seem a logical explanation. See also some paragraphs above R49-58: governments launch open data platforms (106 of the 193 UN member states) and countries have committed themselves to the OGP. In which case the authors state that "Governments have aroused much attention to the sustainability of OGD."
It remains vague what sustainabillity is except that there may be continuous attention (from governments) for OGD.
R97-99 I do not understand well this phrase.
R107 Proposal / Proposed / Else ?
R119-124 This is about development, but what is the relationship with sustainable / sustainability?
R123-124 Reference?
R198 Table 1 The evaluation factors for OGD : Or the evaluation factors for sustainable OGD?
R211 Questionnaires are used to collect data: for the first time questionnaires are discussed here. Furthermore, there is no justification. Where can the questionnaire be found? How are the dimensions and criteria operationalized?R329-332 - Questionnable? Reference?
R365-366 Is this a finding ? How can this be justified by the research?
