Next Article in Journal
Digitalisation of Creative Industries Fostered by Collaborative Governance: Public Innovation Labs in Gipuzkoa
Next Article in Special Issue
Risk Analysis for Short-Term Operation of the Power Generation in Cascade Reservoirs Considering Multivariate Reservoir Inflow Forecast Errors
Previous Article in Journal
Public Administration and Values Oriented to Sustainability: A Systematic Approach to the Literature
Previous Article in Special Issue
Research on Runoff Simulations Using Deep-Learning Methods
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:

Meta-Evaluation for the Evaluation of Environmental Management: Standards and Practices

School of Public Policy and Administration, Xi’an Jiaotong University, No. 28 Xianning Road West, Xi’an 710049, China
Author to whom correspondence should be addressed.
Sustainability 2021, 13(5), 2567;
Submission received: 25 January 2021 / Revised: 18 February 2021 / Accepted: 23 February 2021 / Published: 27 February 2021
(This article belongs to the Special Issue Urban Management Based on the Concept of Sustainable Development)


Environmental management plays a key role in the sustainable development of cities. The effectiveness of environmental management is commonly examined through some evaluation schemes, but the effectiveness of such evaluation schemes also needs to be verified. As such, in this study, meta-evaluation was creatively introduced into environmental management to improve the evaluation of environmental management (EEM). Meta-evaluation is the evaluation of an evaluation scheme, and can verify and enhance the evaluation quality. First, a set of new meta-evaluation standards and criteria was proposed based on the unique characteristics of environmental management, which made meta-evaluation standards more adaptable and effective. After that, the efficacy of the proposed meta-evaluation standards was verified through their application to two evaluation schemes used in different fields of EEM. Based on meta-evaluation, suggestions for improving these two EEM schemes were also provided. The major contributions of this study are to introduce meta-evaluation into environmental management, establish new evaluation standards, and examine the efficacy of EEM. The research showed that it is critical to carry out meta-evaluation before and/or after the implementation of EEM.

1. Introduction

Meta-evaluation is “the evaluations of evaluation” [1], which can identify the limitations of an evaluation process itself [2]. In the literature, researchers have claimed that the reliability of evaluation processes should be ensured and enhanced by carrying out meta-evaluation [3], and meta-evaluation has now entered an era of application [4]. However, to date, few studies have been done on the theme of meta-evaluation, especially in environmental science [5,6]. In fact, most evaluations of environmental management (EEM), which is to discover the issues in environmental management and promote sustainable development [7,8,9], are based on the presumption that there is no problem with the evaluation process itself.
EEM is a first-order evaluation method. According to the evaluation theories, first-order evaluation has many common problems, and the results of EEM may be problematic [1]. For example, in Shaanxi province, China, key ecological function areas were set to refer to areas that played an important role in water-holding, soil and water conservation, flood control and storage, windbreak and sand-fixation, and biodiversity maintenance [10]. The local government improves the ecological quality of these areas through national key ecological function areas management (NKEFAM). By evaluating NKEFAM, the provincial government has claimed that national key ecological function areas were well managed, and the ecological quality of almost all key ecological function areas was improved [11]. However, based on the analysis of remote sensing data, the ecological quality in some areas of Shaanxi is actually deteriorating [12,13]. Even the evaluators of NKEFAM admitted that there were some problems in the evaluation process. For example, there is administrative intervention into environmental monitoring, which makes it difficult to ensure the objectivity and accuracy of monitoring data [14]. Therefore, NKEFAM does not reflect the actual situation of the key ecological function areas, indicating that the quality of EEM has to be evaluated. Otherwise, severe losses may arise during the practice of environmental management [15]. As a systematic tool to control the quality of evaluation, meta-evaluation could identify the defects in EEM and improve the accuracy of environmental management. As such, in order to improve the quality of EEM, meta-evaluation should be introduced into the evaluation process of environmental management.
Many researchers have found that evaluators in many cases showed a low familiarity with the evaluation standards, so that “what standards should be used by evaluators” becomes vital and has to be well addressed in meta-evaluation design [6]. Therefore, in this paper, to introduce meta-evaluation into environmental management, specified meta-evaluation standards that can be used in practice are first established. The proposed meta-evaluation standards are then applied in two cases with respect to two different fields of EEM. By carrying out meta-evaluation for each case from many perspectives, the possible problems involved in the two evaluation schemes are identified, and the efficacy of meta-evaluation standards is demonstrated. In short, this research aimed to introduce meta-evaluation, and verify its efficacy in environmental management.
The remainder of this paper is outlined as follows. In Section 2, the background of meta-evaluation is introduced. The meta-evaluation standards are designed in Section 3. In Section 4, the applications of meta-evaluation in practice are provided. The conclusions are drawn in Section 5.

2. Background of Meta-Evaluation

It is common that some defects exist in the evaluation process itself. Stufflebeam, the former chairman of the Joint Committee on Standards for Educational Evaluation (JCSEE), claimed that “many things can and often do go wrong in evaluation work” [16]. Incorrect evaluation reports and unreliable evaluators could mislead the users of evaluation results and incur great losses. As such, evaluating evaluation process becomes necessary.
Meta-evaluation was first proposed by Scriven half a century ago [1]. Its intuitive definition is to evaluate an evaluation, including the evaluation of an evaluation system or evaluation design [1]. “Meta-evaluation, only applies to evaluations; is frequently (and properly) applied to just one at a time; may assess an evaluand that is entirely qualitative; and refers to any process of evaluating it, or them, or the product of that process.” [17]. Through meta-evaluation, the potential problems of evaluation, such as bias, administrative difficulty, and technical error, can be examined [16]. Meta-evaluation can also provide a backward-driving mechanism. Namely, meta-evaluation can improve the quality of evaluation, and further enhance the subject of the evaluation [2]. The procedure of meta-evaluation mainly includes establishing the evaluation team, confirming the meta-evaluation standards, collecting information, and judging the compliance of meta-evaluation standards [18]. It should be pointed out that the meta-evaluation team must be able to represent the interests of stakeholders, which is the key to the implementation of meta-evaluation [2,16].
Till now, meta-evaluation has been successfully applied in some areas, such as the performance assessment system of Teach for American [18], the operating mechanism of universities [19], the evaluation system of higher education institutions [20], the policy on rural teacher management systems [2], and the maritime traffic safety evaluation scheme [21]. Since the JCSEE program evaluation standards can be directly used in similar fields [2,22], most of the previous meta-evaluation research has focused on educational areas. The JCSEE program evaluation standards contain a couple of standards, and each standard includes many criteria. The specific criteria can be found on the official website of JCSEE [23]. These criteria can be used to evaluate the evaluation programs of education from many angles, such as the value of the programs, the evaluation ethics, and the responsibilities of the evaluation participants. Some of these standards are common. For example, the reliability and validity of the information used for evaluation have to be tested in any evaluation program. JCSEE was founded in 1975. It has more than 2 million members and has been recognized by 12–15 professional organizations [18]. Recently, the JCSEE program evaluation standards have also been recognized by the authorities, such as the American Evaluation Association, and have been recommended as the guideline for designing other evaluation standards [3,24].
With the increasing importance of the sustainable development of the environment, the application of EEM is becoming more and more frequent. However, there is still a lack of effective meta-evaluation tools in environmental management. Since the JCSEE program evaluation standards have been tested by authoritative institutions [17], it provides a reference for the establishment of EEM meta-evaluation standards, and makes it possible to apply meta-evaluation in environmental management.
Although it is feasible to establish the meta-evaluation standards for EEM with reference to the JCSEE program evaluation standards, they cannot be simply copied and applied to EEM since the JCSEE program evaluation standards are formulated for the evaluation of education. Therefore, new meta-evaluation standards for EEM should be established and adjusted to increase the efficacy according to the unique characteristics of environmental management [16]. This work is a useful attempt to extend the application of meta-evaluation to the new area of environmental management, and provides a reliable basis for guiding related environmental management practices.

3. Methodology

3.1. Study Design

The standards of meta-evaluation must be widely recognized by the stakeholders, which is the starting point for the development of a meta-evaluation method [16]. In order to improve the acceptability of the meta-evaluation standards and make them represent the interests of all stakeholders, an evaluation team, including environmental experts, environmental officers, residents, members of an environmental non-government organizations, and government employees (participating in the entire evaluation process), has to be organized. The evaluation process consists of five steps:
Step 1: The recommendations for adjustments to EEM, based on the JCSEE program evaluation standards, are made, which will be reviewed and confirmed by the evaluation team;
Step 2: The meta-evaluation standards are established for EEM, which will be reviewed and confirmed by the evaluation team;
Step 3: The evaluation team presents the results of relative importance among criteria, and the corresponding criterion weights are assigned according to the evaluation’s results;
Step 4: The evaluation schemes are analyzed based on meta-evaluation standards. In this process, environmental experts will confirm whether the criteria are achieved or not;
Step 5: The results of the meta-evaluation will be further studied to figure out the problems in the evaluation schemes. Suggestions for improving EEM will then be provided.

3.2. The Standards of Meta-Evaluation

At present, the JCSEE program evaluation standards are the basis for formulating other standards, and have been recognized by the professional evaluation organization [24]. They are now in their third edition and are still under development [25]. JCSEE has always encouraged its users to make comments and suggestions on their own applications [18,19]. Motivated by the JCSEE program evaluation standards, in order to improve the evaluation schemes and enhance evaluation ability, especially as regards the EEM, we developed the following evaluation standards.

3.2.1. Utility Standard

The utility standard is designed to ensure that an evaluation meets the information needs of the prospective users [18]. Since the goal of EEM is to improve environmental performance, prospective users need credible environmental recommendations. However, there is no corresponding criterion in the JCSEE program evaluation standards, which focused more on the field of education [16]. As such, we propose a new criterion related to the results of the EEM, which could provide effective recommendations to protect the environment. As human activities often cause environmental changes [26,27], humans have the responsibility to protect the environment. Therefore, this criterion was named “human responsibility”.

3.2.2. Feasibility Standard

The feasibility standard requires that an evaluation must be realistic, prudent, and comprehensive [18]. However, an important fact is that the continuity of evaluation is often overlooked. In practice, the evaluation process may take a couple of weeks or sometimes need to continue intermittently. For example, the government requires that the evaluation of NKEFAM in Shaanxi must be conducted every year. The importance of continuous evaluation has been emphasized by more and more researchers [28]. For example, if the health and safety of the evaluator is at risk due to the lack of relevant provisions, the evaluation may be terminated in the next round. Since this type of evaluation cannot be applied continuously, the evaluation accuracy may be degraded. Therefore, the continuity is added to the meta-evaluation standards since it is a second-order evaluation [16].
The criteria “health assurance” and “collaboration mechanism” are designed to ensure the continuity of evaluation. As mentioned earlier, “health assurance” is set to ensure that the health and safety status of evaluators is not in danger (for example, near chemical plants or radioactive sources). The “collaboration mechanism” is to facilitate the next round of evaluation [29]. The measurement of these two criteria is mainly based on the design of the evaluation scheme. The evaluation scheme should have clear health and safety provisions, and an effective evaluator mechanism for cooperation.

3.2.3. Propriety Standard

The propriety standard supports proper, fair, right and just evaluations [23]. In EEM, the evaluators may directly or indirectly cause a negative impact on the environment [30,31]. This is a situation that should be avoided, but it is often ignored when designing the evaluation scheme. For example, a lot of engineering geological drilling is required in the evaluation process of groundwater resource management. If the drilling is not qualified or cannot be restored, it will bring new risks to the safety of groundwater resources. Therefore, the criterion “respect to the environment” was proposed in the propriety standard, which indicates that the EEM process should reduce or even have no negative impact on the environment.
The designs of the above criteria, aiming to improve the EEM, have been approved by the evaluation team. They are based on the possible interaction between the evaluator and the environment, and the interaction between the evaluators.
Note that the process and the results of EEM emphasize the interaction between humans and the environment, which is a major difference between EEM and other evaluation scenarios [32]. During EEM, the evaluator may have an impact on the environment, while at the same time the environment may also have an impact on the evaluator. The EEM results mainly represent the impact of the evaluator on the environment. In addition, the interaction among evaluators may also indirectly affect the environment, and in part determines the quality of EEM. Therefore, the aforementioned criteria are proposed to reflect all these possible interactions, and could well match the unique characteristics of environmental management, as shown in Figure 1.

3.2.4. Summary of the Standards

Besides the new criteria, we selected some widely applicable standards from the JCSEE program evaluation standards to form a complete meta-evaluation standard for EEM. Table 1 summarizes all the standards used for the meta-evaluation of EEM, which have been approved by the evaluation team.

3.2.5. Weights of Criteria

Since it is difficult for an evaluation scheme to meet all the meta-evaluation standards, some important criteria have to be taken into account. Meanwhile, to facilitate the use of meta-evaluation standards in practice, the priority of the criteria needs to be considered and the weights of the criteria should be calculated. In this paper, the analytic hierarchy process was used to assign weights to the criterion system. The analytic hierarchy process is a flexible and practical multi-criteria decision-making method, which can estimate the criteria weights by pairwise comparisons, and has been successfully applied to environmental management [33]. According to the previous study [34], the calculation process is as follows:
First, the relative importance of any two sub-criteria under a higher criterion (criterion A is more important than criterion B, or criterion A is less important than criterion B) can be obtained through the discussion of the evaluation team.
Second, according to the discussion results of the evaluation team, the Saaty 1–9 scaling method (1, equally important; 9, extremely important), as shown in Table 2, was used to obtain the quantitative value ( b i j ) of the relative importance among criteria, and as such the judgment matrix could be formed, as shown in Appendix A.
Third, the largest eigenvalue ( λ m a x ) and the corresponding eigenvector ( w i ) of the judgment matrix are calculated. It should be pointed out that each element of w i is the required weight value in a certain hierarchy. The equation is as follows:
w i = i , j = 1 2 b i j 2
where   b i j is the quantitative value of the relative importance among criteria.
Fourth, the weight coefficient ( a i ) can be obtained by normalizing w i as
a i = w i w i
Fifth, based on the above calculation, λ max can be obtained as
λ max = ( j = 1 n a 1 b 1 j a 1 + j = 1 n a 2 b 2 j a 2 + j = 1 n a 3 b 3 j a 3 + · · · + j = 1 n a n b n j a n ) * 1 n
where   n is the rank of the judgment matrix corresponding to the criterion.
Finally, it is necessary to check the consistency of each individual criterion and of the overall criteria. The equation is as follows:
C R = ( λ max n ) / ( n 1 ) R I
C R = the proportion of consistency;
R I = the average random consistency.
The statistical tool MATLAB 7.0 was used to calculate the previously mentioned variables, which are shown in Table 3.

3.3. Sample

In our study, two types of evaluation schemes in two different fields of EEM were taken as application cases. Each case has been meta-evaluated from a different perspective to reveal the possible problems in the schemes. The comprehensive efficacy of the meta-evaluation standards is also demonstrated.

3.3.1. Urban Sewage Treatment Management Evaluation Scheme

Urban sewage treatment management (USTM) is highly related to the sustainability of city development [36,37], and the performance of USTM has received great attention from the public and the local government. In addition, the Ministry of Housing and Urban Rural Development of China has requested provinces conduct an annual evaluation of USTM since 2010 [38]. Therefore, the evaluation data on USTM are relatively complete, and are eligible for meta-evaluation.
Besides this, the sustainable planning of water resources is an important issue for water resource conservation [39]. From a long-term perspective, water shortage may be a problem that all regions have to face. Currently, the USTM’s sustainability evaluation scheme is only aimed at alleviating water shortages in water-scarce areas. However, the potential contradiction between water supply and demand in water-rich areas should also be considered in the evaluation scheme [40]. As such, meta-evaluation should be applied to evaluate whether the water resources evaluation scheme in all regions carries the concept of sustainable development, regardless of whether the region is rich in water resources or not.
To this end, as shown in Figure 2, the evaluation schemes of USTM in three provinces in China, Guangxi, Shaanxi, and Inner Mongolia, are selected for meta-evaluation, since they are located in the same longitude and have similar GDP levels [41]. The information of the evaluation scheme comes from the official website of the local government [42,43,44]. (Note that the implementation date of the above USTM scheme was 2010, while the official announcement was later than that.)

3.3.2. National Key Ecological Function Areas’ Management Evaluation Scheme

The management of national key ecological function areas is very important to Chinese national ecological security [46]. In 2010, the State Council of China identified 25 national key ecological function zones, two of which (the relevant areas of the Loess Plateau and Qinling-Daba Mountains) are located in Shaanxi province. According to the requirements of the central government, Shaanxi has issued an evaluation scheme of NKEFAM [47]. However, some research-based evidence has questioned the process and its conclusions [12,13,14]. Therefore, we apply a meta-evaluation to reveal the possible problems in the NKEFAM evaluation scheme.

3.4. Meta-Evaluation

The effectiveness of the Delphi method could indicate whether each group can make correct judgements [48]. Therefore, in order to reduce the response error, it was used for meta-evaluation, as shown in Figure 3.
Scholars from environmental departments at different universities were selected as environmental experts by the evaluation team. The evaluation schemes were sent anonymously to five environmental experts, who conducted independent analyses according to the standards of meta-evaluation. Consistent evaluation results from the meta-evaluation should be retained. If there is disagreement, more than half of the same evaluation results will be retained. If not, the evaluation schemes will be sent to other experts until an acceptable evaluation result is obtained. Finally, the results of the meta-evaluation are obtained.

4. Results and Discussion

The utility standard is the most important part of the meta-evaluation standards. The reasons for this are as follows: on the one hand, the maximum weight was assigned to the utility standard by the evaluation team (see Table 3); on the other hand, researchers believed that the utility standard should be ensured prior to the feasibility, propriety, and accuracy standards [18]. Regardless of the performance of other standards, the lack of utility of EEM means that it cannot bring any changes to environmental management. Therefore, the poor performance of the utility standard directly leads to the failure of the EEM scheme. According to the previous studies, when more than half of the utility criteria are not met, it is considered that the utility standard has not been met [2,18]. However, this measurement does not take into account the different weights of each criterion. In this study, based on the assigned weights, if the score of the utility standard is less than half, it is considered to have failed, and the relevant EEM scheme will be deemed to have serious defects.

4.1. The Meta-Evaluation for Urban Sewage Treatment Management Evaluation Scheme

The meta-evaluation results for the USTM evaluation schemes of the Guangxi, Shaanxi, and Inner Mongolia provinces are shown in Table 4.
In overall, it was found that all three provinces have neglected some meta-evaluation criteria, including stakeholder identification, information scope, political viability, cost effectiveness, conflict of interest, valid and reliable information, justified conclusions, respect to the environment, and collaboration mechanism. Among them, some are important, such as stakeholder identification. There is no doubt that the public is an important stakeholder in urban wastewater management [49]. The lack of public participation will lead to many problems: (i) the conclusion drawn from the evaluation may be unfair; (ii) the monitoring of the evaluation may be questionable; and (iii) the proposed strategy for improving urban sewage treatment may not meet the actual needs of the society. It is easy to find that problems of sewage treatment in these three provinces have existed for a long time [50,51,52,53]. Note that currently, the involvement of extensive public participation to promote environmental conservation has been widely recognized [54]. In this study, meta-evaluation provided a further scientific basis for supporting public participation in environmental conservation. Therefore, if the criterion “stakeholder identification” is not emphasized in the evaluation scheme, such a scheme should be considered to have serious shortcomings. The government should encourage public participation in EEM.
Some criteria have been taken into account in the three provinces, including values identification, practical procedures, formal agreements, complete and fair evaluation, fiscal responsibility, program documentation, defensible information sources, analysis of quantitative information, and human responsibility. However, the number of these criteria did not exceed one third of the total, and most criteria were not considered (see Table 4). The evaluation schemes of USTM in Guangxi, Shaanxi and Inner Mongolia are internal, administrative, and bottom-up. The department of Housing and Urban–Rural Development of the three provinces required their subordinate units to evaluate the USTM under administrative orders, and then submit the evaluation results to them [42,43,44]. As a result, the evaluation schemes focus on the operational steps, responsibilities, completed conclusions and information records. Although such an approach can ensure the completion of the EEM activities, it has many weak points, as follows: (i) the internal evaluation lacks public participation and insufficient information dissemination; (ii) evaluation in the form of administrative order may make it difficult to ensure the personal dignity of the lower-level evaluators during the evaluation process; (iii) bottom-up evaluation may cause the evaluator to become a mere information transmitter rather than an evaluator; and (iv) the continuity of evaluation is limited. In addition, the final evaluation results, based on the self-evaluation of the lower-level administrative departments, may be unreliable.
In order to better understand our conclusions, we have interviewed some evaluators. It was found that (i) most evaluators treated the evaluation information as the internal information prescribed by the higher-level authorities, which cannot be released; (ii) most evaluators cared more about whether they have completed the assigned tasks than about the reliability of the evaluation information and the environmental concerns; and (iii) a lack of cooperation among the evaluators made most of them unwilling to conduct the next round of evaluation. These problems are further explored and confirmed by our meta-evaluation, particularly concerning the lack of consideration of the evaluation criteria “respect to the environment” and “collaboration mechanism”. Therefore, the EEM meta-evaluation standards we designed are reliable for problem identification. To solve these problems, the integration of “service orientation” and the “evaluation continuity” should be strengthened, and the administration management approach should be improved. Introducing multiple evaluation subjects, peer review, and random sampling evaluation are feasible solutions for administration, which can together improve the quality of EEM.
The problems mentioned above are common to the three provinces. In order to observe the scheme quality more intuitively and to facilitate a comparison among provinces, we calculated the scores of the three provinces’ meta-evaluation standards based on the criterion’s weight, as shown in Figure 4.
Through meta-evaluation, the utility standard of Guangxi scored less than half (a score < 23.34), which shows that Guangxi’ evaluation scheme of USTM has serious defects. The evaluation scheme of Shaanxi is qualified (score of the utility standard = 25.85), and the evaluation scheme in Inner Mongolia performed even better (score of the utility standard = 37.76). In addition, one finding worth noting is that the more abundant water resources in the province are, the worse the evaluation scheme of the USTM will be. Guangxi has abundant water resources, but its meta-evaluation scores are the lowest among the three provinces. This demonstrated that the abundance of natural resources may cause local people to lack an awareness of sustainable development. Researchers stated that a lack of natural replenishment could threaten the sustainability of water resources [55]. Studies found that sustainable development has been highly down-valued, or even overlooked, in natural resource-endowed areas [56]. Without awareness of sustainable development, even the abundant resources will eventually be exhausted [40]. Therefore, the efficiency of the natural resource management in various regions should be improved, especially in those regions with rich natural resources, in order to maintain sustainable development.
In summary, according to the meta-evaluation, it was found that there are many problems with the existing evaluation schemes of USTM in the Guangxi, Shaanxi, and Inner Mongolia provinces, since each province only obtained a low score (the full score is 100). In particular, Guangxi had the lowest score. According to the backward mechanism of meta-evaluation [2], we could see that the performance of evaluation directly determines the effect of USTM, which further determines the performance of urban sewage treatment. In other words, the results of the meta-evaluation indirectly reflected the performance of urban sewage treatment. Meta-evaluation has played an important role in improving the accuracy of EEM. In order to examine this effect, we demonstrated the situation of untreated urban sewage in the three provinces [57], as shown in Figure 5.
Since 2010, after the Ministry of housing and urban rural development of China launched the urban sewage management evaluation [38], the proportion of untreated urban sewage in the three provinces has declined. However, to date, the effect of urban sewage treatment is still not ideal. For example, although the untreated rate of urban sewage in Inner Mongolia dropped to about 5% in 2018, there were still over 17,200,000 cubic meters of urban sewage left untreated [57]. This clearly showed that the quality of USTM in three provinces had to be improved, which was one of the conclusions drawn from our meta-evaluation. Moreover, there were differences in sewage treatment effect among the provinces. Guangxi had the largest number of sewage treatment plants among the three provinces [57], but the efficiency of urban sewage treatment was the lowest. This indicated that the effect of urban sewage management in Guangxi was the lowest, which is also consistent with our meta-evaluation results.

4.2. The Meta-Evaluation for National Key Ecological Function Areas Management Evaluation Scheme

In this case, meta-evaluation was used to evaluate EEM from the perspective of the evaluation process. According to the guideline of JCSEE [18], the evaluation process contains eight sections, including defining evaluation issues, collecting information, analyzing information, proposing an evaluation report, making an evaluation budget, making an evaluation agreement, managing an evaluation, and hiring an evaluator. Each section corresponds to specific meta-evaluation criteria. This means these criteria can be used to evaluate part of the evaluation process. As such, a matrix of the meta-evaluation criteria and the evaluation process is obtained, and a meta-evaluation of the evaluation scheme of NKEFAM in Shaanxi is carried out. The results are shown in Table 5.
After reviewing the evaluation process, it was found that the evaluation scheme of collecting and analyzing information was relatively good. However, there were obvious deficiencies in the work of managing evaluation and hiring an evaluator, since most of the corresponding criteria were not met. Without effective management, evaluation will be passive and result in blindness in practice. This is evidenced by the lack of criteria and poor work in defining evaluation issues. One of the problems of the Shaanxi NKEFAM evaluation program came from the evaluators, who worked in government departments. Due to the awareness of stakeholders, the evaluators intended to demonstrate the positive effects of public services [58]. This led to unreliable evaluation data and reports (lack of criteria of “valid and reliable information” and “impartial reporting”). This is consistent with the findings of Ding et al. [14]. In addition, the budget evaluation should be improved. Insufficient investment can also limit the effectiveness of the evaluation. Faced with these challenges, we should first strengthen the management of the evaluation system and promote the requirements as regards the qualifications of the evaluators.
In summary, the reliability and practicability of the meta-evaluation have been verified by the above two cases. Since meta-evaluation could discover important problems in environmental management and put forward targeted improvement measures, it can be widely used in the EEM field. In particular, carrying out meta-evaluation before implementing an evaluation scheme can provide diagnostic feedback to improve the EEM activities.

5. Conclusions

This paper creatively introduced meta-evaluation into environmental management to ensure the quality of the evaluation schemes and promote the sustainability of city development. In order make meta-evaluation applicable, we have established the meta-evaluation standards, including utility, feasibility, propriety, and accuracy. In addition, an analytic hierarchy process was used to assign weights to the criterion system. The proposed meta-evaluation standards are shown to be practical and feasible. To demonstrate their applications in practice, two types of evaluation schemes used in two different fields of EEM have been analyzed.
For the evaluation schemes of USTM in Guangxi, Shaanxi, and Inner Mongolia, the following problems were discovered: (i) the public is not treated as a stakeholder in the evaluation schemes of USTM in all three provinces; (ii) all the evaluation schemes used in three provinces are internal, administrative and bottom-up; (iii) the continuity of evaluation is restricted; and (iv) the more abundant the water resources are, the worse the evaluation scheme of USTM will be. These major problems resulted in a series of further problems. For example, the evaluation conclusion may be unfair, the evaluation cannot be properly monitored, the evaluation is not known, and there is no cooperation mechanism among the evaluators. These problems indicate that the management effect of urban sewage treatment is far from perfect, and the urban sewage treatment in the three provinces has low efficiency. These conclusions are supported by the consistency among the meta-evaluation outcomes, the survey results, and the empirical data. The proposed potential solutions include the following: First, the criterion “stakeholder identification” should be added into the evaluation scheme, and the government should encourage public participation in EEM process. It is recommended that the environmental management department set up an evaluation committee and invite public representatives to participate in the evaluation and monitoring process as committee members. Second, the criteria “service orientation” and the “evaluation continuity” must also be considered. The more feasible suggestions are to strengthen the training of evaluators and allow the evaluators to participate in the formulation of the rules of the cooperation mechanism in the evaluation scheme. The reliability of EEM can also be improved by introducing multiple evaluation subjects, peer reviews, and random sampling evaluations. Third, the government should cultivate the concept of sustainable development [59], especially in resource-rich areas. Since green finance plays an important role in the environmental management control system [60,61], the government should effectively encourage green financial services and guide the green production behavior of enterprises so as to promote the achievement of sustainable development goals.
For the evaluation scheme of NKEFAM in Shaanxi province, it was found that the evaluation scheme for collecting and analyzing information is relatively good. However, the methods of managing evaluation and hiring evaluators need further improvement. In addition, the budget evaluation should be improved as well. In order to solve these problems, it is necessary to strengthen the management of the evaluation system, such as by establishing a management evaluation system platform and having dedicating professionals to manage it. In addition, EEM should set higher standard for evaluators through institutionalized means, such as making rules and regulations.
In total, the main contribution of this study is to introduce meta-evaluation into environmental management, and examine its effectiveness in practice. Meta-evaluation can provide diagnostic feedback to improve the EEM activities before actually implementing the evaluation, and can help users verify the reliability of the results after adopting the corresponding evaluation scheme.
Note that the meta-evaluation standards we proposed are applicable to general EEM. However, for some specific EEMs, it is possible that certain criteria may not be applicable and should be removed, while other new criteria need to be developed. In addition, we found that the professional competence and reliability of the evaluators is an issue worthy of in-depth study. Future meta-evaluation research on EEM should pay more attention to the evaluation of evaluators’ qualifications and the inspection of the operation process. Considering the relevance of encouraging green finance in environmental management, it would be valuable to establish financial evaluation criteria in the meta-evaluation standards, in order to promote sustainable development.

Author Contributions

Conceptualization, J.M. and Z.Y.; methodology, Z.Y. and J.M.; validation, J.M., Z.Y., and Z.G.; formal analysis, J.M. and Z.Y.; data collection, Z.Y. and Z.G.; writing—original draft preparation, Z.Y.; writing—review and editing, J.M.; visualization, Z.Y. and Z.G.; supervision, J.M.; funding acquisition, J.M. All authors have read and agreed to the published version of the manuscript.


This research was funded by Shaanxi Provincial Federation of Social Sciences, grant number 2020Z193 and Shaanxi Provincial Social Sciences Fund, grant number 2020R025.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

All the data are publicly accessible, included in the manuscript and accessible on the local governments’ official websites for the consulted provinces, as well as in the press.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table A1. The result of the judgment matrix of utility, feasibility, propriety, accuracy, and sustainability.
Table A1. The result of the judgment matrix of utility, feasibility, propriety, accuracy, and sustainability.
Table A2. The result of the judgment matrix of utility criteria.
Table A2. The result of the judgment matrix of utility criteria.
Table A3. The result of the judgment matrix of feasibility criteria.
Table A3. The result of the judgment matrix of feasibility criteria.
Table A4. The result of the judgment matrix of propriety criteria.
Table A4. The result of the judgment matrix of propriety criteria.
Table A5. The result of the judgment matrix of accuracy criteria.
Table A5. The result of the judgment matrix of accuracy criteria.


  1. Scriven, M. An introduction to meta-evaluation. Educ. Prod. Rep. 1969, 2, 36–38. [Google Scholar]
  2. Yin, Z.Y.; Wang, B. Conception of Implementing Meta-Evaluation of Rural Teacher’s Policies: Objective and Strategies. J. Teach Educ. 2019, 6, 71–77. [Google Scholar]
  3. Stufflebeam, D.L.; Shinkfield, A.J. Evaluation Theory, Models, and Applications; Jossey Bass: San Francisco, CA, USA, 2014; p. 635. [Google Scholar]
  4. Jacobs, S.; Affrodegon, W.S. Conducting quality evaluations: Four generations of meta-evaluation. Spaz. Filos. 2015, 13, 165–175. [Google Scholar]
  5. Coryn, C.L.S.; Wilson, L.N.; Westine, C.D.; Hobson, K.A.; Ozeki, S.; Fiekowsky, E.L.; Greenman, G.D., II; Schrӧter, D.C. A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. Am. J. Eval. 2017, 38, 329–347. [Google Scholar] [CrossRef]
  6. Harnar, M.A.; Hillman, J.A.; Endres, C.L.; Snow, J.Z. Internal Formative Meta-Evaluation: Assuring Quality in Evaluation Practice. Am. J. Eval. 2020, 41, 603–613. [Google Scholar] [CrossRef]
  7. Fadly, D. Greening Industry in Vietnam: Environmental Management Standards and Resource Efficiency in SMEs. Sustainability 2020, 12, 7455. [Google Scholar] [CrossRef]
  8. Herghiligiu, I.V.; Robu, I.-B.; Pislaru, M.; Vilcu, A.; Asandului, A.L.; Avasilcăi, S.; Balan, C. Sustainable Environmental Management System Integration and Business Performance: A Balance Assessment Approach Using Fuzzy Logic. Sustainability 2019, 11, 5311. [Google Scholar] [CrossRef] [Green Version]
  9. Maleki, R.; Atabi, F.; Jozi, S.A.; Arjomandi, R.; Mansouri, N. Sustainable Environmental Management using Life Cycle Sus-tainability Assessment Model in Petrochemical Industry. Pollution 2020, 6, 337–351. [Google Scholar]
  10. Website of the Ministry of Ecology and Environment of the People’s Republic of China. Available online: (accessed on 15 February 2021).
  11. Website of the Department of Environmental Monitoring of Shaanxi. Available online: (accessed on 12 October 2020).
  12. Song, H.M.; Xue, L. Dynamic monitoring and analysis of ecological environment in Weinan City, Northwest China based on RSEI model. Ying Yong Sheng Tai Xue Bao = J. Appl. Ecol. 2016, 27, 3913–3919. [Google Scholar]
  13. Yi, L. Ecological environment impact assessment of Yulin City Based on 3S technology. Master’s Thesis, Shaanxi Normal University, Xi’an, Shaanxi, China, 2014. [Google Scholar]
  14. Ding, Q.; Li, Q.; Liao, H.B. Environmental Monitoring System Construction in Shaanxi Province from the View of Eco-environmental Quality Assessment of National Key Ecological Function Areas. Environ. Prot. 2014, 42, 67–68. [Google Scholar]
  15. De Zwart, F. Administrative Practice and Rational Inquiry in Postmodern Public Administration Theory. Adm. Soc. 2002, 34, 482–498. [Google Scholar] [CrossRef]
  16. Stufflebeam, D.L. Meta-Evaluation. J. MultiDiscip. Eval. 2011, 7, 99–158. [Google Scholar]
  17. Scriven, M. Meta-Evaluation Revisited. J. MultiDiscip. Eval. 2009, 6, iii–viii. [Google Scholar]
  18. Stufflebeam, D.L.; Madaus, G.F.; Kellaghan, T. Evaluation Models: Viewpoints on Educational and Human Services Evaluation, 2nd ed.; Springer: Berlin, Germany, 2011. [Google Scholar]
  19. Arias, S.A.; Naranjo-Lopez, G.; Molina, V.H.; Gomez, H.F. Work in Progress: First steps to university meta evaluation: Research, academy, outreach, innovation, and management. In Proceedings of the 8th IEEE Global Engineering Education Conference, Athens, Greece, 25–28 April 2017. [Google Scholar]
  20. Pinto, R.S.; de Mello, S.P.T.; Melo, P.A. Metaevaluation: A decade of Institutional Evaluation Process of SINAES. Campinas 2016, 1, 89–108. [Google Scholar]
  21. Cho, I.-S.; Cho, K.-M. A Study on the Improvement of Maritime Traffic Safety Assessment Scheme by applying Metaevaluation Model. J. Korean Navig. Port Res. 2013, 37, 383–390. [Google Scholar] [CrossRef] [Green Version]
  22. Elliot, L.G. Metaevaluation: From approaches to possibilities of application. Trial Eval. Public Policies Educ. 2011, 19, 941–964. [Google Scholar]
  23. Website of the Joint Committee on Standards for Educational Evaluation. Available online: (accessed on 16 February 2021).
  24. Leslie, M.; Moodley, N.; Goldman, I.; Jacob, C.; Podems, D.; Everett, M.; Beney, T. Developing Evaluation Standards and As-sessing Evaluation Quality. Afr. Eval. J. 2015, 3, 13. [Google Scholar] [CrossRef]
  25. Yarbrough, D.B. Developing the Program Evaluation Utility Standards: Scholarly Foundations and Collaborative Processes. Can. J. Program Eval. 2017, 31, 284–304. [Google Scholar] [CrossRef] [Green Version]
  26. Lubowiecki-Vikuk, A.; Dąbrowska, A.; Machnik, A. Responsible consumer and lifestyle: Sustainability insights. Sustain. Prod. Consum. 2021, 25, 91–101. [Google Scholar] [CrossRef] [PubMed]
  27. Svermova, P.; Cernik, M. Corporate Social Responsibility of Companies Producing PFOA Containing Waxes for Cross-Country Skiing. Sustainability 2020, 12, 5141. [Google Scholar] [CrossRef]
  28. Zhi, M.J.; Hu, L.L.; Wang, Y.F.; Liu, Y.L. Length of care based on Continuity Assessment Record and Evaluation and its influencing factors for the elderly in nursing home. J. Nurs. Sci. 2020, 35, 95–98. [Google Scholar]
  29. Thomson, A.M.; Perry, J.L.; Miller, T.K. Conceptualizing and Measuring Collaboration. J. Public Adm. Res. Theory 2007, 19, 23–56. [Google Scholar] [CrossRef] [Green Version]
  30. Fořt, J.; Černý, R. Transition to circular economy in the construction industry: Environmental aspects of waste brick recycling scenarios. Waste Manag. 2020, 118, 510–520. [Google Scholar] [CrossRef]
  31. Perea-Moreno, M.A.; Manzano-Agugliaro, F.; Hernandez-Escobedo, Q.; Perea-Moreno, A.J. Peanut Shell for Energy: Properties and Its Potential to Respect the Environment. Sustainability 2018, 10, 3254. [Google Scholar] [CrossRef] [Green Version]
  32. Ge, Z.M. The method of human and the environment intervention and the possibility of its application in China. J. Ocean Univ. China 2003, 2, 79–83. [Google Scholar]
  33. Ouyang, X.; Guo, F.; Shan, D.; Yu, H.; Wang, J. Development of the integrated fuzzy analytical hierarchy process with mul-tidimensional scaling in selection of natural wastewater treatment alternatives. Ecol. Eng. 2015, 74, 438–447. [Google Scholar] [CrossRef]
  34. Chang, L.; Li, Y.H. Construction of performance evaluation criterion system of knowledge sharing among government de-partments. Statistic. Decis. 2017, 24, 47–50. [Google Scholar]
  35. Saaty, T.L. Fundamentals of Decision Making and Priority Theory with the Analytic Hierarchy Process; RWS Publications: Pittsburgh, PE, USA, 1994. [Google Scholar]
  36. Ekama, G.A.; Wilsenach, J.A.; Chen, G.H. Saline sewage treatment and source separation of urine for more sustainable urban water management. Water Sci. Technol. 2011, 64, 1307–1316. [Google Scholar] [CrossRef] [Green Version]
  37. Liu, C.M.; Wang, Z.Y. Urban Sewage Treatment Process and Selection Principles. Appl. Mech. Mater. 2014, 641, 399–402. [Google Scholar] [CrossRef]
  38. Website of the Central People’s Government of the People’s Republic of China. Available online: (accessed on 20 October 2020).
  39. Torretta, V. The Sustainable Use of Water Resources: A Technical Support for Planning. A Case Study. Sustainability 2014, 6, 8128–8148. [Google Scholar] [CrossRef] [Green Version]
  40. Zhang, X.Q.; Sun, S.N.; Liu, H.Q. Research on Sustainable Utilization Programming Model of Region with Relatively Abundant Water Resources. Water Sav. Irrig. 2010, 7, 37–43. [Google Scholar]
  41. China Statistical Yearbook (CSY); China Statistical Publishing House: Beijing, China, 2019.
  42. Website of the Department of Housing and Urban-Rural Development of Guangxi. Available online: (accessed on 20 October 2020).
  43. Website of the Department of Housing and Urban-Rural Development of Shaanxi. Available online: (accessed on 20 October 2020).
  44. Website of the Department of Housing and Urban-Rural Development of Inner Mongolia. Available online: (accessed on 20 October 2020).
  45. Tan, M. A Guide Atlas for the Detailed Explanation of Geography in Middle School; Shandong Map Publishing House: Jinan, China, 2020; p. 88. [Google Scholar]
  46. Hao, H.; Li, Y.; Zhang, H.; Zhai, R.; Liu, H. Spatiotemporal variations of vegetation and its determinants in the National Key Ecological Function Area on Loess Plateau between 2000 and 2015. Ecol. Evol. 2019, 9, 5810–5820. [Google Scholar] [CrossRef] [PubMed]
  47. Website of Department of Ecological Environment of Shaanxi. Available online: (accessed on 20 October 2020).
  48. Jorm, A.F. Using the Delphi expert consensus method in mental health research. Aust. New Zealand J. Psychiatry 2015, 49, 887–897. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  49. Malisa, R.; Schwella, E.; Batinge, B. Augmenting Water Supplies Through Urban Wastewater Recycling (March 2019). IEEE Syst. J. 2019, 14, 1523–1530. [Google Scholar] [CrossRef]
  50. Website of China News. Available online: (accessed on 21 October 2020).
  51. Website of People’s Daily. Available online: (accessed on 21 October 2020).
  52. Website of Department of Ecological Environment of Shaanxi. Available online: (accessed on 21 October 2020).
  53. Website of Xinhuanet. Available online: (accessed on 21 October 2020).
  54. Ma, J.; Hipel, K.W.; Hanson, M.L.; Cai, X.; Liu, Y. An analysis of influencing factors on municipal solid waste source-separated collection behavior in Guilin, China by Using the Theory of Planned Behavior. Sustain. Cities Soc. 2018, 37, 336–343. [Google Scholar] [CrossRef]
  55. Hussain, M.S.; Abd-Elhamid, H.F.; Javadi, A.A.; Sherif, M.M. Management of Seawater Intrusion in Coastal Aquifers: A Review. Water 2019, 11, 2467. [Google Scholar] [CrossRef] [Green Version]
  56. Majambu, E.; Wabasa, S.M.; Elatre, C.W.; Boutinot, L.; Ongolo, S. Can Traditional Authority Improve the Governance of Forestland and Sustainability? Case Study from the Congo (DRC). Land 2019, 8, 74. [Google Scholar] [CrossRef] [Green Version]
  57. Website of the Ministry of Housing and Urban Rural Development of China. Available online: (accessed on 21 October 2020).
  58. Chen, Z.M. Public Management, 2nd ed.; China Renmin University Press: Beijing, China, 2017; p. 379. [Google Scholar]
  59. Luo, P.P.; Sun, Y.T.; Wang, S.T.; Wang, S.M.; Lyu, J.Q.; Zhou, M.M.; Nakagami, K.; Takara, K.; Nover, D. Historical Assessment and Future Sustainability Challenges of Egyptian Water Resources Management. J. Clean. Prod. 2020, 263, 121154. [Google Scholar] [CrossRef]
  60. Brooks, C.; Schopohl, L. Green accounting and finance: Advancing research on environmental disclosure, value impacts and management control systems. Brit. Account. Rev. 2021, 53, 100973. [Google Scholar] [CrossRef]
  61. Falcone, P.M. Environmental regulation and green investments: The role of green finance. Inht. J. Green Econ. 2020, 14, 159–173. [Google Scholar] [CrossRef]
Figure 1. The criteria used to measure interactions.
Figure 1. The criteria used to measure interactions.
Sustainability 13 02567 g001
Figure 2. The location of the Guangxi, Shaanxi and Inner Mongolia provinces [45].
Figure 2. The location of the Guangxi, Shaanxi and Inner Mongolia provinces [45].
Sustainability 13 02567 g002
Figure 3. The application process of the Delphi method.
Figure 3. The application process of the Delphi method.
Sustainability 13 02567 g003
Figure 4. The score of the Guangxi, Shaanxi, and Inner Mongolia provinces in meta-evaluation.
Figure 4. The score of the Guangxi, Shaanxi, and Inner Mongolia provinces in meta-evaluation.
Sustainability 13 02567 g004
Figure 5. Centralized untreated rate of urban sewage treatment plants in three provinces.
Figure 5. Centralized untreated rate of urban sewage treatment plants in three provinces.
Sustainability 13 02567 g005
Table 1. The standards of meta-evaluation for the evaluation of environmental management.
Table 1. The standards of meta-evaluation for the evaluation of environmental management.
UtilityU1Stakeholder IdentificationPerson who is involved in or affected by EEM should be identified.
U2Evaluator CredibilityThe person performing EEM should be competent.
U3Information ScopeThe collected EEM information should be extensive.
U4Values IdentificationThe perspectives, procedures, and rationale of EEM should be carefully described.
U5Report ClarityThe program being assessed should be clearly described, including context, purposes, procedures, and findings.
U6Report Timeliness and DisseminationThe reports of EEM should be disseminated to target users.
U7Evaluation ImpactThe reports of EEM should be reported and applied.
U8Human ResponsibilityThe results of EEM should provide clear and reliable recommendations for environmental protection.
FeasibilityF1Practical ProceduresThe procedures of EEM should be practical.
F2Political ViabilityThe EEM should be planned and conducted with the positions of stakeholders.
F3Cost EffectivenessThe EEM should be efficient and produce information of sufficient value.
F4Health AssuranceThe health and safety of the evaluators should be fully protected throughout procedures of EEM.
F5Collaboration MechanismThe EEM should provide a mechanism to ensure the cooperation of all participants.
ProprietyP1Service OrientationThe EEM should be designed to effectively serve the needs of stakeholders.
P2Formal AgreementsObligations of participants to EEM should be agreed to in writing.
P3Human InteractionsEvaluators should respect human dignity in their interactions with other persons associated with EEM.
P4Complete and Fair EvaluationEEM should be complete and fair in its examination of the program.
P5Disclosure of FindingsThe persons affected by the evaluation and any others with expressed legal rights can receive the results.
P6Conflict of InterestConflict of interest within EEM should be dealt with openly and honestly.
P7Fiscal ResponsibilityThe evaluator’s allocation and expenditure of resources should reflect sound accountability procedures.
P8Respect to the EnvironmentThe EEM should be designed and conducted to avoid negative impact on the environment.
AccuracyA1Program DocumentationThe program being evaluated should be described and documented clearly and accurately.
A2Context AnalysisThe context in which the program exists should be examined in enough detail.
A3Described Purposes and ProceduresThe purposes and procedures of EEM should be monitored and described in enough detail.
A4Defensible Information SourcesThe sources of information used in EEM should be described in enough detail.
A5Valid and Reliable InformationThe information-gathering procedures should be chosen or developed and then implemented.
A6Systematic InformationThe information collected, processed, and reported in EEM should be systematically reviewed.
A7Analysis of Quantitative InformationQuantitative information in EEM should be appropriately and systematically analyzed.
A8Analysis of Qualitative InformationQualitative information in EEM should be appropriately and systematically analyzed.
A9Justified ConclusionsThe conclusions reached of EEM should be explicitly justified.
A10Impartial ReportingReporting procedures should guard against the distortion caused by the personal feelings and biases of any participant.
Note: The researchers stated that, in some situations, certain criteria of the JCSEE program evaluation standards may not applicable [18], and these were removed or changed in our study. The difference between “human interactions” and “collaboration mechanism” is that the former is one of the necessary prerequisites of the latter. The cooperation mechanism emphasizes more heavily the design of rules.
Table 2. The Saaty 1-9 scaling method [35].
Table 2. The Saaty 1-9 scaling method [35].
Quantitative   Value   ( b i j ) Description
1Criterion b i is equally as important as criterion b j .
3Criterion b i is slightly more important than criterion b j .
5Criterion b i is more important than criterion b j .
7Criterion b i is much more important than criterion b j .
9Criterion b i is extremely significantly more important than criterion b j .
1, 1/2, 1/3, 1/4, 1/5, 1/6, 1/7, 1/8, 1/9If criterion b i is less important than criterion b j , b i j = 1/ b j i .
Table 3. Summary of λ max , w i , C R , and adjusted w i of each criterion.
Table 3. Summary of λ max , w i , C R , and adjusted w i of each criterion.
λ m a x w i C R Adjusted   w i
First hierarchyUtility-Accuracy4.03100.4668, 0.2776, 0.1603, 0.09530.0115
Second hierarchyU1-U88.28770.1585, 0.0713, 0.0327, 0.0479, 0.0231, 0.2319, 0.3280, 0.10650.02917.4, 3.33, 1.53, 2.24, 1.08, 10.83, 15.31, 4.97
F1-F55.06800.2634, 0.4174, 0.0615, 0.1602, 0.09750.01527.31, 11.59, 1.71, 4.45, 2.71
P1-P88.28770.3280, 0.0479, 0.0327, 0.2319, 0.1065, 0.0231, 0.0713, 0.15850.02915.26, 0.77, 0.52, 3.72, 1.71, 0.37, 1.14, 2.54
A1-A1010.60500.1110, 0.0778, 0.0381, 0.0543, 0.1569, 0.0269, 0.0194, 0.0151, 0.2879, 0.21280.04511.06, 0.74, 0.36, 0.52, 1.5, 0.26, 0.18, 0.14, 2.74, 2.03
Note: C R < 0.1 means the consistency test has been passed [35]; the sum of criterion weights after adjustment is 100;   λ max is the largest eigenvalue; w i is the required weight value in a certain hierarchy; C R is the proportion of consistency; Adjusted w i is the weight of each criterion of meta-evaluation.
Table 4. The meta-evaluation results of Guangxi, Shaanxi, and Inner Mongolia provinces.
Table 4. The meta-evaluation results of Guangxi, Shaanxi, and Inner Mongolia provinces.
LabelCriterionMeta-Evaluation Objects
GuangxiShaanxiInner Mongolia
U1Stakeholder Identification
U2Evaluator Credibility
U3Information Scope
U4Values Identification
U5Report Clarity
U6Report Timeliness and Dissemination
U7Evaluation Impact
U8Human Responsibility
F1Practical Procedures
F2Political Viability
F3Cost Effectiveness
F4Health Assurance
F5Collaboration Mechanism
P1Service Orientation
P2Formal Agreements
P3Human Interactions
P4Complete and Fair Evaluation
P5Disclosure of Findings
P6Conflict of Interest
P7Fiscal Responsibility
P8Respect to the Environment
A1Program Documentation
A2Context Analysis
A3Described Purposes and Procedures
A4Defensible Information Sources
A5Valid and Reliable Information
A6Systematic Information
A7Analysis of Quantitative Information
A8Analysis of Qualitative Information
A9Justified Conclusions
A10Impartial Reporting
Note: ● indicates that the criterion is reached; ○ indicates that the criterion is not reached.
Table 5. Relationship matrix between meta-evaluation criterion and evaluation process.
Table 5. Relationship matrix between meta-evaluation criterion and evaluation process.
Define Evaluation IssuesCollect InformationAnalysis InformationPropose Evaluation ReportMake Evaluation BudgetMake Evaluation AgreementManage EvaluationHire Evaluator
Note: ● indicates that the criterion is reached; ○ indicates that the criterion is not reached; ▲ indicates the criterion that needs to be evaluated for a specific evaluation section.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Ma, J.; Yin, Z.; Guo, Z. Meta-Evaluation for the Evaluation of Environmental Management: Standards and Practices. Sustainability 2021, 13, 2567.

AMA Style

Ma J, Yin Z, Guo Z. Meta-Evaluation for the Evaluation of Environmental Management: Standards and Practices. Sustainability. 2021; 13(5):2567.

Chicago/Turabian Style

Ma, Jing, Zhaoyun Yin, and Zhengbing Guo. 2021. "Meta-Evaluation for the Evaluation of Environmental Management: Standards and Practices" Sustainability 13, no. 5: 2567.

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop