Evaluation of Public Health Emergency Management in China: A Systematic Review

To summarize the present status of health emergency management assessment in China, a comprehensive search of Chinese databases for research that explicitly mention health emergency assessment indicators and indicator systems was performed. Studies were evaluated using the Ekman quality assessment tool, and data were extracted with an original extraction form. Sixty-one studies were included. There are many types and methods of health emergency management assessment in China, and the dimensions and the indicators involved are complex. Legal, regulatory, and policy bases for such assessment need to be further strengthened. The relevance of the entire assessment process and its practical application should be enhanced. In the occupational practice, appropriate evaluation methods should be selected according to respective evaluation purposes, evaluation objects, and contents. Laws, regulations, and policies in the evaluation of health emergency management should be improved. Finally, further correlational research on health emergency management evaluation system processes should be explored and improved.


Introduction
In 2003, a sudden outbreak of Severe Acute Respiratory Syndrome (SARS) occurred in China. More public health emergencies such as the H1N1 flu epidemic in 2009 and the H7N9 avian flu epidemic in 2013 had serious impacts on China's politics, economy, trade, and people's health, among others. Theory and implementation of health emergency systems have evolved worldwide. China has gradually implemented the construction of a comprehensive emergency management system including the use of theoretical models, assessment systems, and response studies. China's health emergency management system is an important component of the comprehensive emergency management system, and health emergency management assessment is an important part of the health emergency management system. China's health emergency management assessment system has experienced many rigorous tests in response to a series of public health emergencies and has accumulated experience in detecting health emergencies and managing the weaknesses of the evaluation system. There are many types of health emergency management assessments and many methods for health emergency management assessment in China, and the dimensions and the indicators of health emergency management assessment are complex. For example, the evaluation system named the "Disease control agency health emergency capacity evaluation index" has three dimensions, the system known as the "Evaluation

Document Inclusion and Exclusion Criteria
Build a model based on the Sample, Phenomenon of Interest, Design, Evaluation, Research Type (SPIDER) qualitative system evaluation questions to hypothesize about problems and develop inclusion exclusion criteria.
Theoretical research that only mentions the word "indicators of health emergency assessment" but has no specific evaluation index content or only partial evaluation indicators, epidemiological characteristic analysis, and epidemiological monitoring reports.

4.
There is no valid content literature, or the original language is from non-Chinese literature and the literature cannot be downloaded.

5.
Evaluation results fall within the scope of low quality research after using the Ekman quality assessment tool [1].

Literature Search Strategy
A comprehensive system search of Chinese databases-including the China Knowledge Network (CNKI), the Wanfang Data Knowledge Service Platform (WANFANG DATA), and the VIP (VIP data platform)-was implemented that used a search time limit from the establishment of the database as of March 2019. According to the type, the scope, and the research purpose of health emergency management combined with a preliminary search of China's domestic health emergency management assessment literature, two parts of search strategies were identified. The first part included literature encompassing health emergency, public health emergencies, and health and epidemic prevention, and the second part included evaluation, assessment, monitoring, and performance. Each part of the search terms were connected by the logical symbol "OR", and two parts were connected by the logical symbol "AND" to obtain the search results. According to this principle, the corresponding search formulas were formulated according to the characteristics and the requirements of each database. The specific search strategy is detailed in Table 1. Table 1. Search terms and search strategies.

Literature Screening
The literature screening was conducted in two phases. In the first stage, all search documents were found in the form of bibliographies and then exported to the database for review. The searched documents were checked by topic and abstract. In the second stage, the full text was found and read to determine if it should be included. In the above stage, two evaluators independently screened the literature. If any disagreement arose, experts in the field of health emergency and evidence-based medicine were consulted to assist with judgment. The above document screening process was based on NoteExpress.

Data Extraction
A self-made data extraction table was used to extract the data. The data extraction content mainly included: (1) basic characteristics of research (i.e., research topic, first author, publication time, region, research field, article type, evaluation content, research institution or research objects, research methods, and main research processes); (2) research evaluation system dimensional characteristics (i.e., name of research evaluation system, classification of research institutions, number of dimensions, and the composition of indicator system); (3) study indicators (i.e., first-level indicator and most frequently used indicator); (4) application of research evaluation system; (5) legal, regulatory, and policy on the basis of research evaluation system [i.e., evaluation system evaluation content, evaluation system, construction laws and regulations, and the system (or other standards)]. The data extraction process relied on NVivo12.

Quality Evaluation Criteria
The two reviewers independently screened the literature and conducted a literature quality evaluation. If there were any inconsistencies, they were discussed and resolved first. If any differences persisted, an expert was consulted to assist in the judgment. Ekman's quality evaluation list was used to evaluate the quality of the included studies. The evaluation includes 7 aspects, i.e., research analysis, rationality, methodology, data, goal achievement, results, discussion, and conclusions. Each evaluation indicator corresponds to a score; three "*"s indicate high quality research, two "*"s indicate medium quality research, and one "*" indicates low quality research performed.

Data Analysis
Based on the characteristics of the included studies and the extracted data, qualitative data were used to analyze the collected research data.

Literature Screening
According to the search strategy, a total of 9657 articles were obtained in the database, including 2201 CNKI articles, 6218 WANFANG DATA articles, and 1238 VIP articles. Thereafter, 2733 duplicate articles were excluded in NoteExpress, 6924 articles remained, and four references were tracked. According to the inclusion and the exclusion criteria, the reading topics and the abstracts were screened, and 6705 unrelated research documents were excluded, leaving only 223 that were screened. The retrieval process is depicted in Figure 1.

Quality Evaluation
The quality of the preliminary inclusion studies was assessed using the Ekman quality assessment tool. In terms of methodology, 60 studies (98.4%) clearly described the health emergency assessment research methods, of which 22 studies clearly indicated the time period of the cross-sectional study. In terms of data, 60 studies' data (98.4%) were derived from original data, while only one study (1.6%) used secondary data. The scores indicated that three studies (4.9%) were high quality, and the remaining 58 studies' data (95.1%) were categorized as medium quality research. Overall, most studies were given a rating of 2* according to the Ekman quality assessment tool, which scores between 17 and 21; the research quality of China's health emergency management assessment system is relatively good. The specific results are detailed in Table 2, and the numbers 1-61 represent the research serial number.

Dimensional Characteristics of Assessment System
Fifty-five studies have established health emergency assessment systems. Six studies were based on China's Health Emergency Response Survey and Evaluation Standards, China's Public Health Emergency Report Management Information System, and China's former Ministry of Health's disease prevention and control performance appraisal operation. The manual (2009 Edition) and the Chinese Health Department IHR and other standards conducted a comprehensive health emergency assessment. The health emergency assessment system in 24 studies consisted of three dimensions, such as monitoring capability, plan capability, management capabilities, or similar constructs. Overall, in the health emergency assessment systems, the maximum and the minimum number of indicators were as follows: all indicators (maximum of 214, minimum of 4); primary indicators (maximum of 15, minimum of 2); secondary indicators (maximum of 204, minimum of 0); tertiary indicators (maximum of 84, minimum of 0). The specific results are detailed in Table 3.   work capacity self-evaluation, professional knowledge training effect, work capacity constraints, factors improvement, and work ability prediction.

21
Construction of an evaluation system and development of software for the evaluation index of the capacity of the armed police unit's health emergency rescue team.

65
including organizational command, emergency maneuver, injury and treatment, medicinal materials protection, classified and sent, epidemic prevention and anti-health preservation.

22
Indicator system for health emergency teaching case evaluation. 3 47 including material value dimension, structure value dimension, practical value dimension and literary value dimension.

26
Tertiary monitoring and early warning system framework.
3 62 including monitoring system, risk assessment system, early warning system, and system guarantee.

27
Index system for evaluation of community health emergency plan. 3 63 structure, process, and results.

Indicator Characteristics and Application of Assessment System
The indicator characteristics and the application of the health emergency assessment system included in the study were analyzed according to different institutional categories, which were "State and Health Administrative Department", "Disease Prevention and Control Institutions", "Health Institution", "Health Supervision Agency", "School", "Military", and "Border Port Health and Quarantine Department".

Health Supervision Agency Assessment Research
Two articles were concerned with the health emergency assessment of health supervision agencies. The indicators most frequently used were organization command (10.10%), management system (5.05%), equipment reserve (12.12%), training drill (12.12%), and emergency response (6.06%).

School Assessment Research
Two articles were concerned with school health emergency assessment research. The indicators most frequently used were health emergency value mission (13.79%), preparation and recovery (6.9%), detection and monitoring (3.45%), response (3.45%), and materials learning (3.45%).

Health Emergency Assessment Practice Research
Health emergency assessments conducted at all levels of government and health administrations are mainly used in Shanghai, Sichuan, China, and Guangdong Provinces. Health emergency assessments conducted at disease prevention agencies are mainly used in Shanghai, Shandong, Guangxi Zhuang Autonomous Region, Guangdong, Inner Mongolia Autonomous Region, the entire country, and Henan Province. Health emergency assessment in medical and health institutions is mainly applied in Guangxi Zhuang Autonomous Region, Hebei Province, Heilongjiang Province, the entire country, and Beijing. Health emergency assessment in the military is mainly applied to the entire army. Health emergency assessment conducted at the port is mainly applied to Xinjiang's border ports.
Through this research, we gained a comprehensive understanding of the status quo of China's health emergency assessment and explored the main distribution areas of the representative research of China's health emergency assessment system, research output, research design, main dimensions, high frequency indicators, major application cities, and health emergencies. We further evaluated the subjects of the classification. The study only included Chinese studies, which affects the comprehensiveness of the research. The specific results are shown in Table 4. County-level CDC emergency public health emergency response capability evaluation (national). City district-level disease prevention and control center emergency public health emergency response capability evaluation (Shandong Province). County-level disease prevention and control agency/epidemic station emergency public health incident on-site emergency response capability evaluation (Henan Province). City-level disease prevention and control center emergency public health emergency response capability evaluation (Henan Province). Evaluation of public health emergency response capacity of district and township preventive health care centers (Guangdong Province). Municipal, county-level disease prevention and control center (Guangdong Province).

Medical institutions
Monitoring capability, plan capability, material reserve, training, early warning, management command, information monitoring report, system, personnel, medical technology, education, laboratory, medical treatment, bed, logistics support, communication, on-site disposal, drills.

18
County general hospital and township health center (Guangxi Zhuang Autonomous Region). Top three hospitals (Hebei Province). Hospital (Heilongjiang Province). Basic-level emergency public health emergency response assessment (national). Secondary hospitals (Beijing).

Health supervision agencies
Organizational command, management system, equipment reserve, training drill, response.

5
District Health Emergency Capability Survey and Evaluation (Shanghai). District-level public health emergency response capacity assessment (Guangdong Province).

Schools
Value education, preparation and recovery, detection and monitoring, response, materials learning.

Military
Command capability, support capability, equipment and other reserve capability, system construction, response capacity, monitoring capability, medical treatment capacity, early warning capability, education, control capability, service capability, survival capability, team construction, response capability, research capability, drill, rescue capability, mission, medicine reserve.
Border port health and quarantine department Vulnerability, environment, system construction, laboratory construction, network construction, drill, support, on-site disposal measures, monitoring, training, warning.

Legal and Policy Basis for the Evaluation System
Twenty-seven studies (44.3%) referred to laws, regulations, systems, or other standards during the emergency assessment system construction process. Among them, 18 studies (66.7%) were evaluated for health emergency response, one study (3.7%) on a social vulnerability assessment system for major infectious diseases, one study (3.7%) on an early warning system for influenza outbreaks, two studies (7.4%) on health emergency pre-plan assessments, one study (3.7%) on performance assessment, one study (3.7%) on teaching case assessments (on emergency management), and three studies (11.1%) on health emergency work evaluation.

Discussion
Health emergency management assessment is an indispensable part of health emergency management. Through proper evaluation, problems and deficiencies in health emergency management can be detected over time to ensure and enhance the efficacy by which such measures can be implemented in time to maximize the likelihood of preventing and controlling public health emergencies.

Wide Range of Assessments
In health emergency management assessment practice, the types of assessments vary according to different classification criteria such as the purpose, the object, and the content of the assessment as well as the work status [10]. Research clearly indicates that there are a multitude of assessments in China. First, assessments are divided according to purpose, i.e., formative and summative assessments. For example, the evaluation of Yunnan Province's emergency health response capability and core competence assessment index system at district-level control institutions was performed to guide and advance the progress of assessment targets as well as to provide a formative evaluation of management decisions. On the other hand, in Fujian Province, the study is concerned with the state of performance appraisals regarding public health emergencies and other issues, including formulating (such as plan completion rate, emergency material reserve rate, simulation exercises) overall judgments on the purposes of evaluation institutions, including accountability and summative assessment. Second, health emergency management assessment is divided into several stages, including the preparation stage (i.e., pre-emergency evaluation), the disposal stage, the in-process evaluation, and the recovery stage (i.e., post-emergency evaluation). Thus, a preliminary discussion on the early warning index system for influenza outbreaks, the construction of an evaluation index system for hospital nurses' public health emergency response capacity, a weighted analysis of emergency public health emergency response evaluation factors, and the evaluation of emergency drill activities in county-level disease control institutions in Nanchang City are all examples of pre-assessment. In contrast, social vulnerability assessments and analyses of significant factors associated with major infectious diseases are concerned with the assessment of the nature, the type, the extent, and the determining factors of public health emergencies, while epidemiological characteristics studies and normative disposal third-party assessments are all considered post-event assessments that focus on recovery, summarizing, and long-term impact. Third, emergency management work assessment can be divided according to health status, i.e., normal and abnormal assessment. For example, the analysis and the evaluation of the present state of affairs of health emergency work in Qinghai Province was a normal assessment of the state of daily management construction, whereas the comprehensive evaluation of public health emergency implementation in the Fujian Province in 2014 was an assessment of the process and the impact of the emergency response. Fourth, assessment is divided according to the evaluation implementation body, i.e., internal and external evaluation. For example, the evaluation of the public health emergency response capacity of F hospital in Qiqihar City was an internal investigation conducted by an investigation team composed of health emergency management agencies and workers. On the other hand, investigating the epidemiological characteristics of public health emergencies and standard treatment third-party assessments are surveys that involve external evaluation from experts outside the health emergency management and work organization.

Health Emergency Management Assessment Methods
Health emergency management assessment is a process of formulating objective judgments on assessment targets based on specific criteria. Qualitative, quantitative, and mixed methods are most commonly used. This review found that the construction of a health emergency management evaluation system is more of a qualitative approach and is appropriately categorized as exploratory research (e.g., literature review, personal interview, Delphi, brainstorming, expert group discussion, expert meeting business law, field inspection). Some studies focus on data collection, detailing the main laws and characteristics by using descriptive methods. The health emergency management evaluation system weight analysis and application falls more in the domain of the Delphi method, while analytic hierarchy process, entropy weight, rank sum ratio, fuzzy comprehensive evaluation, and matter-element extension methods are relatively new methods for health emergency assessment. The application of the health emergency management evaluation involves possibly utilizing questionnaire surveys, factor analysis, principal component analysis, comprehensive scoring, comprehensive indexes, and analytic hierarchy process methods. Recently, scholars have started to pay attention to health emergency resource allocation research using data envelopment analysis methods. Health emergency management systems are extremely complex and large, as they are concerned with numerous variables and complicated mechanisms and structures.

Health Emergency Management Assessment Dimensions and Indicators Are Complex
The dimensions and the indicators of health emergency management assessment represent, to some extent, a specific aspect of health emergency management work and represent the constituent elements of health emergency management (e.g., health emergency agencies, manpower, materials, information technology, funds). These elements support health emergency management work and are the foundation for the smooth implementation of health emergency management. This review found that the assessment dimensions predominantly included national and local state health administrative departments' normal emergency management and abnormal emergency command organization management. Assessment dimensions were also concerned with health emergency professional technical institutions, including health emergency level disease prevention and control institutions, professional disposal management, personnel management, materials management, information management, fund management, etc., at various levels of health institutions. Moreover, health emergency assessment indicators are embodied in the duties and the authority of the state in the management of normal/abnormal emergencies, duties of professional technical institutions, classifications and responsibilities of technical and health emergency personnel, emergency materials categories and management methods, information monitoring, source, report, confidentiality, and supervision, and aspects of health emergency fund sources to support policy and supervision.

Legal and Policy Basis Require Further Reinforcement
Due to the differences in organization, system, economy, culture, and history of various countries, the organization, the function, and the structure of their respective national health emergency management and evaluation systems differ. In April 2018, the central and the state institutions of China reformed and established the National Emergency Management Department, equipped with primary functions responsible for the management of natural and accidental disasters. Social security incidents were assigned to the Political and Legal Committee, and public health events became the responsibility of the Health Committee. In the future, more comprehensive, targeted, and operational public health events, social security incidents, natural disasters, and accident disasters related laws and regulations and various policy systems will likely be gradually developed, and corresponding event-related authorities will also enhance the supporting plan, the disposal plan, and the process.
This study found that the main Chinese environmental laws and regulations utilized in the process of constructing a health emergency assessment system included the Emergency Regulations for Public Health Emergencies, the Law of the People's Republic of China on Emergency Response, the Law of the People's Republic of China on the Prevention and Control of Infectious Diseases, "The disease prevention and control work performance appraisal operation manual", the State Council Emergency Office "Emergency Drill Guide", and so on. In addition, the International Health Regulations (IHR2005), the "US Center for Disease Control (CDC)" public health preparation, and the response capacity scale "Expert Advisory Standards", in addition to others, were also used. Nonetheless, the research in reference to laws, regulations, systems, or other standards is still marginal, and there persists a lack of detailed studies on legal, regulatory, and policy bases for health emergency management assessment before the establishment of the National Emergency Management Department. After the establishment of the National Emergency Management Department in April 2018, China's emergency management functions were gradually clarified. Laws, regulations, and policies for health emergency management assessment need to be continuously updated, and in-depth research needs to be conducted in order to provide a basis for health emergency management assessment. See the Appendix A.

Relevance of Health Emergency Management Evaluation System
The various system elements of the health emergency management system are interrelated and are organic as a whole, and the connecting paths between various elements are worth further exploration. This study found that most research is narrowly focused on particular topics; indeed, evaluation research is typically carried out on specific problems. Yet, it is rare to explore the interrelationships and the causal paths among the various elements of the evaluation system. Specific problems should be found in the health system, and relationships and mechanisms of system elements should undergo in-depth analysis to accurately determine the underlying causes of a health emergency problem and fully comprehend the full process management of public health emergencies.

Need to Strengthen Practical Application of Assessment Systems
This study found that research on constructing health emergency assessment systems, analyzing and evaluating present systems based on existing standards, and evaluating practical applications based on the construction of management evaluation systems is scarce. In the application evaluation research, the types of application institutions are predominantly from disease prevention and control institutions, medical and health institutions, government and health administrative departments, and military and border ports. Merely building a health emergency assessment system cannot meet the needs to form a comprehensive emergency system, nor can it identify the weaknesses in management efficiently. Practical research on the evaluation of health emergency systems should be conducted rigorously, and attention should be paid to the actual evaluation and application of health supervision institutions, schools, and other institutions.

Limitations and Strengths
The study solely included Chinese research, and this arguably limits the ecological validity of the findings. Yet, this study still conducted a broad and comprehensive investigation of the Chinese literature, traced references, and conducted an in-depth analysis of medium-quality and high-quality research, effectively reducing bias due to insufficient literature included.

Conclusions
The qualitative system evaluation of China's health emergency management assessment revealed that, although China has carried out substantial research related to health emergency assessment, in the face of frequent global public health emergencies, the current assessment of health emergency areas still requires continuous improvement. Foreign health emergency management assessment related research is carried out earlier and applied in practice. The research on China's health emergency management assessment has been carried out late, the evaluation system is flawed, the evaluation criteria are not standardized, the evaluation model is unstable, and the assessment of health emergency management processes lacks attention. A first step in improving the situation is to enhance the relevance of the process of health emergency management assessment. For more than ten years, the health emergency management assessment has prioritized the evaluation of staged and key management work. At present, there is still a lack of research across all processes related to the health emergency management evaluation system. Second, the process of the health emergency management evaluation system is weak. Previous research on a health emergency management assessment system typically targeted certain types of public health emergencies and lacked the exploration of health emergency management assessment systems based on the commonality of multiple types of public health emergencies. Third, the application of a health emergency management assessment system is not strong. The content and the objectives of the previous health emergency management assessment systems are relatively scattered, lack organic integration of the assessment process, and create a state of under-applied appraisal.
The representative research of China's health emergency assessment system is mainly distributed in the eastern part of China. The design of such research mainly focuses on cross-sectional studies. While the dimensions and the high-frequency indicators of the health emergency assessment system have been extracted, the applicability of the health emergency assessment system needs to be improved. China should emphasize research and application of health emergency assessment in central and western regions, including ethnic minority areas, and the health emergency assessment system should be improved from the perspective of classification and integration of health emergency assessment subjects, as health emergency management assessment is an important part of the health emergency management activities. Through evaluation, problems in health emergency management can eventually be found, and the effective prevention and control of public health emergencies can be achieved to the greatest extent possible. There are many types of and methods for health emergency management assessments, and the related dimensions and indicators are complex. In health emergency management assessment, it is extremely important to select appropriate assessment methods according to different assessment purposes, targets, and contents. Laws, regulations, and policy bases of health emergency management assessment should be strengthened, especially after the establishment of the National Emergency Management Department in April 2018.
The most important finding of the current research is that there are many types of health emergency management assessments and multiple health emergency management assessment methods in China. The dimensions and the indicators of health emergency management assessment are also complicated. In general, the dimensions of health emergency assessment are mostly in the secondary dimension. The evaluation indicators mainly include management ability, improvement ability, exercise ability, and reporting ability. Laws, regulations and policy bases for health emergency management assessment also need to attend to differences and linkages between national health emergencies and national emergency laws, regulations, and policies. It is necessary to improve the relevance of the entire process of the health emergency management assessment system. The practical application of the health emergency management assessment system also needs to pay attention to the application of health supervision, schools, and other institutions. The correlation research of the entire process of the health emergency management evaluation system needs to be further explored and improved. In addition, China should build a health emergency management assessment system that is in line with international standards and focuses on Chinese characteristics, which needs further exploration in the future.
China's comprehensive promotion of health infrastructure construction requires improvement of disaster prevention, mitigation/emergency response capabilities, and improvement of the emergency health emergency response system. Future research should explore the optimal methods of realizing China's health emergency management, evaluate relevance and commonality, and develop sound and applicable standardization of health emergency management assessment systems to meet the developing requirements of potential emergencies in China. This research can be used to develop health emergency assessment research in other countries and contribute to the development of a global health emergency movement.