Investigating the Effectiveness of Government Public Health Systems against COVID-19 by Hybrid MCDM Approaches

Abstract

To elucidate the effectiveness of the containment strategies against the pandemic, a Multi-Criteria Decision Making (MCDM) model is established to evaluate the government’s performance against COVID-19. In this study, the Analytic Hierarchy Process (AHP), Entropy, and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) method are used in determining the performance of the public health system. We adopt both subjective and objective weighting methods for a more accurate evaluation. In addition, the evaluation of performance against COVID-19 is conducted in various aspects and divided into different periods. Data Envelopment Analysis (DEA) is applied to evaluate the sustainability of the public health system. Composite scores of the public health system are determined based on the performance and sustainability assessment. The five countries, South Korea, Japan, Germany, Australia, and China are rated with higher composite scores. On the country, the US, Indonesia, Egypt, South Africa, and Brazil receive lower rating scores among the countries for evaluation. This modeling study can provide a practical quantitative justification for developing containment policies and suggestions for improving the public health system in more countries or areas.

1. Introduction

COVID-19 has been spreading worldwide for over two years since the outbreak in 2019 [1]. Most governments make efforts to contain the virus transmission. Thus, many stringent or even intrusive non-pharmaceutical interventions have been established, such as international travel restrictions, cancellation or suspension of public events, online contact tracing, social distancing, and lockdown measures [2,3,4,5]. These interventions and the containment performance represent the public health system preparedness and effectiveness against COVID-19.

Due to the different levels of socioeconomic status, developed and developing countries have their unique public health systems, resources, and management capacities. Hence, the containment performance against COVID-19 should correspond to the public health system’s preparedness and capacities. Many countries with more advanced socioeconomic development status should have been able to exploit the potential resources and contain the pandemic better, but do not seem to perform well. Thus, it is necessary to investigate the essential issues to prepare for future epidemics. Ranking the performance against COVID-19 by various governments can assist in identifying the good policies and accelerate the ending of the pandemic. Additionally, some government control strategies will bring certain economic losses. These measures may pose an economic burden on society and mental pressure on related individuals [6,7]. Assessing the national containment performance can assist in optimizing the strategies and relieving the pressure of some stringent policies [8]. Thus, a comprehensive analysis with essential indicators is needed to find critical factors in disease control [9,10].

Before the outbreak of COVID-19, the epidemic preparedness index was evaluated based on a range of indicators to quantify the epidemic preparedness for most countries [11]. The global health security index was also raised to evaluate the preparedness for pandemics [12]. However, it is found that the performance in the COVID-19 pandemic may not be consistent with the assessed rankings [13]. Specific factors during the pandemic were not considered. Further, the efforts in containing COVID-19 by governments not only affect the short-term society and economies, but also lead to a long-term impact on the socioeconomic consequences [14]. Thus, it is vitally critical to evaluate the performance against the pandemic.

In all, there are doubts regarding the association between the prosperity of a country and the performance against COVID-19 [15]. To answer the question and improve the preparedness for the next pandemic, a more comprehensive evaluation is carried out in this work. The study not only determines the manifest performance against the pandemic, but also analyzes the efficiency of resource utilization by different countries. Specifically, the mathematical model evaluates the comprehensive public health system performance against COVID-19 in three aspects. The first aspect is the direct performance ranking for the public health system against the pandemic. The second is the sustainability [16] of the public health system. The third aspect is the potential [13] of the public health system, which represents the capacity of the public health system to utilize the available resources. In this study, multi-criteria-decision-making (MCDM) approaches are adopted to provide a foundation to carry out the multi-attribute evaluation analysis. A combined MCDM method is used, including Analytic Hierarchy Process (AHP), Entropy, and Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS), to evaluate the performance of the public health system with 15 indicators. Further, the indicators are divided into inputs and outputs. The concept of production efficiency is introduced to evaluate the efficiency of the systems against transmissible diseases. Data Envelopment Analysis (DEA) can be adopted to evaluate the current utilization of public resources. The sustainability and potential of the public system in combating COVID-19 are determined using this approach. A paradox may be elucidated as to why some countries with high scores in pandemic preparedness before do not achieve a corresponding impressive performance against COVID-19.

The remaining part of this article is structured as follows (Figure 1). Section 2 reviews the evaluation of public health system preparedness and associated methods. Data and data preprocessing are introduced in Section 3. The modeling procedure is given in Section 4 and Section 5 presents the evaluation results. After discussion in Section 6, the conclusions are drawn at the end.

2. Literature Review

2.1. Public Health System Preparedness

Prior to the COVID-19 pandemic, there was already some research on the assessment of the governments’ performance or capacities against transmissible diseases or the system preparedness, including the epidemic preparedness index [11] and the global health security index, which characterizes the preparedness for pandemics [12]. Since the outbreak of the COVID-19 pandemic, the evaluation of containment performance has attracted attention. Because it can guide the improvement of the entire public health system and control the pandemic [17]. Lai et al. analyzed the effect of various non-pharmaceutical interventions (NPIs), including social distancing and travel restrictions, on the transmission of COVID-19 in China [18]. The NPI effects were evaluated based on the number of cases, and a combination of NPIs was suggested. The impact of interventions by local governments was also analyzed within a single country against COVID-19, including Italy, Germany, Singapore, the UK, and Vietnam [19,20,21,22,23]. The comparison of NPI impacts among various countries offered a better evaluation of the effectiveness of the government interventions [24,25,26,27,28,29]. The effectiveness of interventions by 41 governments was evaluated by Brauner et al. [30]. However, most selected countries were limited to developed countries [31]. Some interventions demonstrated different effectiveness among various countries [32]. Hence, socioeconomic factors in different governments should be taken into consideration. Though comparative evaluations on the effectiveness of government interventions are performed, the evidence regarding the intervention effectiveness was still insufficient [24]. Besides, the analysis of the specific impact of interventions among countries is affected by the duration of government intervention implementation, which was often omitted in these studies.

In addition to analyzing the effects of interventions within specific regions, improving the public health system and control strategies could also benefit from the ranking of COVID-19 containment performance. Nevertheless, numerous limitations exist in evaluating the system preparedness against emergencies, including a lack of universal standard and selection of indicators [9]. The socioeconomic factors are not included in some modeling for evaluation [33]. The side effects of extreme stringent strategies [34,35] should be avoided by assessing the effectiveness of policies. Based on the doubling time for the infected cases and deaths, a ranking of 35 countries was proposed. The evaluation of 34 Organisation for Economic Cooperation and Development (OECD) countries is also carried out for selected weeks [36]. However, most studies only considered the confirmed cases and deaths as the major indicators [33]. The rankings lack the analysis of the contribution of proposed containment strategies [37]. In addition to carrying out the evaluation merely based on the number of COVID-19 deaths and confirmed cases, the efficiency of utilizing the public health resources should be taken into consideration. It leads to discovering the potential to improve the current public health system further against COVID-19 or future epidemics. The efficiency of preventing the spread of COVID-19 was evaluated for 19 countries by Ghasemi [38]. The productivity of containment strategies was also assessed [39]. However, these performance evaluations were limited to relatively few countries and a short period of time. Additionally, the data processing relied on relatively few indicators. Comprehensive analyses with essential indicators were needed to find critical factors in disease control [10,11]. The evaluation of performance integrated with more indicators can be used to predict the policy modification’s impact better and optimize the public system to contain the pandemic.

2.2. Evaluation Approaches

As COVID-19 spreads, the governments’ strategies may vary, leading to different long-term performances. Multi-criteria decision-making (MCDM) methods are effective in several practical problems and optimization engineering applications [40,41,42,43]. Few MCDM approaches are adopted in evaluating government interventions [23]. To conduct a comprehensive evaluation of the performance and inform the decision on national containment strategies, MCDM could serve as a good approach. AHP method [44] can be applied in the various areas as one of the MCDM methods. It is used to evaluate the sustainability of remote medical services [45]. AHP is also used to evaluate the water-health-environment-nutrition relationship [46]. Further, the fuzzy methods can be improved with hybrid approaches [47,48,49,50,51]. Combined with entropy or TOPSIS method [52], modified AHP methods were used for assessing vaccines and candidates to get the vaccination, digital systems, rating of disasters, and selection of supply chain [53,54,55], which may be beneficial for the improvement of the complex public health system. The research is put forward to evaluate the public health system’s comprehensive performance against COVID-19 from multiple aspects. The spread of COVID-19 is vast, and it is a very complex system to control effectively. The amount of data is relatively large, and many factors are related to the economic level, social, political, and economic indicators. In order to establish an effective COVID-19 containment public health system, the containment strategies must be optimized while economic development, basic living, and medical resources should be guaranteed. Using TOPSIS as the MCDM method integrated with subjective and objective weighting is a suitable approach to solve the problem. In addition to evaluating the direct performance of COVID-19 containment, the efficiency of the available resources against COVID-19 is emphasized in this study. The impact of socioeconomic factors was analyzed on the HIV global epidemic [56] by DEA, and recommendations were proposed for the investments and distribution of inputs [57,58]. In this work, DEA is used as the evaluation method to measure the sustainability and potential of the countries against COVID-19. This study adopts MCDM approaches to provide a foundation for carrying out the multi-attribute evaluation analysis.

3. Data

3.1. Data Collection

In this work, the data are collected from the website “Our World in Data” [59,60] to perform the modeling analysis. The data contain 62 original indicators from 223 countries and territories from 24 February 2020 to 1 December 2021, including most socioeconomic factors and related public health indexes. Before the modeling, the indicators are weighted, and data are preprocessed. Sixty countries with higher populations are selected for the analysis as shown in Figure 2. The red color of a country indicates a high population.

3.2. Data Preprocessing

The transmission and containment of a pandemic could be analyzed as divided by many periods (stages or phases) [61,62]. Similar to the three-phases analysis [62], the time frame since the COVID-19 outbreak is divided into three time periods (stages). The data can be smoothed for a certain period to eliminate short-term impacts. The variation of the effects of the interventions over time can be obtained. The length of the stages $T_i$ should depend on the performance against COVID-19 and interventions by different countries. This study sets the period of each stage $T_i,i=1,2,3$ to be 162 days, which is relatively long, trying to minimize the effects brought by the different beginning times of COVID-19 outbreak in different countries. The three stages are denoted as Stage I, II, and III, respectively. The performance against COVID-19 is evaluated in these different stages, which are demonstrated by the varying numerical values of indicators.

We calculate the mean value ${\overline{x}}_{i,j,k}$ by moment estimation to represent the indicator data as

$${\overline{x}}_{i,j,k}=\frac{1}{T_i}\sum _{n\in U_{T_i}}{x}_{i,n},$$

(1)

where ${\overline{x}}_{i,j,k}$ is the indicator j of the country i in stage k. In this work, there are 15 indicators of 60 countries in stages II and III. There are 13 indicators considered in stage I due to missing data for two indicators related to vaccines in most countries.

4. Comprehensive Evaluation Models

To enable a comprehensive assessment of the public health systems against COVID-19 for 60 selected countries, three aspects, including performance, sustainability, and potential are considered in the evaluation models by MCDM methods. Performance indicates the direct results of a public health system, such as the current infected cases, to contain the pandemic. Sustainability indicates the efficiency of utilizing the resources and maintaining its current effectiveness against the pandemic. The potential indicates the ability of a public health system to be further improved.

4.1. Assessment of the Performance Based on AHP-Entropy TOPSIS Evaluation Model

4.1.1. Decision-Making Criteria

The decision-making criteria are established for the performance evaluation (AHP hierarchy structure shown in Figure 3). In the AHP-Entropy TOPSIS model, four major factors (higher level of indicators) are considered to measure the public health system of different countries in response to COVID-19. The factors and indicators are listed in

Table A1. Indicators for public health system against COVID-19.

Factor	Indicator	Symbol	Impact	Description
Responsiveness	NC	$O_{1j}$	N	New confirmed cases of COVID-19 per 1,000,000 people
	ND	$O_{2j}$	N	New deaths attributed to COVID-19 per 1,000,000 people
	PV	$I_{1j}$	P	Total number of people who received at least one vaccine dose per 100 people in the total population
	NT	$I_{2j}$	P	New tests for COVID-19 per 1,000 people
No. of cases	TC	$O_{3j}$	N	Total confirmed cases of COVID-19 per 1,000,000 people. Counts can include probable cases, where reported
	TD	$O_{4j}$	N	Total deaths attributed to COVID-19 per 1,000,000 people. Counts can include probable deaths, where reported
Reserves	TE	$I_{3j}$	P	Tests conducted per new confirmed case of COVID-19, given as a rolling 7-day average
	TT	$I_{4j}$	P	Total tests for COVID-19 per 1,000 people
	HB	$I_{5j}$	P	Hospital beds per 1,000 people
	TV	$I_{6j}$	P	Total number of COVID-19 vaccination doses administered per 100 people in the total population
Government abilities	SI	$I_{7j}$	P	Government Response Stringency Index: composite measure based on 9 response indicators including school closures, workplace closures and travel bans
	HDI	$I_{8j}$	P	A composite index measuring average achievement in three basic dimensions of human development—a long and healthy life, knowledge and a decent standard of living.
	MA	$I_{9j}$	N	Median age of the population, UN projection for 2020
	EP	$I_{10j}$	N	Share of the population living in extreme poverty, most recent year available since 2010
	PD	$I_{11j}$	N	Number of people divided by land area, measured in square kilometers, most recent year available

.

(1)

Responsiveness: The speed of response refers to how quickly the countries’ public health systems respond to a significant public health emergency, such as this outbreak of COVID-19. It is one of the critical evaluation factors for measuring the performance, sustainability, and potential of the public health system of a country. The faster a country responds, the stronger the public health system is. New deaths (per million), new cases (per million), new tests (per thousand), and vaccinated persons (per hundred) are selected as secondary indicators. For example, if the number of new deaths is low, the country responds quickly to COVID-19.

(2)

No. of cases: The number of cases refers to the number of people infected in each country during the epidemic. The total cases (per million) and total deaths (per million) are selected as secondary indicators. The lower the number of cases and deaths, the better the country controls the epidemic and the stronger the public health system.

(3)

Reserves: The reserves indicate the amount of relevant medical supplies that countries store in their daily production life. It is related to a country’s capacity to produce supplies. It is also an essential measure of the strength of a public health system. According to the definition of the factor, we select hospital beds (per thousand), total tests (per thousand), tests (per case), and total vaccinations (per hundred) as the detailed indicator of the number of material reserves.

(4)

Government ability: Government ability refers to the long-term human development indicators, such as extreme poverty and population density. Five indicators are used to characterize the factor: population density, stringency index, median age, extreme poverty, and human development index [63].

Further, the above 15 indicators are categorized into inputs and outputs. Input indicators represent the investment of the public health system containing the pandemic. Output indicators demonstrate the effectiveness of the public health system in containing the pandemic. The characteristics of the indicator are shown in Table A1. The letter “P” in the impact column indicates that the larger indicator should positively impact the prevention and control of a pandemic. “N” indicates the opposite case (negative effect). We use $I_{i,j}$ and $O_{i,j}$ to denote input and output indicators, respectively. i denotes the country number and j denotes the indicator number, i.e., $i=1,2,\cdots ,60$, $j=1,2,\cdots ,15$.

Before assessing the direct containment performance, the weight of each indicator is determined by using the combination of the AHP and entropy (AHP-Entropy) weight methods [64]. AHP is a subjective weighting method, and the entropy weighting method is objective. Both subjective importance of indicators and the objective characteristics of data are taken into account to evaluate the performance against COVID-19 by AHP-Entropy weighting. TOPSIS, as an effective MCDM approach for ranking and has the fewest rank reversals, is used to evaluate the performance against COVID-19 [65].

4.1.2. AHP-Based Weight Determination

Three experts provide relative importance of the criteria to determine the comparison matrix.

Step 1:

Build the comparison matrix

Factors and indicators are divided into different layers. The first layer is sorted by the importance subjectively: Reserves ≥ Government ability ≥ Responsiveness ≥ No. of cases. The comparison matrix $\left(a_{ij}\right)n\times n$ is obtained:

$$\begin{array}{cc}\hfill & \begin{array}{cccccc}\hfill & \hfill & A_1\hfill & A_2\hfill & A_3\hfill & A_4\hfill \end{array}\hfill \\ \hfill & \begin{array}{cc}\hfill & A_1\hfill \\ \hfill & A_2\hfill \\ \hfill & A_3\hfill \\ \hfill & A_4\hfill \end{array}\left(\begin{array}{cccc}1& 5& 1/3& 1/2\\ 1/5& 1& 1/4& 1/3\\ 3& 4& 1& 3\\ 2& 3& 1/3& 1\end{array}\right),\hfill \end{array}$$

(2)

where $A_i(i=1,2,\cdots ,4)$ represents the factors Responsiveness, No. of cases, Reserves, and Government ability. $a_{ij}$ is the relative importance of $A_i$ compared with $A_j$.

Step 2:

Consistency check

Based on the established comparison matrix, we calculate the largest eigenvalue ${\lambda }_{max}$ and the corresponding eigenvector v. In this case, the random inconsistency $RI$ is 0.9 when the order of comparison matrix $n=4$ [66]. The consistency index $CI$ and the consistency ratio $CR$ are determined by the following calculation procedure:

$$CI=\frac{{\lambda }_{max}-n}{n-1}$$

(3)

and

$$CR=\frac{CI}{RI},$$

(4)

$CI=0.0742$ and $CR=0.0834$, which is less than 0.1, implying the comparison matrix is consistent [67].

Step 3:

Determine the weights of indicators

The eigenvector v after normalization is the weight vector corresponding to $w_{i1},i=1,2,3,4$. The weight of Responsiveness is 0.1982; the weight of No. of cases is 0.0736; the weight of Reserves is 0.4901; the weight of Government ability is 0.2381. Similarly, all the weights of each indicator are determined under different factors (Figure 4).

4.1.3. AHP-Entropy Weight Determination

The information entropy $E_j$ is a measure of uncertainty of indicator j. A larger entropy corresponds to a more significant variation. In the comprehensive evaluation of the public health system, the information entropy $E_j$ is used as an objective weight of each indicator. The calculation procedure of the entropy $E_j$ is as follows:

Step 1:

Matrix normalization.

$${\widehat{x}}_{ij}=\left\{\begin{array}{cc}\frac{x_{ij}-min\left(x_i\right)}{max\left(x_i\right)-min\left(x_i\right)}\hfill & \mathrm{if}\mathrm{indicator}j\mathrm{is}\mathrm{an}\mathrm{positive}\hfill \\ \frac{max\left(x_i\right)-x_{ij}}{max\left(x_i\right)-min\left(x_i\right)}\hfill & \mathrm{if}\mathrm{indicator}j\mathrm{is}\mathrm{an}\mathrm{negative}\hfill \end{array}\right.$$

(5)

Step 2:

Determination of the ratio of each indicator.

The ratio of the indicator j in country i is the varying size of the indicator as follows:

$$p_{ij}=\frac{{\widehat{x}}_{ij}}{{\displaystyle \sum _{i=1}^m{\widehat{x}}_{ij}}},i=1,2,\cdots ,m,j=1,2,\cdots ,n.$$

(6)

Step 3:

Entropy weight determination

The information entropy of the ratio of each indicator

$$E_j=-ln\frac{1}{n}\sum _{i=1}^m{p}_{ij}ln{p}_{ij},$$

(7)

When $p_{ij}=0$, $E_j$ is defined as 0.

Step 4:

Entropy weights

Calculate the weights ${\omega }_{j2},j=1,2,\cdots ,15$ by the entropy weight method as

$${\omega }_{j2}=\frac{1-E_j}{{\displaystyle n-\sum _{j=1}^n{E}_j}},j=1,\cdots ,15.$$

(8)

Step 5:

AHP–Entropy combined weighting

Together with the subjective weight ${\omega }_{j1}$ by AHP, the comprehensive AHP–Entropy weight of each indicator j is calculated based on the following equation:

$${\displaystyle {\omega }_j=\frac{{\omega }_{j1}{\omega }_{j2}}{{\displaystyle \sum _{j=1}^{15}{\omega }_{j1}{\omega }_{j2}}}}$$

(9)

The subjective weighting (AHP weighting), objective weighting (Entropy weighting) and combined weighting (AHP-Entropy weighting) are calculated and presented in Figure 5.

4.1.4. AHP-Entropy TOPSIS Evaluation Modeling

The Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) is based on the idea that the best solution should have the shortest distance from the ideal optimal solution and the farthest from the ideal worst solution [68,69]. It is used to find the relationship between data of each country and the optimal ideal value, and the worst ideal value. The weighted distance between the inferior ideal values is the standard of superiority and inferiority.

Step 1:

Matrix normalization

After normalization, the original data matrix X (the data of indicators in 4.3.2) is denoted as matrix Z.

Step 2:

Optimal and worst solution calculation

The ideal optimal solution takes the optimal value of the evaluation index in the system, denoted as $Z^{+}$. On the contrary, the ideal worst solution is defined as $Z^{-}$.

Step 3:

Distance calculation

With the weight of each indicator ${\omega }_j,j=1,2,$ ⋯$,15$ determined by the previous subsection, the ideal optimal solution $D_i^{+}$ for the distance of the indicator vector of the country $i(i=1,2,$ ⋯$,60)$ and the opposite ideal worst solution $D_i^{-}$ are given by

$$D_i^{+}=\sqrt{\sum _{j=1}^n{\omega }_j{\left(Z_j^{+}-z_{ij}\right)}^2}$$

(10)

$$D_i^{-}=\sqrt{\sum _{j=1}^n{\omega }_j{\left(Z_j^{-}-Z_{ij}\right)}^2}$$

(11)

Step 4:

Performance score determination. The comprehensive score of the country i performance is

$$R_i=\frac{D_i^{-}}{D_i^{+}+D_i^{-}},i=1,2,\cdots ,60.$$

(12)

$R_i$ is normalized to get the final relative score of the public health system of each country:

$$R_i^{*}=\frac{R_i}{{\sum }_{i=1}^m{R}_i},i=1,2,\cdots ,60$$

(13)

4.2. Assessing Sustainability and Potential Based on Data Envelopment Analysis (DEA)

4.2.1. DEA Modeling

Sustainability to be evaluated is defined as the ability to effectively contain the spread of infectious disease in the short term and maintain the containment strategies in the long term, which requires systematic planning and efficient use of medical resources. The potential of the public health system refers to the capacity to be further improved against the spread of infectious diseases. The potential can be enhanced by improving resource utilization and response of a public health system. Data Envelopment Analysis (DEA) is a method to evaluate the input and output of multiple indicators and measure the effectiveness of the system [70], which has various operation modes, such as CCR mode, BBC mode, cross mode, and A & P mode. The analysis for the sustainability and potential of the system is performed in the following procedure:

Step 1:

Decision-making unit establishment

There are 60 countries as a decision-making unit. Each country has 11 kinds of inputs and 4 kinds of outputs. $x_{ij},i=1,\cdots ,n,j=1,2,\cdots ,11)$ are denoted as the input indicator j of the country i. $y_{ik},i=1,\cdots ,n,k=1,\cdots ,4$ is the output j of the country i. $u=(u_1,u_2,\cdots ,u_{11})$ represent the input weight vector. $v=(v_1,v_2,v_3,v_4)$ represent the output weight vector.

Step 2:

DEA efficiency evaluation model (CCR mode) establishment

The benefit evaluation index of the decision-making unit for country i is denoted as $e_i$:

$$e_i=\frac{{\displaystyle \sum _{k=1}^4{v}_{ij}{y}_{ik}}}{{\displaystyle \sum _{j=1}^{11}{u}_{ij}{x}_{ij}}},i=1,2,\cdots ,n$$

(14)

The idea of the CCR mode linear programming model is to express the input and output of decision-making unit i as a linear combination of other units. We introduce $ϵ$, a very small number, and the slack variable $s_i^{-}$, $s_i^{+}$. The CCR mode linear programming model is:

$$\begin{array}{c}min{e}_i-\epsilon \left({\displaystyle \sum _{i=1}^{11}}{s}_j^{-}+{\displaystyle \sum _{k=1}^4}{s}_k^{+}\right),\\ \mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^n{u}_i{x}_{ij}+s_j^{-}=e_i·x_{ij},j=1,2,\cdots ,11,}\hfill \\ {\displaystyle \sum _{i=1}^n{v}_l{y}_{lj}-s_k^{+}=y_{ik},k=1,2,3,4,}\hfill \\ u_{ij}\ge 0,v_{ik}\ge 0,s_j^{-},s_k^{+}\ge 0,i=1,\cdots ,n.\hfill \end{array}\right.\end{array}$$

(15)

Step 3:

Solution by the hierarchical sequence method. Object 1 of the solution is to find the minimum value of $e_k$. Object 2 is achieved under the condition that $e_i$ is known.

$$\begin{array}{c}\mathrm{Object}1:min{e}_i,\\ \mathrm{Object}2:maxϵ\left({\displaystyle \sum _{i=1}^{11}}{s}_j^{-}+{\displaystyle \sum _{k=1}^4}{s}_k^{+}\right),\\ \mathrm{s}.\mathrm{t}.\left\{\begin{array}{c}{\displaystyle \sum _{i=1}^n{u}_i{x}_{ij}+s_j^{-}=e_i·x_{ij},j=1,2,\cdots ,11,}\hfill \\ {\displaystyle \sum _{i=1}^n{v}_l{y}_{lj}-s_k^{+}=y_{ik},k=1,2,3,4,}\hfill \\ u_{ij}\ge 0,v_{ik}\ge 0,s_j^{-},s_k^{+}\ge 0,i=1,\cdots ,n.\hfill \end{array}\right.\end{array}$$

(16)

4.2.2. DEA Output Analysis

In the DEA, the overall efficiency (OE) represents the production efficiency of the decision-making unit input elements at the optimal scale. It evaluates the resource allocation and resource utilization efficiency of a country in this study. The higher the efficiency, the higher the sustainability of the public health system in response to the pandemic. The OE value corresponds to the sustainability score $S_i$ for country i, $S_i=10e_i$. If the sustainability score of the public health system is low, the public health system resource utilization should be further enhanced. To evaluate the corresponding enhancement of the containment performance if investing more resources of the public health system, the concept of “Returns to scale” is introduced.

Definition 1

(Returns to scale). Returns to scale refers to the relationship between the proportion of input changed and the output changed when inputs are changed at a fixed proportion λ under certain technical conditions.

There are three cases: increasing returns to scale, decreasing returns to scale, and constant returns to scale. Let the function between the input and output of the pandemic containment be

$$Q=f(L,K),$$

(17)

where Q is the function with the degree of homogeneity r, and

$$f(\lambda L,\lambda K)={\lambda }^{r}f(L,K),\lambda >1,$$

(18)

where the degree of homogeneity r can be used to judge the return to scale. According to the DEA model, we calculate r for each country at different stages. The larger the scale of investment to contain the epidemic, the lower the potential. Here, we define the potential score of different scales as 0, 1, and 2, respectively. The score is determined by r.

4.3. Composite Score Determination

Considering the evaluation of the public health system in three aspects, a comprehensive quantitative score can be determined based on the performance and the sustainability score, which is used to rate the comprehensive performance of each country against the pandemic. The calculated performance score $R_i$ and the sustainability score $S_i$ are converted into a compound score $C_i$ to directly demonstrate the performance of each country i in response to the epidemic. The composite score $C_i$ for country i is the sum of the performance score $R_i$ and the Sustainability score $S_i$ over the three stages $\{\mathrm{I},\mathrm{II},\mathrm{III}\}$.

Based on the composite score $C_i$, some countries which possess a higher score or achieve lower comprehensive performance are selected (

Table 1. The relationship of returns to scale and system potential.

Scale of Investment	r	Returns to Scale	Potential
Small	$>1$	Increasing	2 (High)
Appropriate	$=1$	Fixed	1 (Medium)
Huge	$<1$	Decreasing	0 (Low)

). The countries with higher scores are defined as better-performing countries. On the other hand, the countries with lower scores are denoted as worse-performing countries. Then, the strengths and weaknesses of countries’ performance in containing COVID-19 are analyzed. Specifically, 11 input indicators are evaluated and a concept of Degree of Dominance is introduced for the assessment of indicators.

Definition 2.

Degree of Dominance. The Degree of Dominance of better-performing countries compared to worse-performing countries is defined as the difference

$$\mathsf{\Delta }{\overline{x}}_j={\overline{x}}_{B,j}-{\overline{x}}_{W,j},$$

(19)

where ${\overline{x}}_{B,j}$ and ${\overline{x}}_{W,j}$ are the means of indicator j for the better-performing (subscript B) and worse-performing countries (subscript W), respectively.

5. Evaluation Results

5.1. Performance Ranking

After determining the weights of the indicators by the hybrid AHP–Entropy method, the evaluation of the performance of the public health systems containing COVID-19 is carried out by the previous model using TOPSIS. The scores of the countries are shown in Figure 6.

Darker colors indicate higher scores. The performance of countries may vary over time due to the control strategy adjustment, production abilities, policy change, and virus mutation. Thus, the scores are demonstrated in different stages. Based on the performance score of various countries, it is found that East Asia countries such as Korea, Japan, and European countries including Germany and France are rated with higher scores. Many high-score countries are developed countries that are able to utilize more medical resources including vaccines as well as other medical resources. By analyzing the specific indicators behind the performance score, it is found that the high score of Australia is due to the massive PCR testings and testings per new case. In stage I, there are nine countries with a performance score higher than 3 points, four of which are developed countries. There are 21 countries with a performance score of 2–3, including six developed countries. There are 12 countries with a performance score higher than 3 in stage II. In stage II, the number of countries with a performance of more than 2 points is 39. In the second stage, most countries have strengthened their epidemic prevention and control measures. However, in stage III, only eight countries scored more than 3 points. There are a total of 18 countries with a performance greater than 2, and many developing countries have significantly lower performance scores compared to the second stage. Vietnam is rated high in the previous stages, primarily relying on the stringent containment index. However, in the later stage, the overall score declines since the stringent containment index is lowered. From the performance score of the three stages, developed countries have relatively small fluctuations in epidemic prevention and control, while developing countries countries have relatively large fluctuations in long-term epidemic control, which depends on more medical resources, and the ability to maintain containment strategies.

5.2. Sustainability Score

The score of “Sustainability” is the $OE$ value of the DEA output result. $OE$ value represents the productivity of countries utilizing the inputs (corresponding to the available resources). If the efficiency is high, it means that high-efficiency output (the performance against COVID-19) can be achieved with less input, so the sustainability score is high.

As shown in Figure 7, the darker the color, the higher the score of sustainability. The public health systems of China, Canada, and India receive a high sustainable score. Countries such as the United States and Brazil have lower sustainability scores. The United States does possess a high vaccination rate and a good human development index. Brazil has a low population density and a low median age. They could have performed better in the containment of COVID-19. However, the output indicators of these countries, such as new cases and new deaths per million populations, rank high among the 60 countries, demonstrating the insufficient utilization of medical resources. Therefore, their sustainability scores are low. The United States had a higher sustainability score in stage II, but a lower score in Stage III because of the sharp increase in the number of new deaths per million people in Stage III. Although India and other countries rank higher among the 60 countries in terms of new cases and new deaths per million people, they have a high population density and fewer medical resources, such as the number of hospital beds per thousand people. The sustainability scores are relatively high due to the utilization of the medical resources. Based on the sustainability score, which depends on the value of $OE$, the efficiency of the public health system can be differentiated.

Strong effectiveness: When the $O{E}_i=1$ and $\forall j,l,s_j^{-}=s_k^{+}=0$. The system achieves the most efficient technology and the most efficient scale at the same time. Weak effectiveness: When the $O{E}_i=1$ and $\exists j,k,\mathrm{s}.\mathrm{t}.s_j^{-}=0\mathrm{or}{s}_k^{+}=0$, only one of the most effective techniques and the most effective scale is reached.

Ineffectiveness: When the $O{E}_i<1$, it is neither the most effective technology nor the most effective scale. The number of countries in different efficiency states is determined. In stage I, over half the countries in the evaluation fail to reach an effective state. Then, the proportion of countries in strong effectiveness states increases. However, in Stage III, the trend is growing towards more countries with inefficient states, which calls for corresponding strategies.

5.3. Potential of the Public Health System

The performance against COVID-19 of a country could be further improved. We adopt the concept of “Potential” to represent the potential of the public system to be further improved against the pandemic. The concept of returns to scale is introduced. A higher returns to scale indicates a larger output value per unit change in the input. Thus, the system has a greater potential. The relationship between investment scale, r, returns to scale, and the potential is shown in

Table 2. Better and worse performing countries.

Performing	Country
Better	South Korea, Japan, Germany, Australia, China
Worse	United States, Indonesia, Egypt, South Africa, Brazil

. The potential of each country at each stage is shown in Figure 8. A darker color corresponds to a higher potential score. For instance, Brazil has a potential of 2 in the three stages, corresponding to increasing returns to scale r. It shows that if Brazil increases its investment in medical resources (such as the number of vaccines and hospital beds per thousand people.), the number of new cases and new deaths per million people should drop significantly. Investing in those medical resources will bring relatively more benefits in containing COVID-19. The returns to scale will increase. However, for the countries with smaller potential scores, the returns to scale may not increase at the same level corresponding to the investments in those input indicators. The potential scores of Spain in the three stages are 2, 1, and 0, namely increasing returns to scale, fixed returns to scale, and decreasing returns to scale. The system’s gradually decreasing potential score indicates that it effectively utilizes medical resources in the containment of COVID-19.

5.4. Composite Score

After determining the performance score $R_i$ and the sustainability score $S_i$, the compound score $C_i$ are calculated for each country i against the epidemic.

A visual overview of the better-performing countries is given in terms of the degree of dominance of 11 indicators (Figure 9a). The top 5 indicators with the largest $\delta {\overline{x}}_j$ are the advantage of the better-performing countries. The indicators are hospital beds (per thousand), tests (per case), human development index, total vaccinations (per hundred), and new tests (per thousand). The strengths of the countries that performed better are reflected in the adequacy of medical supplies, the speed of response, and the government’s greater emphasis on long-term human development.

The mean values ${\overline{x}}_{B,j}$, ${\overline{x}}_{W,j}$, $j=1,2,\cdots ,5$ of the above five indicators for the two groups of countries are shown in Figure 9b. The values on the blue pentagon indicate the mean values of the five indicators for the better-performing countries. The values on the yellow pentagon indicate the mean values of the five indicators for the worse-performing countries.

6. Discussion

There are some studies evaluating the preparedness or performance against the global pandemic. Most of these studies are performed before the outbreak of COVID-19, or at the early stage of the pandemic [37,38]. Right before the COVID-19 outbreak, the global preparedness for a pandemic was assessed [11,12]. Some developed countries, including US and UK, were rated with high scores. However, the performance in containing the spread of COVID-19 is not relevant to the previous assessment. There should be a gap between the capacity and the outcome to control the pandemic.

In this work, the DEA method evaluates the efficiency of public health resource utilization. It is expected that the potential of the public health system could inform the policy-makers. The effect of government interventions is related to the level of development of a country or region. However, many studies only consider developed countries and regions. This study provides a comprehensive evaluation. The government’s measures are also time-sensitive. The UK gets higher scores when it adopts more robust measures at certain times, but its control measures have been relaxed. Thus, we chose a more extended period and evaluated the performance in three different time stages. To the best of our knowledge, the period for evaluation in this study is the longest. Further, the investigation adopts hybrid MCDM approaches in which the weighting is based on a combination of the AHP and the entropy methods. The performance of Vietnam against COVID-19 is good in the initial stage, as assessed similarly by Shirouyehzad [71]. It is one of the few developing countries, which perform well in containing COVID-19 in the early stages. However, Vietnam’s sustainability score decreases as we extend the evaluation period. In our evaluation, Australia and Japan achieved a high score and performed well among the 60 countries, which agrees with the other MCDM approach [38]. Additionally, the evaluation should not be limited to a few countries or a short period since the comparison can guide improving public health systems.

There are some limitations in our analysis. The established comparison matrix could also be further improved by consulting more experts. Only three experts were consulted for the AHP evaluation from the same country. This study also lacks a profound selection procedure of indicators. Hence, we have to consider all available data. The selection of indicators can be optimized [72]. Some data may not be comprehensive. The PCR testing in developing countries is limited to capacity. Additionally, the definition and counting of deaths due to COVID-19 differs in various countries. Additionally, we overlooked the regional differences within a country. The ranking of performance is given on a national scope. Local public health systems can play a significant role against the pandemic as well [26]. However, the overall scores could be undermined by other regional performances. Besides, we did not consider the impact of age structure in evaluating public health performance. Nevertheless, it should be acknowledged that the stratification of age groups should have a significant role in the COVID-19 transmission and fatalities [32,73]. Multiple MCDM methods [48,51] can be integrated for the performance evaluation, such as complex proportional assessment, full consistency method, and step-wise weights assessment ratio analysis. The countries for evaluation can be extended, although the evaluation has already covered many countries. The assessment is limited to countries with higher populations. The countries with relatively fewer populations could be considered. The improvement in the evaluation method can inform the government to determine an optimized plan [74].

7. Conclusions

In summary, we established integrated models to evaluate public health systems against COVID-19. The model considered 15 indicators of 60 countries and evaluated public health systems from three aspects: performance, sustainability, and potential using MCDM methods. A comprehensive evaluation of the three aspects was presented, and a composite evaluation was raised based on the performance and sustainability scores. The countries with higher or lower composite scores were enumerated, and associated indicators were given. Not only the current performances were assessed, but also the potential preparedness for the future global pandemic. The potential scores demonstrate that there is much room for improvement. This study can provide some guidance to improve the public health systems according to the comprehensive evaluation results from many aspects.

The study should be developed further in the following parts. First, currently, the modeling is based on all available data indicators since we lack the process of selecting indicators based on scientific criteria. The future evaluation can be more well-grounded by adding this section. Second, the composite score is currently the sum of performance and sustainability scores. Appropriate weighting of the performance and sustainability can be proposed to achieve a more reliable comprehensive evaluation. Thirdly, sensitivity analysis can be carried out in future work to identify the significance of essential indicators to improve public health systems precisely. In addition, predicting the impact of the variation of essential indicators can be analyzed to contain the development of COVID-19 in future studies.