Struggling with COVID-19—A Framework for Assessing Health System Performance

: Currently, no guidelines exist on how to evaluate the performance of health systems ﬁghting the SARS-CoV-2 pandemic. Therefore, this study seeks (1) to develop a conceptual framework that would be helpful in the given context, and (2) to test the feasibility of the proposed approach. The framework is conceptualized based on investigating critical dimensions and indicators for the successful design of a method for evaluating the performance of health systems. Subsequently, the taxonomic measure of development is used and a correlational analysis made in order to run a pilot test of the proposed concept. The ﬁnalized conceptual framework has ﬁve input dimensions (demographical burden, epidemiological burden, health-related quality of life, ﬁnancial resources, and access) described by 18 indicators, and two output domains (outcomes, productivity) described by six indicators to monitor system performance under the COVID-19 pandemic. The pilot-test conducted in European Union countries and the United States proved the model to be useful and feasible. The proposed framework can be used to assess the performance of health systems ﬁghting novel pathogens, such as SARS-CoV-2, worldwide. Our methodological approach can be used as a benchmark for international agencies such as the World Health Organization in developing their own frameworks. The paper presents the ﬁrst research exploring the evaluation of a health system during the COVID-19 pandemic. It has the potential to help monitor a health system’s performance during the pandemic by accounting for speciﬁc country-related circumstances. In a broader sense, it can contribute to boosting health market competitiveness in terms of quality.


Introduction
On 30 December 2019 a cluster of pneumonia cases of an unknown origin in Wuhan City was reported to China's National Health Commission. The next day, the World Health Organization (WHO) was alerted and on 7 January 2020 the Chinese authorities were able to identify a novel coronavirus as the cause of the pneumonia [1]. Four days later, the first death was reported. By the end of January there had been almost a thousand new cases reported in China (mostly in Hubei Province), accompanied by first diagnoses in Thailand and Japan. Ever since, the COVID-19 pandemic has affected almost all countries globally, putting at risk both their nations and the global economy.
The scope and the scale of the COVID-19 pandemic varies from country to country but, according to the WHO [2], the local epidemics share characteristics. Firstly, the virus demonstrates a very high human-to-human transmission rate and thus presents a capacity for an explosive spread. Because of this, the outbreak has continued to grow internationally at a rapid rate, with cases of secondary transmission confirmed around the world. By 23 August 2021 over 211 million people had been infected globally, with over four million deaths [3]. The worst situation at the end of August 2021 was observed in the USA, India, Brazil, the Russian Federation, the United Kingdom, France, Turkey, Argentina and Columbia. Regions and countries characterized by high population density rates and high levels of deprivation and poverty have been seen to be hit the hardest ( Table 1). The overall situation has stabilized in the Western Pacific and European regions, although intense transmission clusters are still being discovered in a number of countries, overwhelming their national health systems. The Americas have remained the global center of the pandemic since mid-April 2020. Secondly, SARS-CoV-2 has a potentially detrimental effect on human health. The virus can cause severe illness (acute respiratory disease) and death, although approximately 80% of the cases appear to be mild [1]. The severe and critical cases are mostly observed among the older patients and the chronically ill, putting at risk the aging populations and those with multimorbidity.
Thirdly, the COVID-19 pandemic has caused a shock not only to human health and national health systems, but also to the economies of individual countries and the global economy. The widespread business shutdowns have impacted economic activity. Although the exact magnitude of their impact on GDP growth is very difficult to quantify, these measures have clearly led to sharp reductions in output, corporate investment, international trade and household spending [4]. The lockdown imposed on most national economies has mostly affected output in retail and wholesale trade and in professional and real estate services. The affected sectors account for between 30 and 40 percent of total output in most economies. According to OECD estimations [4], the overall direct initial hit to GDP has typically ranged between 20-25% in the G7 countries.
The aforementioned epidemic and socio-economic consequences of the COVID-19 pandemic call for a response strategy that will address public health issues and economic policy challenges. That strategy must involve ongoing performance measurement in order to make more informing choices and set policy priorities. Therefore, the purpose of this work is to develop a framework for evaluating the performance of health systems affected by SARS-CoV-2 and to examine the feasibility of the generated indicators through a pilot test. The paper is structured as follows: the first section highlights the rationale for performance evaluation of public health sectors; this is followed by a presentation of the original conceptual framework and a wide range of potential indicators; and, finally, the feasibility of the proposed approach is tested. The finalized framework is developed using the taxonomic measure of development (TMD) and a correlation analysis based on Pearson's and Spearmen's coefficients.

Theoretical Background
The idea of the need to evaluate the performance of public health sectors can be traced back to New Public Management (NPM). NPM was developed during the 1980s, and since then many governments and public sector entities have been adopting this approach in order to boost the effectiveness, efficiency and accountability of the public sector [5,6]. In this context, there is a growing recognition of the crucial role played by performance evaluation in health care [7,8]. Performance evaluation is thus considered as a central activity leading to improvements in health systems and upgraded rationality in decision-making. Consequently, it means that there is a need for reporting comparative information regarding national health systems based on meaningful and fair metrics. Given the complexity of health care, this task poses a substantial challenge [9]. The challenge is related to the fact that health care services are influenced directly and indirectly by a huge spectrum of elements, shaped by both professionals and organizations responsible for health care delivery [10]. If we additionally consider that the final output of care, i.e., population health, is strongly contextual in terms of demography and epidemiology, the evaluation process appears as a very complex and comprehensive one. Some studies have already pointed to such difficulties when developing a performance evaluation framework in public administration [11,12].
As regards the health care sector, many authors have focused on adopting comprehensive performance measurement frameworks such as the balanced scorecard and benchmarking at the organizational [13][14][15], regional [16], national [17,18] or international levels [19][20][21]. Other authors have looked beyond the tools already well-established in the management literature and proposed original conceptual frameworks and models to be used by public health care authorities [22][23][24]. At the same time, international organizations, such as the WHO [25,26] and the Organisation for Economic Co-operation and Development (OECD) [27], have proposed their own approaches in order to help countries in building effective tools for international comparisons.
The main idea behind the concept of performance refers to achieving the objectives of the health system, namely improving health, enhancing responsiveness and assuring fairness in financial contribution [28]. The first of these goals is usually defined in terms of outcomes, the second in terms of ethical issues and client orientation, and the third in terms of financial risk protection against incurring health care costs.
There are several dimensions that have been employed to describe system performance, among which the most common are effectiveness, efficiency, quality and equity [27,29]. The most popular assessment frameworks introduced by the WHO and by the OECD apply different configurations of these dimensions. Their contents and structures are discussed in detail in the literature [30]. The framework released by the WHO indicates five elements of performance, while adopting distribution and the average level as measures to achieve the first two objectives. It was redesigned in 2007 in response to global challenges. Ever since, the emphasis has been put on the functions of the health system, and the new model has been framed in six domains, namely (1) service delivery, (2) information, (3) health workforce, (4) medical products, vaccines and technologies, (5) stewardship, and 6) financing. [31] By comparison, the OECD model underlines the financial aspects of the health system by incorporating the level of health expenditure as a goal [30].
Since 2000, which was when the first WHO report was published, establishing standards for data has remained a major problem for performance evaluation of health systems. The unification of data and the establishment of common performance indicators would help to conduct cross-country comparisons and support reporting at the European Union (EU) level and worldwide. The second major challenge is that of coordinating methodologies. Most scholars agree that the main obstacle to an effective management of performance evaluation systems is posed by the complexity of selecting measures [32][33][34]. Overcoming these obstacles would definitively support health policy-makers and facilitate comprehensive analyses in the public sector.

Materials and Methods
The paper aims to (1) develop a conceptual framework that would be helpful to evaluate the performance of health systems struggling with the SARS-CoV-2 pandemic and (2) test the feasibility of the proposed approach.
We posed a research question: does a higher assessment of a country's health and health system capacity profile give better results in combating the health crisis in that country?
The existing system performance evaluation models can be treated as references, but the novelty of the current epidemiological situation makes their permanent adoption a difficult task. Therefore, we designed a conceptual framework that would help to assess the performance of a health system during the COVID-19 pandemic. The idea behind the proposed framework is quite simple. We assume that there are several elements (variables) that determine the country's health and health system capacity profile. Health variables pose a certain burden to the functioning of health systems, consequently affecting the size and structure of the demand for healthcare services. On the other hand, a health system has a certain capacity profile that probably affects the way the pandemic is being managed. Finally, both sub-profiles pose a challenge to a health system, resulting in the outcomes. These outcomes can be interpreted both from the epidemiological as well as from the technical point of view.
The input variables pertain to the demographic, epidemiological, health-related quality of life, access, and financial dimensions ( Table 2). In the subsequent step, based on experts' opinions we formulated six output variables which could be considered suitable for system performance evaluation during the COVID-19 pandemic: (1) observed case-fatality ratio (the number of deaths per 100 confirmed cases); (2) deaths per 100,000 population (this represents a country's general population, with both confirmed cases and healthy people); (3) total cases per million population; (4) total tests per 1000 population; (5) daily tests per 1000 population (7-day smoothed); and (6) COVID-19 Case Recovery Rate (CRR-Recoveries/Confirmed Cases). They can be grouped in two performance domains and six corresponding performance indicators (Table 3). The proposed model for evaluating health systems during the COVID-19 pandemic consists of five input dimensions and two output domains ( Figure 1). Its main goal is to monitor the capacity of a health system and improve its response. In a broad sense, the idea behind the evaluation framework is to contribute to response strategy setting and the management of the global pandemic. The data on the coronavirus pandemic were as of 28 June 2020. The analysis covered EU member states (EU-28) and the United States.
As the Open Data Repository is at the developing stage in Poland, we were not able to include the proper link to the database. The research was conducted in two stages. First, a "country health and health system capacity profile" ("the input" - Figure 1) for 29 countries was built. Next, its impact on the outcomes and productivity of the health systems during the COVID-19 pandemic was studied (a health system performance model -"the output" - Figure 1). A "country health and health system capacity profile" was designed based on available data characterizing the situation of the given country in terms of its demographic situation and the capacity of its health system (see Table 2).
In the first stage, the taxonomic measure of development was applied to assess and rank countries according to their health situation. The problem of the health profile is considered as a multivariate phenomenon with many characterizing aspects. The idea behind applying methods for ordering a linear set of objects is to rank them, i.e. to determine the order of objects according to a specific criterion. This particular measure is a synthetic measure of development that can serve as a tool for such ordering. The application of the A pilot test was conducted to examine the feasibility of the proposed approach based on the data generated for the selected set of system and performance indicators shown above. We analyzed statistical data collected routinely by: The data on the coronavirus pandemic were as of 28 June 2020. The analysis covered EU member states (EU-28) and the United States. As the Open Data Repository is at the developing stage in Poland, we were not able to include the proper link to the database. The research was conducted in two stages. First, a "country health and health system capacity profile" ("the input"- Figure 1) for 29 countries was built. Next, its impact on the outcomes and productivity of the health systems during the COVID-19 pandemic was studied (a health system performance model-"the output"- Figure 1). A "country health and health system capacity profile" was designed based on available data characterizing the situation of the given country in terms of its demographic situation and the capacity of its health system (see Table 2).
In the first stage, the taxonomic measure of development was applied to assess and rank countries according to their health situation. The problem of the health profile is considered as a multivariate phenomenon with many characterizing aspects. The idea behind applying methods for ordering a linear set of objects is to rank them, i.e., to determine the order of objects according to a specific criterion. This particular measure is a synthetic measure of development that can serve as a tool for such ordering. The application of the taxonomic measure of development makes it possible to classify objects described by many features using one synthetic variable. There are multiple ways of constructing this indicator. Four basic scales are distinguished in the theory of measurement: nominal, ordinal, interval and ratio [35]. Systematics of scales refers to transformations which retain relations of the respective scale. These results are well-known and have been presented by various authors, for example [36]. The interval and ratio scales are high [37]. In the present study, the authors applied metric data (interval, ratio). In general, methods are divided into pattern and patternless. In our study the pattern formula was used [38]. Hellwig's method [39] is one that is based on standardized values of the diagnostic features. The distance-the so-called Euclidean distance measure-between each test object and the established developmental pattern is determined- [40] in the following manner: where: i-the number of the object (country), i = 1, 2,..., N, k-the number of variables (characteristics), k = 1, 2,..., K, z ik -the normalized value of the k variable for the i object, z ik = x ik −x k S k (x ik -value of the k diagnostic variable for the i object; x k -arithmetic average of the diagnostic variable x k , S k -the standard deviation from the diagnostic variable x k ,), z 0k -the development pattern value for the k variable (it takes the maximum value of the variable for the tested objects if the variable is stimulant or its minimum value if it is a destimulant).
The distance d i can be used to compare the level of development of the examined objects: the lower the value, the higher the level of development achieved by the given object. In order to normalize the indicator's value, the relative taxonomic measure of development (TMD), also known as the Taxonomic Measure of Investment Attractiveness (TMAI) [41], can be applied: replaces a description of the examined objects with a description of a single aggregate value. This method is often used to assess the level of economic development of countries and regions [42,43].
In the present study, 28 EU countries and the United States were examined for the 18 diagnostic variables listed in Table 2. Variables No. 5 and Nos. 10-18 were considered as stimulants, i.e., the higher their values, the better the situation. The other variables were deemed as destimulants, i.e., the lowest possible values were preferred.
In the second stage of the study, correlation coefficients were used to examine the impact of the "country health and health system capacity profile" ("the input"- Figure 1) on the performance of the given health system during the COVID-19 pandemic ("the output"- Figure 1):

•
Pearson's coefficient-to measure the relationship between the values of the taxonomic development measure and the values of selected variables, • Spearman's coefficient-to measure the relationship between the country ranking positions according to the "country health and health system capacity profile" and selected variables.

Results
A "country health and health system capacity profile" for the 29 countries was built using the TMD according to formula 2. There were 18 diagnostic variables (the I1-I18 indicators listed in Table 2) used. The main characteristics of these variables are shown in Table 4. The countries were ranked from the highest to the lowest value ( Figure 2). It turned out that, taking into account selected variables, the best health status of the population and the best health system capacity were primarily identified for Ireland, the United States, Sweden and Luxembourg. Source: author's elaboration.
The countries were ranked from the highest to the lowest value ( Figure 2). out that, taking into account selected variables, the best health status of the po and the best health system capacity were primarily identified for Ireland, th States, Sweden and Luxembourg. Indicators (as shown in Table 3) characterizing the performance of a health system during the COVID-19 pandemic were adopted. The correlation coefficients were measured between two conceptualized parts of the model (the input and the output as in Figure 1), i.e., between the TMD and the variables O1-O6.
The values of Pearson's and Spearman's correlation coefficients were similar but not identical (Table 5). This was due to the unequal distances between the values of the synthetic measure of development for the individual countries (as shown in Figure 2). The results (a positive relationship for the variables O1-O5) can be interpreted as follows-the better "the country health and health system capacity profile" of the country, the higher the values of the variables: observed case-fatality ratio-O1, deaths per Sustainability 2021, 13, 11146 9 of 13 100,000 population-O2, total cases per million population-O3, total tests per 1000 population-O4 and daily tests per 1000 population (7-day smoothed)-O5.
It should be noted that the impact of the country health profile on the O1 variable was negligible. The number of deaths per 100 confirmed cases were actually independent of the national health situation. However, it turned out that the countries demonstrating a better (higher) health profile also performed more SARS-CoV-2 tests (Pearson's coefficient for variables O4 and O5). Unfortunately, there were also proportionately more incidents and COVID-19 deaths (the variables O2 and O3) there. The relationship between the TMD and the variable O6 was low and negative, and statistically insignificant. Therefore, we cannot make the case recovery rate dependent on the level of the health profile.
The higher COVID-19 incidence and death rates of (variables O2 and O3) and the lower case recovery rates (variable O6) in the countries boasting better health profiles may have been related to higher numbers of tests performed (variables O4 and O5) and more people being hospitalized in them. This made the statistics more accurate (the number of patients, the causes of death). The same was indicated by our analysis of the correlations between the individual variables of the model input (I1-I18) and the variables of the model output characterizing health system performance (O1-O6). The highest values of Pearson's coefficient (>0.7) pertained to the correlations between the variables I5-O3, I5-O5 and I15-O3. This means that the higher the GDP per capita and total health care expenditure, the more total cases per million population were diagnosed. Moreover, the higher the GDP per capita, the more daily tests per 1000 population were performed.

Discussion
Although COVID-19 has been around for some time now, there is a limited number of related research publications. The studies made so far have explored various aspects of the pandemic. For example, Kufel [44] indicates usefulness of the econometric model for predicting the dynamics of COVID-19 cases at the different stages of the pandemic, i.e., its development, the peak level of daily cases, and its decline. Other studies have analyzed the adverse effects the pandemic has had on the economy [45]. The impact of the COVID pandemic on a research agenda of health policies has also been studied [46], as well as country policy responses and citizens' perceptions in the EU during COVD outbreak [47]. Other authors have also stressed the importance of drawing lessons from the situation experienced [48], emphasizing the need to test the responsiveness and resilience of health systems. The model framework that we proposed can be seen as such. Considering the existing health sector performance evaluation frameworks, none of them refer directly to the system performance indicators during Sars-CoV-2 pandemic. However, WHO has already established two valuable conceptual models in order to cope with a pandemic, i.e., JEE (Joint External Evaluation) [49] and (Pandemic Influenza Preparedness and Response Plan) [50]. Both documents were produced in 2005 and address the issues of country monitoring, evaluation and recommendations for national measures before and during pandemics.
The Joint External Evaluation-International Health Regulations entered into force in 2007. The JEE Tool focuses on the assessment of country capacity to prevent, detect, and rapidly respond to public health threats independently of whether they are naturally occurring, deliberate, or accidental [49]. Unlike our framework, this one is intended to provide health policy performance indicators within three key areas: (1) prevention (including national legislation, policy and financing), (2) detection (including national laboratory systems, disease reporting according to WHO requirements, and workforce development), and (3) responsiveness (including preparedness, emergency response operations, a national framework for transferring medical countermeasures, and risk communication). Thus, the JEE/IHR represents more of a health policy approach, whereas ours is more of a health system approach.
On the other hand, PIPP concentrates on recommended actions before, during and after a pandemic of influenza. These actions are organized into the five basic components, i.e.,: (1) planning and coordination, (2) situation monitoring and assessment, (3) reducing the spread of diseases, (4) continuity of health care provision, and (5) communications [50]. Comparing our evaluation framework to PIPP, one can see the similarities in the area of pandemic surveillance systems, which in both cases include a case fatality rate as a potential health indicator of severity. Our model complements the set of indicators describing the outcomes with additional ones, namely: deaths per 100,000 population due to COVID-19, total COVID-19 cases per million population and COVID-19 Case Recovery Rate.
Moreover, following the Global Health Security Agenda (GHSA) [51] launched in 2014 and refined in 2018 [52] it is worth highlighting that this approach emphasizes the need for strong multisectoral engagement, including human and animal health, security, agriculture, defense, law enforcement, foreign affairs, development assistance, research, and finance sectors, among others [52]. Comparing our approach to those mentioned above one can draw the conclusion that our framework basically seeks a health system approach with less attention to a disease-specific or all-hazard approach already addressed in GHSA and JEE/IHR.
The literature also highlights the problem of patient cost burden and the willingness to pay for health services. This issue might be of great importance while discussing health care coverage and delivery during the global health crisis (pandemic). Vuong Q.H. et al. evaluated the sensitivity of Vietnamese healthcare consumers against two groups of factors (demographic and socioeconomic-cognitive) regarding payment for periodic GHE (general health examinations) which are not covered by insurance [53]. A paper by Vuong Q.H. Ref. [54] represents the first research attempt to estimate the probabilities of Vietnamese patients falling into destitution due to financial burdens occurring during a curative hospital stay. Such studies tend to focus on Asia and developing countries. We did not address this issue in our research, which can be seen as a study limitation. The working and unemployed are insured in Poland within the system of universal social insurance (employed), and to some extend within state/regional budgets (unemployed). They receive health benefits in life-threatening situations as well as primary health care services.
The results obtained in our study could not be directly compared to the findings of other authors. The reason is that there is a lack of up-to-date studies that would cover exactly the same issues i.e., health system performance under the COVID-19 pandemic. Furthermore, existing frameworks proposed by WHO and the GHSA seem to be insufficient and incomplete in terms of the wide range of problems generated by pandemics (such as setting priorities in accessing and using care, mapping out the role of primary care physicians, cost burden reduction etc.). Our proposal is also not universal, but can be perceived as a first attempt to fit the model framework to the contextual conditions of care in a pandemic.

Conclusions
SARS-CoV-2 is a new pathogen, with the COVID-19 pandemic caused by the virus having affected regions around the world. National health systems are responding to this challenge by aiming to ensure safety and provide appropriate health care to their citizens. Regardless of the circumstances, public policy makers must be prepared to evaluate health system performance based on accurate and reliable data.
The present study is the first attempt to conceptualize an evaluation framework and test it. It contributes to the performance measurement methodology. The study has the potential to help monitor the performance of a health system during the pandemic by accounting for specific country-related circumstances.
The proposed framework can be used to assess the performance of health systems tackling SARS-CoV-2 worldwide. Our methodological approach can constitute a benchmark for international agencies such as the WHO in their efforts to develop their own models. Nevertheless, as the situation caused by the COVID-19 pandemic is a dynamic one, research needs to be revised and developed to account for new input and output variables.