1. Introduction
COVID-19 is a respiratory disease caused by the SARS-CoV-2 virus belonging to the coronavirus family, a major threat to public health worldwide [
1]. Symptoms of the infection in humans by this type of virus range from upper respiratory tract infections (as in a common cold) to severe acute respiratory syndrome, pneumonia, and bronchitis [
2]. Elders, children, immunosuppressed people, and those with weak immunologic systems usually express a more severe form of the disease when they get infected, which may result in death [
3]. The virus was first identified in Wuhan city, China, in January 2020 [
4]. It soon spread all over the globe, primarily because of most governments’ inability to control their own countries’ borders [
5]. By August 2022, 32 months after its discovery, COVID-19 had infected over 586 million people worldwide, of whom 6.42 million have passed away (death rate of about 1.1%). In the same time, about 62.6% of people worldwide were fully vaccinated by the end of December 2020, 12.4 billion doses having been administered (Data retrieved from the “Our World in Data” website, a project of the Global Change Data Lab founded by Max Roser and based at the University of Oxford. Website:
https://ourworldindata.org/explorers/coronavirus-data-explorer, accessed on 10 December 2022). The development of COVID-19 vaccines has accelerated since the World Health Organization declared a pandemic in March 2020 [
6,
7]. After conducting a literature review and meta-analysis, Zheng et al. [
8] concluded that these vaccines, especially the Pfizer-BioNTech and Moderna vaccines, are highly protective against SARS-CoV-2 virus-related diseases in real-world settings. However, there are still about one million new infections and over three thousand deaths daily (August 2022), a worrying figure resulting perhaps from the relatively low rate of people fully vaccinated.
The first patient infected with SARS-CoV-2 in Portugal was detected in March 2020. Since then, over half of the country’s population has been infected: in August 2022, there were 5.37 million Portuguese citizens infected, with 24,707 COVID-19-related deaths. This corresponds to an average death rate of 0.46%, far below the world average. One can attribute part of this success in saving lives or avoiding COVID-19-related mortality (compared to peers) to some sequential strategies implemented, including lockdowns [
9], limitation of access to essential services like hospital care, and massive testing and vaccination, despite a low adherence/high hesitance at the beginning [
10]. Compared to the global average, Portugal has performed well regarding vaccination. In total, 86.4% of Portuguese citizens are fully vaccinated against SARS-CoV-2, 24.8 million doses having been administered so far (August 2022). Perhaps because this high rate, most stringency measures taken to limit the outbreaks were lifted in early 2022. Still, Portugal is amongst the countries with the highest infection rates, which does not seem to worry the Portuguese Ministry of Health, given the high full vaccination rate achieved.
Although beneficial to containing the virus, blocking access to healthcare services was not without adverse effects on citizens’ quality of life. Medical appointments, surgeries, exams, and other treatments in primary and secondary healthcare levels were postponed, and admission to emergency rooms and other hospital services was limited to those severely ill. However, denying or delaying care to those searching for it will eventually cause the exacerbation of their illnesses, which become more complex to treat and, ultimately, costlier. This fact jeopardizes the Portuguese publicly available National Health Service’s (already weakened) financial and social sustainability. In addition, the barriers created for entering the health system could have contributed to the increase in excess mortality, not directly related to COVID-19. For instance, the non-COVID-19 excess mortality in Portugal fluctuated between 51% and 92% in 2020 [
11].
Any healthcare provider, hospitals included, aims at improving patients’ quality of life while being economically, socially, and environmentally sustainable [
12]. In Portugal, given the universal and general nature of its National Health Service (NHS), these providers must be capable of treating all patients efficiently without denying them timely, appropriate, evidence-based, and safe care services, equitably and regardless of the patients’ purchasing power [
13]. Excellence or quality in healthcare provision is achieved when all these aspects are met, i.e., when providers are efficient and effective in treating their patients, and there are no barriers to access. Meanwhile, managers should be held responsible for the poor or good healthcare results regarding both resource consumption and response to society [
14].
It should be evident that the effects of a pandemic on hospital performance may not be the entire responsibility of its managers and staff. Part of the results during such a period result from policy measures imposed at the central, regional/federal, or local levels, but also from other factors non-discretionary to hospital managers. Nevertheless, one must assess hospital quality results to search for best practices, i.e., the resilient entities that kept or even improved their performance during the difficult times of the most recent known pandemic. By doing so, hospitals with poor outcomes may adopt and adapt those practices to improve their own, hopefully becoming more resilient in future outbreaks [
15]. Also, the design and implementation of good contingency plans could contribute to a quicker response by these players, avoiding events like the overload and saturation of health systems’ resources in the early stages of a pandemic [
16]. With this study, we do not aim at demerit hospitals with poor performance during pandemic times.
Several studies have been published on this topic, evaluating the performance of hospitals or healthcare systems [
17,
18,
19]. Besides the fact that most studies we found are focused on comparing countries or states (leaving a considerable gap in the literature), researchers seem more interested in the technical efficiency of resource utilization rather than effective results in healthcare. Most results derived concern the first year of the pandemic, while more than twenty months have passed since. Given the growth rates of infected cases and mortality because of COVID-19, it is difficult to understand whether the results are already outdated. In our perspective, evaluating hospital performance requires efficiency and effectiveness analyses in a time range broader than one year to check if low-quality hospitals remain as such during the pandemic, or if high-quality hospitals become less resilient, worsening their health results. No study before this has tried to understand if changes in hospital performance could be affected by COVID-19-specific spread features. Moreover, only a few studies have been concerned about the quality of care or access to hospital services [
19]. This being one of our goals, it is clearly an innovation.
Additionally, one must account for the fact that hospitals do not all operate in the same environment, and some are impaired because of the hazardous conditions in which they operate. Allegedly, the earlier studies seem to have overlooked this important aspect. However, an adjustment mechanism should be developed for hospital benchmarking. This is another innovation as, to the best of our knowledge, no other study has conducted a benchmarking exercise of hospitals during the COVID-19 pandemic with a correction or adjustment for the non-discretionary operational environment.
The objectives of this study are fourfold: first, to estimate an aggregated measure of the hospitals’ quality-based performance individually, with a correction for the operational environment and using benchmarking; second, to compare performance results and evaluate a trend; third, to investigate whether the hospital quality worsened because of the outbreak of SARS-CoV-2; fourth, to test if the disease-spreading features (e.g., reproduction rate, hospital admissions due to COVID-19 per million inhabitants) help to explain the estimated performance evolution. To estimate the quality-based performance, we have used variables like the rate of appointments or surgeries within the legal time, the prevalence of undesirable events within the hospital ward (including, but not limited to, the cases of septicemia or pulmonary embolisms), and even the efficiency of resource usage (including the occupancy of beds and the number of patients per doctor or nurse). Finally, the environmental variables, not controllable by the hospital management, were related to COVID-19 spreading, and include the infection rate, the mortality rate, and the rate of hospital admissions related to COVID-19 infection.
Based on these objectives, we have formulated three main research questions and two hypotheses per question. In line with the paper’s primary goal, these research questions are linked to hospital performance before and during the COVID-19 pandemic. We consider the period before (January 2017 to February 2020) as a baseline to which we can compare the results of the next time interval (March 2020 to May 2022). By the time we wrote this paper (December 2022/January 2023), the pandemic was not considered extinct. Thus, comparing hospital performance before, during, and after the pandemic is impossible. As commonly done in statistics, we state our null hypotheses considering that there is a chance that hospitals may have worsened their performance during the outbreaks, but there is also a chance that they have improved it, at least in some performance dimensions. In other words, we assume the absence of performance change over time in the null hypothesis, and search for evidence disproving it. This is because there are no solid theoretical suggestions concerning this topic. That is, the research questions (Ri) and corresponding null (H0) and alternative (H1) hypotheses are as follows:
(R1) How did the static performance of Portuguese public hospitals evolve from January 2017 to May 2022?
H0(R1): The static performance of Portuguese public hospitals did not change in the considered period.
H1(R1): Portuguese public hospitals’ static performance changed in that period.
(R2) How did the dynamic performance of Portuguese public hospitals grow from January 2017 to May 2022?
H0(R2): The dynamic performance of Portuguese public hospitals did not change in the considered period.
H1(R2): Portuguese public hospitals’ dynamic performance changed in the period.
(R3) Can SARS-CoV-2-related indicators justify the performance evolution in Portuguese public hospitals?
H0(R3): SARS-CoV-2 does not influence the performance evolution of Portuguese public hospitals.
H1(R3): SARS-CoV-2 influenced the performance evolution of Portuguese public hospitals.
Please note that, in our study, the static performance concerns the efficiency and access to safe and appropriate hospital care at each moment. There is an empirical frontier (that should be close to a theoretical one) where benchmarks or best practices are placed. The greater the distance to the frontier, the lower the performance level. The hospital performance will be static should the frontier be constructed using data of just one moment (one year or month). When evaluating the performance evolution over time, one must account for two potential scenarios: frontier and hospital shifts. A change in hospital position regarding the frontier (regardless of the frontier shift) constitutes a static performance evolution. However, benchmarks themselves may also change their positions with time, improving or worsening their performance. That way, the frontier will likely shift alongside the benchmarks. The relative position of two frontiers constructed using data from two instants constitutes the dynamic performance.
The reminder of this manuscript is set out as follows.
Section 2 contains a comprehensive literature review of the works dealing with the performance of healthcare systems concerning fighting against SARS-CoV-2 and its impacts on healthcare provision.
Section 3 presents the materials and methods used in this work. Particularly, we define quality in healthcare provision, given the complexity of this concept. Then, we identify the main variables used to quantify the performance of Portuguese hospitals, which resulted mainly from the literature review undertaken, as well as the data availability in the official sources. The evolution of these variables in the Portuguese public hospitals context is also described. Once these variables have been identified and characterized, we highlight the quantitative methods utilized to construct a composite indicator capable of describing and quantifying the hospitals’ performance before and during the pandemic. We also identify a relational model and non-discretionary variables to try to explain the performance evolution. In
Section 3, case studies and the figures related to COVID-19 in Portugal are described as well.
Section 4 presents the main results concerning the main topics that characterize quality in healthcare: efficiency and productivity, access, and safety and care appropriateness.
Section 5 discusses these findings, and finally,
Section 6 concludes this document with the main lessons learned.
2. Literature Review
Several studies have been published on this topic, evaluating the performance of hospitals or healthcare systems. Concerning the former scenario, Nepomuceno et al. [
17] used a Data Envelopment Analysis (DEA) model for hospital bed vacancy and reallocation during the COVID-19 pandemic. That way, the authors communicated the existence of 3772 beds able to be evacuated by 64% of health units. Kamel and Mousa [
18] estimated the operational efficiency of isolation hospitals in Egypt. They found that less than half of the sample achieved efficiency, and the results were explained more by the numbers of nurses and beds than by the numbers of medical doctors. Through a value-based DEA model, Henriques and Gouveia [
19] assessed the outbreak’s impact on the efficiency of Portuguese state-owned enterprise hospitals. They reported that a lack of resources does not explain inefficiency levels.
There are several other studies comparing countries or states within one country [
20,
21,
22,
23,
24,
25,
26]. Aydin and Yurdakul [
20] integrated machine learning (ML) with a stochastic imprecise DEA model to assess the performance of 142 countries against the COVID-19 outbreak. In short, the machine learning techniques included clustering analyses with k-means and hierarchical clustering methods. The authors found that gross domestic product, smoking rates, and the rate of diabetic patients do not explain the effectiveness level of countries. Taherinezhad and Alinezhad [
21] also integrated ML with DEA to evaluate nations’ performance during the outbreak. Ibrahim et al. [
22] utilized DEA for a country efficiency analysis and concluded that nearly 90% of studied countries were inefficient in pandemic control, and 80% were inefficient during the treatment of positive cases. Using an inverted DEA model, Ferraz et al. [
23] constructed a COVID-19 index for Brazil’s microregions; unsurprisingly, the poorest regions are the most vulnerable. Differently, Mariano et al. [
24] used the Network-DEA (NDEA) to analyze the regional discrepancies within the same country, having identified the state of Amazonas as the least efficient. Similarly, Hamzah et al. [
25] also used an NDEA model to investigate the relative efficiency level while managing COVID-19 in Malaysia. The authors verified the good overall performance of the Malaysian health system, reflecting the robust preparedness and rapid reallocation of resources during the outbreak. Also resorting to the DEA model, Klumpp et al. [
26] compared the performances of countries within the OECD, and found that systems oriented to primary care were more efficient than others. In opposition, healthcare systems that are highly dependent on public funding and heavily regulated were less efficient.
Researchers on hospital performance tend to consider population demographics and epidemiology variables to characterize the operational environment [
27]. Pecoraro et al. [
28] analyzed and compared the efficiency of ordinary hospital bed management in France, Germany, Italy, and Spain. The authors concluded that the structural components of these countries explain part of the pressure in hospital systems during pandemics. These structural components included the heterogeneity of COVID-19 case distribution and the availability of beds. Concerning the first component, it is worth mentioning that the distribution of cases is linked to some population characteristics, particularly the education level, related to compliance with the implemented sanitary measures. For instance, Carlucci et al. [
29] concluded that women, most educated people, middle-aged individuals, and health workers are more likely to adhere to lockdowns and other imposed policies. Jabbari et al. [
30] and Oyeyemi et al. [
31] reached similar conclusions, indicating that people’s education (health literacy) plays a vital role in containing outbreaks. Furthermore, the success of hospitals largely depends on the complexity of cases handled therein [
32]. Concerning COVID-19, evidence has suggested that aging, gender, chronic underlying disease (like hypertension and diabetes mellitus), mental status impairment, length of hospital stay, and high risk of acute deterioration are risk factors associated with unfavorable outcomes [
33,
34]. It should be evident that hospitals in areas with higher rates of chronic diseases or older/less educated populations face worsen conditions in treating their patients.
5. Discussion
During the COVID-19 pandemic, hospitals faced significant challenges in providing quality care due to the overwhelming number of cases, resource constraints, and the need to implement infection control measures. While the specific quality of care varied across different regions and healthcare systems, there are some common factors that affected hospital quality during this time: surges in COVID-19 cases, resource constraints, staffing issues, delayed or deferred non-urgent care, infection control measures, and adaptation to new care models.
The sudden influx of COVID-19 patients put immense strain on healthcare systems, leading to overcrowding and potential shortages of beds, equipment, and medical personnel. This surge impacted the overall quality of care as hospitals struggled to manage the increased workload. Meanwhile, some strategies have been implemented worldwide to mitigate this surge: expanding hospital capacity, repurposing existing spaces for isolation spaces, optimizing resource allocation, developing protocols for judicious resource use, prioritizing patients based on severity, enhancing staffing levels (addressing staff shortages by hiring additional workers, redeploying staff from non-urgent areas, and seeking support from external sources, like volunteers), implementing infection control measures like regular staff testing, enhancing testing and contact tracing, expanding telehealth services (remote consultations, monitoring patients, and triage cases). Flexibility, adaptability, and the continuous evaluation of interventions are crucial for an effective response to the surge in COVID-19 cases. In Portugal, a reform of letting patients be interned in their own home is being tested, which could be a solution for non-COVID-19 cases requiring hospital admission and stay at ward during the surge.
The scarcity of critical resources such as personal protective equipment, ventilators, and testing supplies created challenges in providing optimal care for both COVID-19 patients and those with other medical conditions. Limited resources could affect the ability to offer timely and appropriate treatment. Surpassing surmounting resource constraints in hospitals during a surge can be challenging, but there are strategies that can help optimize resource utilization and alleviate some of the constraints: resource allocation and prioritization through protocols and guidelines prioritizing patients based on clinical need and prognosis; collaboration among healthcare facilities and providers within a region and resource sharing to distribute the burden of patient care; seeking external support via partnerships with other organizations, such as government agencies, non-profit organizations, private businesses, and local communities to raise awareness about the resource needs; resource conservation and efficiency of usage through proper training and education, implementing systems to track and manage inventories effectively as well as protocols to reduce waste via evidence-based practices to minimize unnecessary procedures, tests, or treatments; staff optimization via telehealth and reallocation; and supply chain management via enhanced coordination and communication with suppliers and vendors to anticipate and address potential shortages.
Healthcare workers were at the forefront of the pandemic response, working tirelessly under immense pressure. Staff shortages due to illness, burnout, or the need for self-isolation further strained hospitals’ ability to maintain high-quality care. Burnout among health staff is a critical aspect during surges, and addressing/mitigating its impact is crucial for maintaining their well-being and effectiveness. Some strategies include prioritizing staff well-being, creating a supportive environment that values and prioritizes their mental and emotional health, encouraging open communication and providing channels for staff to express concerns or seek support, and avoiding excessive workloads and fatigue by improving staffing levels, while allowing for flexible scheduling and breaks. Recognition, appreciation, training, education, clear communication, transparency, and employee involvement in decision-making are also important strategies related to implement.
To allocate resources and reduce the risk of COVID-19 transmission, many hospitals had to delay or defer non-urgent procedures and appointments. This situation could lead to longer waiting times, the potential worsening of certain conditions, and overall dissatisfaction among patients. Avoiding care delays or deferrals during pandemics requires a careful balance between managing the surge’s cases and ensuring timely access to essential healthcare services. For that, one should try to minimize care delays via risk stratification and prioritization based on severity, the introduction or enhancement of telehealth services for remote consultations, follow-ups, and monitoring for non-urgent or low-risk cases, creating dedicated COVID-19 facilities, improving the collaboration with primary care providers to ensure that non-urgent care is appropriately managed outside of the hospital setting, and finally continuously monitoring and evaluating the impact of care delays or deferrals on patient outcomes.
Hospitals implemented strict infection control protocols to prevent the spread of COVID-19 among patients and healthcare workers. While necessary, these measures, such as isolation protocols and visitor restrictions, could impact the patient experience and affect the quality of care, particularly in terms of emotional support and family involvement.
Many hospitals had to rapidly adapt their care models to accommodate the challenges posed by the pandemic. This included telehealth services, remote monitoring, and triaging mechanisms to manage patient volumes effectively. While these innovations were crucial, they also required adjustments in workflows, and could potentially impact the patient–provider relationship.
It is important to note that the quality of care varied among different regions and hospitals. Some facilities successfully managed the challenges and maintained high-quality care, while others faced more significant difficulties. A closer look at some of the benchmarks during the pandemic period allowed us to see that they have adopted a meaningful set of the abovementioned strategies, and for that reason, they became more resilient against the surge. Healthcare providers have been continuously learning and adapting throughout the pandemic to improve care delivery in the face of ongoing challenges. However, the pandemic features only explain a small share of hospital performance evolution, with the noteworthy exception of access dimensions. But one should note that delayed care and deferrals were mostly the result of the Central Government’s strategy to contain the virus, forgetting that other pathologies needed attention. This caused a worsening of the severity of illnesses, and it is well-known that later stages of diseases are usually costlier than the earlier ones. Aside from the issue of access, the pandemic does not clearly explain the evolution of hospital performance in terms of efficiency and care safety and appropriateness. This result suggests that changes in performance should be interpreted in the light of internal structure (poor preparedness, staff burnout, hysteria, among others). As the World Health Organization has already decreed the pandemic’s end, it is important that healthcare providers and governments prepare themselves for likely future pandemics, by following the best practices (benchmarks) and designing reliable contingency plans [
104,
105,
106].
Academic implications: Beyond the findings about the strategic management of healthcare systems in the presence of a pandemic (vide infra for the practical implications), it is worth mentioning that there are some academic implications for researchers in the economics and operational research fields. In fact, the use of a weight-constrained BoD together with the GDF-based TFP, as a first stage, and then the productivity indices used in the relational model, would be innovative and open doors to future uses, even in different case studies. It should be noted that the use of this kind of models is relevant to assessments of performance levels and potential drivers, especially when the systems’ sustainability is at stake [
107,
108,
109,
110].
Practical implications: The COVID-19 pandemic has exposed significant challenges faced by hospitals, including resource constraints, staffing issues, delayed non-urgent care, and infection control measures. These challenges require strategies to maintain quality care and address the strain on healthcare systems. Our results have highlighted the importance of developing comprehensive contingency plans, optimizing resource utilization, addressing healthcare worker burnout, balancing urgent and non-urgent care, maintaining patient experience and emotional support, adjusting care models, analyzing regional variations in quality of care, and assessing and improving internal structures to enhance the response to pandemics like COVID-19. These strategies can lead to better resource management, improved healthcare worker well-being, optimized patient care, and increased resilience during challenging times. By learning from past experiences and benchmarking against successful strategies, hospitals can improve preparedness for future public health crises. Healthcare providers should also prioritize staff well-being, provide a supportive environment, encourage open communication, and implement strategies to manage workloads and fatigue. Balancing urgent and non-urgent care, patient experience, and emotional support is crucial during pandemics. Adapting care models, analyzing regional variations in quality of care, and assessing internal structures can help hospitals better respond to future challenges. Overall, implementing these strategies can lead to better resource management, improved healthcare worker well-being, optimized patient care, and increased resilience during challenging times.
6. Conclusions
This work has dealt with three important research questions that have not been answered in the literature. Evidence suggests that hospital performance has been heavily conditioned by the Ministry of Health’s external impositions, especially concerning the barriers created to contain the pandemic’s dissemination. The COVID-19 pandemic’s features explain part of the hospitals’ performance after March 2020, especially the access and efficiency dimensions. Although hospitals have adopted safe and appropriate clinical practices internally, some resource availability problems need to be addressed urgently. The lack of medical doctors and nurses in public healthcare provider services is not a new problem, and was exacerbated by the harsh conditions in the workplace created by the pandemic. Introducing policies that condition access to those services increased the existing barriers to the NHS. The postponement or cancellation of medical appointments and surgeries and the constraints on the emergency departments only resulted in the worsening of patients’ quality of life, and increases in costs in the future, because the related clinical status becomes more complex and severe to treat. It becomes evident that these policies put the NHS’s sustainability and mission at stake. If future pandemics occur, it is paramount to assess the past best practices within the Portuguese NHS and internationally. But this benchmarking exercise does not end with identifying those best practices: one needs to adapt and then adopt them in the NHS, such that the healthcare providers’ resilience and capacity to respond to the demand for care are not at risk. In the current times, characterized by an ongoing pandemic, the exodus of clinical staff to the private sector or abroad, along with political instability, it is challenging to implement the required structural reforms that allow the NHS to fulfill its mission. Nonetheless, they are undoubtedly necessary, and we should focus on improving clinical staff working conditions, hiring more staff, fostering meritocracy, reducing bureaucracy, and guaranteeing equitable access to quality healthcare for the entire population (in particular, the people living in the countryside).
Our study is not absolutely foolproof, as one may identify some limitations. In particular, the variables and models used to quantify hospital performance can always be debatable. In fact, some dimensions, like the in-hospital mortality for low-severity cases, can be included to improve the performance estimation. Of course, we are always bounded by the availability, which may hamper the model’s refinement. Regarding the model, we should highlight its compensatory and benevolent nature, meaning that larger weights are assigned to variables in which hospitals have better performances, while zero-weights can be obtained when the performance is low. This is because the optimization model seeks to maximize performance, so no other system of weights would result in a higher performance score. This is simultaneously a merit and a drawback of the method, as some variables may outweigh others, as long as this would result in a larger score. That being said, it is important to compare our results against those achieved with, for example, a pessimistic benchmarking model.