Using Failure Mode, Effect and Criticality Analysis to Improve Safety in the COVID Mass Vaccination Campaign

Vaccination against SARS-CoV-2 will likely be the most promising way to combat the pandemic. Even if mass vaccination is urgent, it should still always be supported by appropriate patient safety management. The aim of this study, based on failure mode, effects and criticality analysis (FMECA), was to identify possible failures and highlight measures that can be adopted to prevent their occurrence. A team of resident doctors in public health from the University of Padua and specialists in risk analysis in public health examined the mass vaccination process. A diagram was drafted to illustrate the various phases of mass vaccination, analyze the process, and identify all failure modes. Criticalities were ascertained by rating the severity, frequency and likelihood of failure detection on a scale of 1 to 10. We identified a total of 71 possible faults distributed over the various phases of the process, and 34 of them were classified as carrying a high risk. For the potentially high-risk failure modes, we identified 63 recommended actions to contain the cause of their occurrence or improve their detection. For the purpose of detecting potential failures, FMECA can be successfully applied to mass vaccination, which should be considered a high-risk process.


Introduction
Deadly pandemics and large-scale epidemics have challenged human existence throughout history. While these crises were once separated by centuries, or at least many decades, they are now becoming much more common. Since 2003, we have experienced severe acute respiratory syndrome (SARS) (a near pandemic), an influenza pandemic (H1N1pdm in 2009), a chikungunya pandemic (2014), a Zika pandemic (2015), and a widespread pandemic-like extension of Ebola over five African countries, with cases exported globally (2014 to 2015) [1].
SARS-CoV-2 is a new type of coronavirus that causes a serious, contagious disease (COVID-19) [2]. It was first reported in Wuhan, China in November 2019 and WHO confirmed the first case on 31 December 2019. Since then, it has been spreading rapidly and the outbreak was declared a global pandemic on 11 March 2020 [3,4]. The rapid worldwide spread of the virus has resulted in over 140 million people becoming infected and more than 3 million deaths (data updated to April 2021), causing a global health, social and economic crisis [5].
Although hundreds of clinical trials have been initiated since the outbreak of COVID-19, an antiviral drug that is effective in all patient groups has yet to be developed and assessed. Hence the urgent need to vaccinate the whole population against the SARS-CoV-2 virus. Vaccination will likely be the most effective way to counter the COVID-19 pandemic [6]. The process of drug discovery is costly and time-consuming. Only widespread preventative vaccination can help to lower the burden of the pandemic, playing a pivotal part in efficiently and sustainably protecting people from viral infections, and either eliminating or significantly reducing their transmission within the population [7]. Inducing herd immunity by means of mass vaccination programs has been a very successful strategy for preventing the spread of other infectious diseases. It also protects the most vulnerable population groups unable to develop immunity, such as individuals with immunodeficiencies or a weakened immune system due to underlying medical or debilitating conditions [8].
While the development of a safe and effective COVID-19 vaccine has not been easy, its production and distribution, and its administration especially, also pose extraordinary challenges [6]. Mass vaccination can be seen as a critical process because of the complexity of the organizational machine needed to manage huge numbers of people as quickly as possible, in other spaces as well as healthcare facilities, while always maintaining patient safety standards.
Two methods are used to manage the risks associated with critical processes. One is reactive: measures are taken after an adverse event has occurred to prevent it from happening again. The other is proactive: processes are analyzed a priori to prevent adverse events from occurring in the first place. One of the most popular proactive methods is failure mode, effect and criticality analysis (FMECA). According to the Joint Commission, FMECA is a systematic, analytical technique used prospectively by a team to prevent the appearance of problems associated with a process before they occur [9]. This is a step-by-step approach to identifying "failures" that lead to poor-quality, unsafe, unreliable, or inefficient care [10]. It is based on the concept that a risk is related not only to the likelihood of a failure occurring, but also to the severity of the failure's consequences and the feasibility of detecting and intercepting a failure before it occurs. The FMECA approach provides a systematic way to identify failures, and also prioritizes the most important ones for improvement [11]. It was originally developed by engineers to study complex systems and is typically applied to high-risk industries, such as nuclear power generation and commercial aviation, where an error can have very serious consequences [12,13].
These step-by-step assessment methods have increasingly been used to maximize the safety and quality of clinical care in many healthcare settings and are now widely used to proactively evaluate complex clinical processes.
The purpose of this study was to conduct a prospective and systematic analysis of the various stages of the mass vaccination process, applying FMECA to identify possible errors and enable preemptive measures to be taken.

Materials and Methods
The FMECA was conducted in accordance with the process described in Figure 1. The analysis was conducted in April 2021, about 4 months after the start of the COVID-19 vaccination campaign in Italy when about 7,401,431 people (12.3% of the total population) had received a single dose of vaccine while 3,333,644 (5.5% of the total population) had completed the vaccination course [14].
After an introductory session held with a team of experts to explain the features of FMECA, mass vaccination against SARS-CoV-2 was identified as the high-risk process to investigate. The team consisted of resident doctors in public health from the University of Padua who were actively involved in the mass vaccination program underway in Veneto (Italy) in different local health units, together with in public health specialists with expertise in risk analysis. A process diagram was drawn up, illustrating the various phases of the mass vaccination process ( Figure 2). The analysis was conducted in April 2021, about 4 months after the start of the COVID-19 vaccination campaign in Italy when about 7,401,431 people (12.3% of the total population) had received a single dose of vaccine while 3,333,644 (5.5% of the total population) had completed the vaccination course [14].
After an introductory session held with a team of experts to explain the features of FMECA, mass vaccination against SARS-CoV-2 was identified as the high-risk process to investigate. The team consisted of resident doctors in public health from the University of Padua who were actively involved in the mass vaccination program underway in Veneto (Italy) in different local health units, together with in public health specialists with expertise in risk analysis. A process diagram was drawn up, illustrating the various phases of the mass vaccination process ( Figure 2). Analyzing the process enabled all failure modes to be identified, and criticalities were pinpointed by rating these failure modes in terms of their: occurrence (probability of the event occurring); detectability (probability of the event going undetected, and of it reaching the patient); and severity (effect of the error on the patient). Each item was classified on a scale of 1 to 10. Table 1 shows the rating scale used in this study, which is an adapted  The analysis was conducted in April 2021, about 4 months after the start of the COVID-19 vaccination campaign in Italy when about 7,401,431 people (12.3% of the total population) had received a single dose of vaccine while 3,333,644 (5.5% of the total population) had completed the vaccination course [14].
After an introductory session held with a team of experts to explain the features of FMECA, mass vaccination against SARS-CoV-2 was identified as the high-risk process to investigate. The team consisted of resident doctors in public health from the University of Padua who were actively involved in the mass vaccination program underway in Veneto (Italy) in different local health units, together with in public health specialists with expertise in risk analysis. A process diagram was drawn up, illustrating the various phases of the mass vaccination process ( Figure 2). Analyzing the process enabled all failure modes to be identified, and criticalities were pinpointed by rating these failure modes in terms of their: occurrence (probability of the event occurring); detectability (probability of the event going undetected, and of it reaching the patient); and severity (effect of the error on the patient). Each item was classified on a scale of 1 to 10. Table 1 shows the rating scale used in this study, which is an adapted Analyzing the process enabled all failure modes to be identified, and criticalities were pinpointed by rating these failure modes in terms of their: occurrence (probability of the event occurring); detectability (probability of the event going undetected, and of it reaching the patient); and severity (effect of the error on the patient). Each item was classified on a scale of 1 to 10. Table 1 shows the rating scale used in this study, which is an adapted version of those used in already-published studies. In particular, to quantify the potential effects of a failure mode associated with the loss of vaccine and delays in the vaccination of prioritized population groups, a rating higher than 7 was assigned to the severity of all the related failure modes. The same applied to all events that were not an immediate source of danger to users, but were potential sources of contagion, such as gatherings of people failing to comply with social distancing requirements. For each possible failure, five team members assigned a score to the three rating scales for severity, frequency and likelihood of detection. The numerical score quantifying the three items was used to calculate a risk priority number (RPN), which is the product of the severity, occurrence and detectability scores (RPN = occurrence × detectability × severity). The median value of the failures' RPNs was used to assess the final score, and we also reported the ranges of the risk priority numbers (RPN).
Thus, as part of the FMECA, the RPN is a numerical estimate of a risk attributed to a process, or a step in a process; it identifies the elements most likely to contribute to medically serious failures.
The maximum RPN is 1000 and any RPN > 100 was used to identify a high-risk failure [15]. This is a conventional value that is only used to prioritize failures, but all failures should be addressed, albeit more or less promptly.
For each failure mode, the team analyzed the criticality index and decided whether the risk was acceptable or improvements were needed. The highest scoring failures were considered high-risk and identified as priority areas in which improvements, through safety strategies, should be made.

Results
The assessment process detected a total of 71 possible failures distributed over the various phases of the process (Figure 2), and 34 of them were classified as high-risk. The phases with the largest number of possible failures were: "spatial layout of vaccination center (MVC)" (14 failures); and "pre-vaccination screening" (13 failures).
The RPNs obtained (Table 2) ranged from 10 to 378. Every phase of the mass vaccination process, with the exception of "access to area outside vaccination center", "reception prior to entering vaccination center" and "exit", were potentially critical with RPNs > 100. The most important failures producing the highest RPNs were found in the "postvaccination observation" and "pre-vaccination screening" phases. They included, for instance, "Failure to investigate the clinical history (e.g., previous positivization, prior anaphylaxis for vaccine components)", "Erroneous medical details provided by user" and "Users leave MVC before completing the required observation period (based on their risk factors)". Presence of unnecessary companions 10 (7-60)  For the potential high-risk failure modes, at least one recommended action was identified with a view to reducing their occurrence or improving their detection. This resulted in the 63 recommendations listed in Table 3. Table 3. Actions recommended to improve the mass vaccination system.

Process Step Recommended Actions
Spatial layout of MVC
Signaling areas with inadequate flooring, notifying the technical office for repairs, identifying alternative routes 3.
Identifying alternative routes for users with disabilities 4.
Improving signage and removing architectural barriers 5.
Ensuring availability of hand sanitizer or equipped handwashing stations at entrance to vaccination sites and health facilities 6.
Installing and monitoring air conditioning system 7.
Periodically checking web security and connection 8.
Installing temporary lighting fixtures to ensure adequate local lighting 9.
Providing a backup alternative current generator Table 3. Cont.

Process Step Recommended Actions
Vaccination programming lists 1.
Reminders with correct dates of appointments for vaccination 2.
Opportunity for self-booking by users belonging to previously-validated lists (selection based personal data or prescription charge exemptions, or lists of workers) 3.
Creating automated date calculators and dedicated workstations to notify users of appointments for second doses of vaccine before they leave 4.
Cross-referencing of users' tax codes with prescription charge exemption database for diseases and pharmaceutical flows 5.
Cross-referencing of users' tax codes with their vaccination history 6.
Cross-referencing of users' tax codes with results of SARS-CoV-2 tests (to identify previous COVID patients, who require only one dose of vaccine) Access to area outside MVC 1.
Suitable choice of location for MVC, taking parking space into account 2.
Adequate signage of user pathways Reception prior to entering MVC

1.
Organize periodic check on instrumental equipment for triage access 2.
Identify separate outside space for users with high body temperature (BT) awaiting second temperature check 3.
Adequately marking routes with markers for distancing, and for directing users where to go Check-in (personal identification and documentation of appointment)

1.
Create clear and standardized communication channels for the whole team conducting the vaccination campaign. Avoid using oral communications or unofficial channels (e.g., use mail, not messages) 2.
Diversify tasks for colleagues during a single shift to improve concentration 3.
Avoid permissiveness in reception phase (e.g., clear criteria regarding accompanying persons eligible for vaccination) Vaccine storage and conservation 1.
Staff training on general storage and handling principles, and standard operating procedures for vaccine management 2.
Designate a primary vaccine coordinator, responsible for ensuring all vaccines are stored and handled correctly 3.
Adequacy of refrigerator maintenance program 4.
Provide for remote temperature control 5.
Ensure the manager/pharmacist to call in the event of an alarm can be reached by phone 6.
Training of staff loading-unloading vaccines 7.
Monitoring of vaccine transport contract based on a checklist 9.
Automatic reporting on daily stocks to the MVC manager 10. Equipment check list for warehouse handling vehicles and containers 11. Alarms connected with a security service Reconstitution of vaccines 1.
Storage in separate refrigerators with clearly-visible labels/signs: use different colors, and avoid waste by rotating stock to ensure the first in is the first out 2.
Organize vaccination sessions with no more than 2 types of vaccine 3.
Staff training on reconstitution of different types of vaccine 4.
Suitable environment: minimize distractions; use a dedicated room, physically separated from the clinical activities; use separate workstations with dedicated staff for the various types of vaccine; prepare all the materials needed to dilute 1 bottle with a final check (e.g., that 6 syringes have actually been filled from 1 Pfizer vial) 5.
Create operating instructions and have flowcharts on display 6.
Write the date and time of thawing on the vial and check that all vials have been labeled with the "use-by" date and time alternative routes for the disabled, solutions to guarantee social distancing) involved in adapting spaces normally used for other activities but able to accommodate large numbers of people and ensure patient safety.
In the phase where the vaccines are reconstituted, the RPNs were ≥200 for every failure mode identified ("Incorrect identification of vials", "Errors in dilution procedure", "Failure to recognize vials of vaccine to be rejected", and "Failure to identify use-by date and time"). This finding could be due to the complexity of the procedures and the high workflow typical of mass vaccination resulting in failures that are hard to detect but have potentially severe effects on public health (by reducing the efficacy of vaccination) and users' health (by causing anaphylactic reactions). For many years, the emphasis has been on the adverse effects of vaccines after their administration, while little is known about the delicate phase of vaccine dose preparation. Paparella's work [18] highlights the importance of the various errors occurring in this phase of vaccination programs. Our work replaces her findings because one of the most commonly reported errors concerned the failure to identify the use-by date and time on a vial of vaccine. Strategies to avoid confusion and harm to users have already been published, such as: weekly checks for expired vaccines as well as each time a vial is collected from the stock; avoiding waste by rotating stock to ensure that the first in is the first out; checking people's age and comorbidities before selecting and administering vaccines; and keeping track of immunization schedules. Considering their significant impact and the different settings and workflows analyzed, we propose some additional prevention strategies, such as: dedicating a room that is physically separate from the clinical area; using separate workstations with dedicated staff for the various types of vaccine; creating operating instructions and having flowcharts on display; writing the date and time of thawing on the vial; checking that all vials have labels showing the use-by date and time; and training staff on the reconstitution of the different types of vaccine.
Judging from the RPNs, the pre-vaccination screening and post-vaccination observation phases should be considered the phases at highest risk: "Failure to investigate clinical history (e.g., prior anaphylaxis for vaccine components)" or "Users leave MVC before completing the required observation period (based on their risk factors)" could have the most severe effects on users (adverse events in an unprotected environment, or a patient's death). We recommend several actions to prevent these most risky failures, such as giving users written notification of their exit time, checking users at the exit, or giving users precise indications on potential symptoms of an adverse reaction. The most important recommendation, however (also contained in the COVID-19 Vaccination Program: Guidance for Healthcare Practitioners published by Public Health England [19]), is to ensure that staff are continuously given updated training on COVID-19 vaccine contraindications and precautions.
As the RPNs show, the vaccination phase also deserves healthcare workers' utmost attention. A systematic review of the medical literature showed that the most common vaccination-related error involved the administration of the wrong vaccine. Several trends were identified relating to this issue, including the incorrect administration of vaccine products with similar names or products licensed for use in specific age groups. Vaccines with different age-based formulations tended to be more often associated with administration errors than vaccines with a single dosage for all ages. Although most errors were reported as having caused "no adverse reaction" [20], we judged that the same failure should be considered a high-risk failure because of the presence of specific recommendation for different age and comorbidity groups. We recommend some actions to take, including "Organize separate lines for different vaccines, with clearly-marked routes", and "Accurately check the vaccine prescription and register the type of vaccine administered (equip outpatient clinics with IT support)".
To the best of our knowledge, this study is the first to have applied FMECA to the mass vaccination process and the organization of an MVC with a view to making recommendations for improvements to ensure the safety of the mass vaccination process.
FMECA is liable to several limitations, the main one being the unavoidably subjective selection of the failure modes and calculation of the criticality indexes. To minimize this problem, explicit criteria were stipulated for assessing the frequency, severity and detectability of failures, and every failure was discussed by all of the team members. That said, although the failures reported could have external validity and be more broadly applicable elsewhere, the heterogeneity of healthcare systems and structural features in different countries (regarding the availability of information technology systems, for instance) makes the definition of RPN-based scores context-specific (e.g., healthcare systems supported by technology but without integrated and unified databases). In further research on the topic of FMECA, we suggest interviewing personnel employed in other countries in order to draw up a more generalized model.

Conclusions
FMECA can be usefully performed on a mass vaccination process to help identify potential failures. Safety strategies were recommended for each failure mode identified by our analysis and these recommended actions could be considered for practice and for further studies in the field.