Risk Management and Patient Safety in the Artificial Intelligence Era: A Systematic Review

Background: Healthcare systems represent complex organizations within which multiple factors (physical environment, human factor, technological devices, quality of care) interconnect to form a dense network whose imbalance is potentially able to compromise patient safety. In this scenario, the need for hospitals to expand reactive and proactive clinical risk management programs is easily understood, and artificial intelligence fits well in this context. This systematic review aims to investigate the state of the art regarding the impact of AI on clinical risk management processes. To simplify the analysis of the review outcomes and to motivate future standardized comparisons with any subsequent studies, the findings of the present review will be grouped according to the possibility of applying AI in the prevention of the different incident type groups as defined by the ICPS. Materials and Methods: On 3 November 2023, a systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines was carried out using the SCOPUS and Medline (via PubMed) databases. A total of 297 articles were identified. After the selection process, 36 articles were included in the present systematic review. Results and Discussion: The studies included in this review allowed for the identification of three main “incident type” domains: clinical process, healthcare-associated infection, and medication. Another relevant application of AI in clinical risk management concerns the topic of incident reporting. Conclusions: This review highlighted that AI can be applied transversely in various clinical contexts to enhance patient safety and facilitate the identification of errors. It appears to be a promising tool to improve clinical risk management, although its use requires human supervision and cannot completely replace human skills. To facilitate the analysis of the present review outcome and to enable comparison with future systematic reviews, it was deemed useful to refer to a pre-existing taxonomy for the identification of adverse events. However, the results of the present study highlighted the usefulness of AI not only for risk prevention in clinical practice, but also in improving the use of an essential risk identification tool, which is incident reporting. For this reason, the taxonomy of the areas of application of AI to clinical risk processes should include an additional class relating to risk identification and analysis tools. For this purpose, it was considered convenient to use ICPS classification.


Introduction
Healthcare systems represent complex organizations [1] within which multiple interconnected factors (physical environment, human factors, technological devices, quality of care) form a dense network whose imbalance is potentially able to compromise patient safety [2].The latter, defined as "the absence of preventable harm to a patient and reduction of risk of unnecessary harm associated with healthcare to an acceptable minimum" [3], is a fundamental principle of healthcare and is part of the patient's rights.According to the World Health Organization (WHO), there is a one in a million chance of aviation accidents, while the possibility of harming a patient in the Clinical process is one in three hundred [3].As specified by the Global Patient Safety Action plan 2021-2030, patient safety incidents are a growing problem and one of the major causes of death and disability worldwide [4].In this scenario, the need for hospitals to expand clinical governance programs is easily understood.Clinical governance is defined as the system through which healthcare organizations improve the quality of care and guarantee high standards of care, striving for excellence [5].
One of the main pillars of this healthcare quality system is clinical risk management, which refers to the set of proactive and reactive clinical tools, procedures, and processes used to detect, monitor, reduce, and prevent potential risks and errors in clinical practice to safeguard patient safety [6].Risk assessment instruments include incident reporting [7], reviews of medical records, safety walk-arounds, administrative data obtained from hospital discharge forms, patient complaints, and information derived from claims litigation.These methods allow for the identification of potential or actual problems that may cause or have caused an adverse event for the patient or healthcare workers.Risk can be managed by using several approaches, including the FMEA and the FMECA [8,9], morbidity and mortality review, clinical auditing [10], significant event auditing, the London Protocol, the SHELL model, and root cause analysis [11].The information collected after identifying and analyzing the biases present in the clinical care process is preparatory to the introduction of future risk prevention strategies, aimed at improving the quality of care.This objective is achieved through the introduction or implementation of procedures and protocols, by ensuring continuous training for healthcare workers and introducing new technologies [12].
Nowadays, the introduction and development of new technologies in the healthcare sector is going to transform medical service and offer the opportunity to promote harm minimization.Artificial intelligence (AI), the application of which has grown exponentially in various sectors in recent times, fits well in this context [13].
AI, coined in 1956 [14], is a modern informatics technology that simulates human behavior and makes devices efficient for achieving tasks that usually require skilled human intelligence.It is used in different fields and over the past few years, AI application in medicine has been growing [15].AI encompasses several domains, such as machine learning (ML), which relies on different algorithms to learn and improve from experience without being programmed [16]; deep learning (DL), based on artificial neural networks [17]; or speech recognition, that is, the ability of a machine to convert a speech signal into a sequence of words, creating an interface between humans and technology [18].AI can enhance medical achievements using automated clinical decision support (CDS), which assists health workers in making complex decisions in clinical practice by combining relevant patient information [19].This application is facilitated by increasing diffusion of electronic medical record (EMR) and computerized provider order entry (CPOE) systems [20].Moreover, natural language processing (NLP) is a technique able to convert unstructured text (e.g., medical records) into datasets easily analyzable by ML models [21].Lastly, AI supports Internet of medical things (IoMT) technologies, which are bio-analytical tools capable of collecting, analyzing, and transmitting health data to increase the efficiency of human care [22].
Previous studies have reported that AI can improve the quality of healthcare [23], although its application still has numerous limitations [24].This systematic review aims to investigate the state of the art regarding the impact of AI on clinical risk management processes.The International Classification for Patient Safety (ICPS), developed by the WHO, provides a taxonomy for the types of healthcare incidents that can occur, grouping them according to common characteristics, and facilitating benchmarks between results deriving from multiple sources, both at a national and an international level [25].To simplify the analysis of the review outcomes and to motivate future standardized comparisons with any subsequent studies, the findings of the present review will be grouped according to the possibility of applying AI in the prevention of the different incident type groups as defined by the ICPS.

Materials and Methods
On 3 November 2023, a systematic review of the literature according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [26] was carried out using the SCOPUS and Medline (via PubMed) databases, using the following search strings: (artificial intelligence OR AI) AND (patient safety); (artificial intelligence OR AI) AND (risk management).

Inclusion and Exclusion Criteria
The inclusion criteria were as follows: case report; original article; short survey; article in English; human study; medical and nursing field; full text available; publication date between 1 January 2013 and 3 November 2023.A ten-year interval research was chosen to focus the review on the most recent studies.
The exclusion criteria were as follows: articles not in English; abstract; editorial; review; erratum; book chapter; note; conference paper.All articles focused on other topics were excluded.

Quality Assessment and Critical Appraisal
M.F. and G.B. independently evaluated the entire texts of the articles.The articles on which there was a disagreement were discussed with the senior investigator, R.L.R., for the final decision.

Risk of Bias
The main risk was linked to the keywords selected for the search strings.Therefore, the Kappa interobserver variability coefficient showed "almost perfect agreement" (0.80) [27].

Characteristics of Eligible Studies
A total of 297 articles were identified; 25 duplicate articles were removed, and 21 articles did not meet the inclusion criteria.After the selection process, 36 articles were included in the present systematic review.

Results and Discussion
Of the 297 articles found, 36 met the inclusion criteria (Figure 1).The main features of each article included are summarized in Table 1.To evaluate the use of an assistive AI device to facilitate image acquisition for regional anesthesia Use of an AI assistive ultrasound device was associated with improved ultrasound image acquisition and interpretation

Clinical process
Hameed et al. [30] To evaluate the value of AI model which identifies safe and dangerous zones of dissection during laparoscopic cholecystectomy AI is a useful tool for providing support and feedback to surgeons Clinical process

Datar et al. [31]
To evaluate the decision-making impact of an artificial intelligence-enabled prognostic test in the management of diabetic kidney disease The test has greater relative importance to primary care physicians (PCPs) than albuminuria and estimated glomerular filtration rate (eGFR) do in making treatment decisions

Clinical process
Scholz et al. [32] To analyze the potential impact an automatic speech recognition (ASR) could have on stroke recognition at emergency medical services An ASR can potentially improve stroke recognition by Emergency Medical Dispatchers (EMDs) and subsequent stroke treatment Clinical process

Torrente et al. [33]
To present an AI-based solution tool for cancer patients' data analysis and improve their management The platform can provide real-time feedback by assessing risk of relapse, performing a stratification of patients, and predicting response to a certain treatment or utility of follow-up tests Clinical process Scala et al. [34] To evaluate use of different artificial intelligence models to predict surgical site infections according to different risk factors The K-nearest neighbors algorithm better handled the imbalanced dataset observing the highest accuracy value

Healthcare-associated infection
Brown et al. [35] To use the electronic prescribing system to identify unintentional prescription of low molecular weight heparins (LMWHs) to patients prescribed direct-acting anticoagulants (DOACs).

The anticoagulant alert prevented duplicate anticoagulant prescription Medication
Festor et al. [36] To evaluate the safety of AI-based clinical decision support systems in sepsis treatment AI consistently leads to a lower number of unsafe decisions in different clinical scenarios

Clinical process
Levivien et al. [37] To attest to whether prescriptions with low risk of drug-related problems (DRPs) ruled out by a digital tool using machine learning with AI in everyday practice were effectively free of any DRPs with potentially severe clinical impact To evaluate the feasibility of using Unified Medical Language System (UMLS) semantic features for automated identification of reports about patient safety incidents by type and severity UMLS-based semantic classifiers were effective in identifying incidents by type and extreme-risk events

Incident reporting systems
Fong et al. [43] To identify health information technology (HIT)-related events from patient safety event (PSE) report free-text descriptions The feature constraint model provides a method to identify HIT-related patient safety hazards using a method that is applicable across healthcare systems with variability in their PSE report structures

Incident reporting systems
Lu et al. [44] To evaluate various text classification methods of adverse nursing events based on deep learning The results show the exceptional performance of the proposed mechanism in terms of various evaluation metrics

Incident reporting systems
King et al. [45] To use machine learning (ML) models to predict erroneous medication orders and identify their contributing factors The methodological approach of using ML algorithms for predicting medication errors has two potential applications: to identify factors associated with order entry errors that potentially represent generalizable knowledge for mitigating such errors and to guide patient safety efforts that are targeted towards medication orders within high-risk contexts Medication Table 1.Cont.

Reference Aim of the Study Findings Safety Domain
Fong et al. (2).[46] To apply active learning techniques to support classification of patient safety event reports as HIT-related Active learning can be used to identify HIT-related events from large datasets where human annotation is a major barrier to understanding trends and patterns in the data

Incident reporting systems
Barmaz et al. [47] To propose a method to compute the probability of AE underreporting that could complement a machine learning mode developed to enhance patients' safety while reducing the need for on-site and manual clinical quality assurance (QA) activities in clinical trials This approach reduces the need for on-site audits, shifting focus from source data verification to pre-identified, higher risk areas; it will enhance further QA activities for safety reporting from clinical trials and generate quality evidence during pre-approval inspections

Incident reporting systems
Zhou et al. [48] To propose an automated pipeline to identify medication event reports and reduce the workload of patient safety experts The pipeline is expected to save time and reduce the workload for clinicians to analyze event reports and better discover valuable information from the reports to facilitate the development of strategies for preventing medication events

Incident reporting systems
Wong et al. [49] To develop a medication rights detection system using natural language processing and deep neural networks to automate medication incident identification using free-text incident reports This study developed a medication rights detection system via DNN to automate medication incident identification using free-text incident reports and provide reference guidelines for training DNN models to classify patient safety incidents The deep learning method shows promise for the efficient exploration of textual reports of medication incidents

Incident reporting systems
Yang et al. [50] To develop a deep learning model to identify allergic reactions in the free-text narrative of hospital safety reports and evaluate its generalizability, efficiency, productivity, and interpretability A deep learning model can accurately and efficiently identify allergic reactions using free-text narratives written by a variety of health care professionals; this model could be used to improve allergy care, potentially enabling real-time event surveillance and guidance for medical errors and system improvement

Ting et al. [51]
To understand how identification confusion of look-alike images by human occurs through the cognitive counterpart of deep learning solutions and to suggest further solutions to approach them This model outperformed identification using conventional computer vision solutions and could assist pharmacists in identifying drugs while preventing medication errors caused by look-alike blister packages Medication Tabaie et al. [52] To explore the use of natural language processing (NLP) algorithms to categorize contributing factors from patient safety events (PSEs) Applying the information-rich sentence selection algorithm boosted contributing factor categorization performance

Incident reporting systems
Table 1.Cont.

Reference Aim of the Study Findings Safety Domain
Zhao et al. [53] To explore ways of using structured electronic health record data that can be exploited to detect a wide range of adverse drug events (ADEs), which could be adopted in a general decision support system that alerts for potential ADEs Machine learning can be applied to electronic health records for the purpose of detecting adverse drug events and proposed solutions

Reference Aim of the Study Findings Safety Domain
Zhu et al. [60] To understand the generalizability of a machine learning automated surgical site infection (SSI) detection algorithms SSI detection machine learning algorithms developed at 1 site were generalizable to another institution The main features of each article included are summarized in Table 1.
Table 1.Main results of the systematic review.

Patient Safety Domains
In January 2009, the World Health Organization published a technical report providing the International Classification for Patient Safety (ICPS), a conceptual framework set out to allow for the codification of patient safety issues [62].Specifically, "incident type" is one of the ten level classes comprising the framework, useful for categorizing patient safety incidents (clinical process, documentation, healthcare-associated infection, medication, blood, nutrition, oxygen, medical device, behavior, patient accidents, infrastructure, resources) [62].
Based on the aforementioned classification, results were grouped into ICPS patient safety domains to facilitate analysis from a risk management perspective.The studies included in this review allowed for the identification of three main domains: clinical process, healthcare-associated infection, and medication.Furthermore, the topic of incident reporting was discussed in an additional paragraph, since thirteen studies focus on this issue.

Clinical Process
In recent times, AI has become increasingly prominent in the healthcare field, mainly in diagnosing, managing, treating, and screening pathologies [63][64][65][66].Consequently, it is intuitive that the use of AI, in a clinical risk management perspective, concerns mainly these aspects of the Clinical process.By applying the ICPS classification, the use of AI in studies aimed at preventing the misinterpretation of radiographic investigations [28,29,56] or the operating field [30], translates into the potential advantage of avoiding inadequate treatments and procedures other than incorrect interventions.
Automated voice recognition software [32] and data analysis [33] have proven valuable in recognizing pathologies such as stroke and cancer, facilitating early detection and avoiding a delay in treatment.CDS systems [31,36] or intelligent checklists [59] can avoid harm to patients deriving from inadequate treatment or non-conformity to guidelines and good clinical practices.

Healthcare-Associated Infections
Healthcare-associated infections (HCAIs) represent a major public health problem, with significant impacts on patients [67].According to the Agency for Healthcare Research and Quality, HCAIs, and in particular sepsis, represent one of the ten leading causes of death in the United States [68].Hence, as mortality in septic patients increases proportionally with the delay of antibiotic treatment, early prediction is crucial for timely interventions.
Machine learning algorithms applied to datasets can map several variables to predict the risk of Surgical Site Infections and sepsis [34,62] that could be undetected by healthcare workers, leading to a considerable reduction of morbidity and mortality-related infections.However, if the ML system is applied on a different dataset from the one on which it is deployed, the so-called dataset shift can occur, leading to the AI's ineffective performance.For this reason, a clinician's supervision is always necessary to evaluate any discrepancy between the clinical evaluation and the AI prediction or external validation to ascertain the general applicability of the ML system [61].

Medication
An adverse drug event (ADE) consists of a medical therapy error that causes harm to the patient.It is estimated that this type of adverse event affects approximately 1 out of every 30 patients in health care [69].
Medication errors are due to different factors, including the human factor [70].A very common error in clinical practice related to the human factor is prescription error [71].One of the best-known human factor-related prescription error examples is that of lookalike or sound-alike (LASA) drugs, in which the error occurs due to orthographic or phonetic similarity or packaging between drugs [72,73].AI could be useful in clinical practice to prevent so-called "look-alike" errors due to similar packaging between different medications by applying deep learning to drug images [52,58].
Furthermore, the error due to the simultaneous prescription of drugs that have the same effect can be avoided through the use of CDS systems which generate an alarm capable of alerting the healthcare professional [35].However, these systems require high accuracy and a low number of alerts burns and false positives to avoid generating further stress in the healthcare worker, which would contribute to fueling the risk of errors related to the human factor [60].
Several studies have demonstrated that over half of ADEs occur during drug ordering [74].To promote the best pharmaceutical care, it is essential to carry out a patient's medication review, which is a critical evaluation of the drugs assumed to evaluate potential interactions and consequent adverse reactions.However, this part of clinical practice is often not exhaustive and takes a long time [75].Machine learning combined with CDS systems or applied to data extracted from electronic health records are able to support pharmacovigilance activities and appears to be safe in performing medication reviews [37,54,55].In addition, natural language processing linked to clinical notes can determine the reason for the pharmaceutical prescription to verify its appropriateness [57].
Computerized provider order entry (CPOE) systems reduce the risk of misinterpretation of the pharmacological order, as the integration of these systems with electronic health records promotes coordination of drug ordering, and cooperation between healthcare professionals allows for contextual collaboration between healthcare staff [74][75][76].On the other hand, the interaction of the healthcare worker with the CPOE has proven to be a source of error [77,78], potentially hesitating in an ADE.In this regard, the prevention of these adverse events can be promoted through the use of machine learning systems capable of identifying the factors predisposing to medication ordering errors [46].

Incident Reporting Systems
In healthcare facilities, incident reporting systems are essential for managing clinical risk by notifying providers of adverse events [79].It consists of reporting adverse events, near misses, risks, and potentially unsafe conditions to healthcare professionals [80], including falls, HCAIs, transfusion [81], and patients' and operators' aggressions [82].By using databases, healthcare facilities can identify, map, and analyze adverse events that occur to prevent them from occurring again.One of the limitations of this method concerns the inexperience of the different categories of healthcare professionals who carry out the reporting [83].It has been noted in several studies that an absence of codified terminology is one of the main obstacles [84,85], resulting in an under-analysis of the event and, consequently, an inability to learn from it.Due to the high volume of data collected by the IT systems responsible for these purposes, using free text for reporting these events reduces its effectiveness due to the difficulty in aggregating the data [86].Of the 36 articles included in this systematic review, 13 [38][39][40][41][42][43][44][47][48][49][50][51]53] investigate the applications of AI to improve the efficiency of incident reporting systems.The studies predominantly focus their attention on the possibility of standardizing events according to their type and severity.Furthermore, machine learning systems can evaluate the reporting rates of adverse events and estimate the risk of under-reporting [48] or to analyze contributing factors that predispose to their genesis [53].
The use of AI in this regard has a dual purpose.On the one hand, it reduces the workload of human work risk management staff, allowing them to dedicate themselves to other activities to implement patient safety [47,87].On the other hand, the conversion of unstructured data into structured information has proven effective in identifying situations that are potentially fatal or capable of causing serious harm [51], prioritizing adverse events with significant consequences for patients [40].The potential exclusive application of AI in real-world settings requires further studies, since a part of a reported event can be related to more than one type of accident [38].In addition, the use of abbreviations or acronyms requires manual review [43,49].At present, AI systems are not able to perfectly replace manual review [39], and the machine can only act as a support to risk management experts.

Conclusions
This review highlighted that AI can be applied transversely in various clinical contexts to enhance patient safety and facilitate the identification of errors.To facilitate the analysis of the present review outcome and to enable comparison with future systematic reviews, it was deemed useful to refer to a pre-existing taxonomy for the identification of adverse events.
For this purpose, it was considered convenient to use ICPS classification, which includes ten classes of "incident type".The main fields of application, according to the codification mentioned above, concern the prevention of errors concerning clinical processes, medication errors and the development of HAIs.
Additionally, the results of the present study highlighted the usefulness of AI not only for risk prevention in clinical practice, but also in improving the use of an essential risk identification tool, which is incident reporting.It follows that ICPS classification could be limiting for the analysis of the application of AI to clinical risk management systems, as it relates to the clinical aspects of healthcare risk.For this reason, the taxonomy of the areas of application of AI to clinical risk processes should include an additional class relating to risk identification and analysis tools.
The advantages of using AI in risk management systems translate into a reduction in the workload of the risk manager, who can devote more time to developing procedures and paths to prevent mistakes, and the healthcare staff, who can spend more time with patients.However, aside from diminishing existing risks, as in the case of dataset shift, there is also the possibility of introducing new risks, such as false positive alerts that increase cognitive stress, thereby enabling human error.Furthermore, its use necessarily requires human supervision.To conclude, AI appears to be a promising tool to improve clinical risk management, although its use requires human supervision and cannot completely replace human skills.

14 Figure 1 .
Figure 1.PRISMA flow diagram of this systematic review.

Figure 1 .
Figure 1.PRISMA flow diagram of this systematic review.

Table 1 .
Main results of the systematic review.