A Tool to Retrieve Alert Dwell Time from a Homegrown Computerized Physician Order Entry ( CPOE ) System of an Academic Medical Center: An Exploratory Analysis

: Alert dwell time, defined as the time elapsed from the generation of an interruptive alert to its closure, has rarely been used to describe the time required by clinicians to respond to inter ‐ ruptive alerts. Our study aimed to develop a tool to retrieve alert dwell times from a homegrown CPOE (computerized physician order entry) system, and to conduct exploratory analysis on the impact of various alert characteristics on alert dwell time. Additionally, we compared this impact between various professional groups. With these aims, a dominant window detector was developed using the Golang programming language and was implemented to collect all alert dwell times from the homegrown CPOE system of a 726 ‐ bed, Taiwanese academic medical center from December 2019 to February 2021. Overall, 3,737,697 interruptive alerts were collected. Correlation analysis was performed for alerts corresponding to the 100 most frequent alert categories. Our results showed that there was a negative correlation ( ρ = − 0.244, p = 0.015) between the number of alerts and alert dwell times. Alert dwell times were strongly correlated between different professional groups (phy ‐ sician vs. nurse, ρ = 0.739, p < 0.001). A tool that retrieves alert dwell times can provide important insights to hospitals attempting to improve clinical workflows.


Introduction
Interruptive alerts, in the form of suspended pop-up windows, have been widely used in clinical decision support (CDS) and computerized physician order entry (CPOE) systems to prevent medical errors during diagnosis and treatment [1][2][3]. In contrast to soft-stop and passive alerts, interruptive alerts require the physician to take one or more actions to close the alert dialog box. Although several studies have shown that interruptive alerts are an effective way to prevent dangerous prescriptions by ensuring that physicians are likely to see them, they can also lead to unintended consequences [4][5][6][7], such as additional cognitive burden, delays in the delivery of appropriate therapy [8][9][10], and general alert fatigue.
Alert fatigue, defined as "the mental state that is the result of too many alerts consuming time and mental energy [11,12]", usually occurs when interruptive alert systems unnecessarily and frequently distract clinicians from their thought processes [13]. Consequently, both clinically important and unimportant alerts may be ignored. Causes of alert fatigue include, but are not limited to: (1) a large number of alerts [14]; (2) clinically irrelevant alerts or alerts of low clinical value [15,16]; and (3) miscommunication or lack of communication of the meaning of the alert [17].
Previous studies have primarily used alert override rates (total number of alerts that do not change the physician's prescribing behavior divided by the total number of alerts presented to the physician) to evaluate alert efficiency [18][19][20]. However, using alert override rates to evaluate alert efficiency still assumes that users read alerts to determine their relevance [21]. In this study, we assess "alert dwell time" instead, defined as the time elapsed from an interruptive alert being generated to it being closed or overridden, which more broadly quantifies the time required for clinicians to respond to interruptive alerts, whether they read them or not [22].
To date, few studies have used alert dwell time to evaluate the efficiency of their alert system, with the majority focusing only on clinical alerts (e.g., prescribing and drug-drug interactions) [21][22][23]. However, CPOE systems like ours generate both clinical and administrative (low or no clinical relevance) alerts [24][25][26][27], with up to 80% being administrative as reported in our previous study [16]. Therefore, neglecting administrative alerts may limit the improvements that can be made to the alert system, especially in relation to alert fatigue.
Here, we studied alert dwell times for both administrative and clinical alerts. To do so, we developed a window retrieval technique, which detects how much time the user stays in every program window while operating the computer. Using data collected by the alert log collector (ALC, a method used to collect alerts in a homegrown CPOE system) [16], the specific alert dwell times for each alert could be identified. Additionally, we conducted an exploratory analysis of the impact of common alert characteristics (i.e., number of alerts and alert message length) on the alert dwell time, as well as potential differences between professional groups (physician, nurse, other).

The Implementation of Alert Log Collector and Dominant Window Detector
The implementation of an alert log collector was described in our previous study [16]. The Go (Golang) programming language (Version 1.15.7) was used to build the dominant window detector, operating independently from the hospital's CPOE system. The database was maintained on MySQL. A demonstration of the dominant window detector program is on GitHub: https://github.com/alanjian/DominantWindowDetector (accessed on 16 March, 2021). The dominant window detector was first implemented in the Taipei Municipal Wan Fang Hospital (WFH) in December 2019, running continuously on the computer unless terminated by the user or from an unexpected system crash. The dominant window is defined as the one that is currently active at the forefront of the computer screen. The principal processes of the dominant window detector are in Supplementary Materials. When the user changed the dominant window, the dominant window detector recorded data relevant to this study, and transmitted them to the database for further analysis.

Data Combination Process
We used an alert log collector alongside a dominant window detector to collect relevant data [16]. However, the alert log collector only logs the alerts without their respective alert dwell times; meanwhile, the dominant window detector records all alert dwell times without the contents of the alert messages ( Figure 1). Therefore, we combined these data. First, the data merging program checks for consistency in the computer's IP address and dates between records of user-selected windows (collected by the dominant window detector) and alert log records (collected by the alert log collector) ( Figure 2). If consistent, the time gap between the two records is calculated. The process of calculating the time gaps between the respective ALC (Alert Log Collector) and DWD (Dominant Window Detector) records is shown in Figure 3. The smallest time gaps between the dominant window records and the alert log records were interpreted to indicate that these records had been derived from the same alert window (thicker line). For each alert category, titles have corresponding message contents. For example, the message content corresponding to "Alternative medication remind" is "[Drug A] has been suspended. Do you want to use [Drug B] as an alternative option?" (Table S1). However, the content may contain different drugs. Thus, we use "(*)" as a Regular Expression to indicate that this position can be any drug name. Finally, the data merging program checks whether the title of the dominant window and the alert content correspond to predefined pairs of categories. There were 501 predefined pairs in this study. If the window title and alert content were validated, the records were merged successfully. The data merging program keeps checking and merging data until all records have been compared. For our study, the data combination process took about four days to complete.

Data Analysis
The MySQL Relational Database Management System (RDBMS) was used to store and manage all data. The outpatient clinic alerts and window-switching event records in WFH were stored. We tested for normality among variables related to different alert characteristics using Kolmogorov-Smirnov and Shapiro-Wilk tests. We assessed correlation using Spearman's rho (ρ) coefficient. Ranges of absolute ρ (rho) values were used to define strong correlation (0.70-0.99), moderate correlation (0.30-0.69), and weak correlation (0.0-0.29) [28]. IBM's SPSS Statistics 19 (Armonk, NY, USA) was used to compute descriptive statistics, normality tests, and correlation analyses.

Alert Distribution
We successfully developed a dominant window detector using the Golang programming language, and retrieved alert dwell times in the homegrown CPOE system of an academic medical center in Taiwan. The alert log collector was implemented on 11 November 2017, in WFH: in total, 3,737,697 triggered alerts were collected in the 14-month study period (from 25 December 2019, to 20 February 2021). The numbers of alerts were highly concentrated to the top 100 most frequent categories, consistent with the results of our previous study [16]. First, in the data combination process, 462,463 alerts were excluded due to missing alert dwell time values because some clinicians would terminate the dominant window detector before starting the visits. The missingness is 12% for all data, so we believed that removing these data with missing values would not affect our results. A further 28,763 alerts were excluded for being outside the top 100 most frequent categories. Ultimately, 3,246,471 alerts relating to the top 100 most frequent categories were included in our study ( Figure 4).

Normality Test and Descriptive Statistics
The normality tests showed that the alert dwell time, the number of alerts, the length of the alert message, alert dwell time (physician), alert dwell time (nurse), and alert dwell time (other professional group) were not normally distributed (Kolmogorov-Smirnov test statistic p = 0.001; Shapiro-Wilk normality test p = 0.001) (Table S2). Thus, the median was used for average values of each alert characteristic.
The descriptive statistics are shown in Table 1. All the different alert characteristics were represented across 100 unique alert categories. The median number of alerts was 5380 times (range, 1201-760,690 times), and the mean number of alerts was 32,465 times. In order to know whether users spend more time reading alerts with longer messages, the alert message length was calculated as the number of words in the message. Only meaningful words were counted; blanks, punctuation marks, and line breaks were excluded. The median alert message length was 15 words (range, 4-278 words). The mean alert message length was 23 words. The median alert dwell time was 1.

Top 10 Most Frequent Alert Categories and Correlation Analysis
The top 10 most frequent alert categories are displayed in Table 2. Most of the alerts were triggered on the physician's computer (8 out of 10), and most of the alert dwell times were between 1 s and 10 s. For the top 10 most frequent alert categories, the median alert dwell times ranged from 0.027 s to 2.635 s. However, the maximum alert dwell time was up to 9186.9 s, a clear outlier that occurred due to the user being away from the keyboard. There was a weak negative correlation (ρ = −0.244, p = 0.015 < 0.05) between the number of alerts and alert dwell time. This suggests that clinicians tend to spend less time reading the alerts that were triggered more frequently. The examples of less frequent alert categories with longer alert dwell times are in Table S3. Examples of alert categories with higher clinical relevance are in Table S4.
In terms of the impact of alert message length on alert dwell time, this was not statistically significant (p = 0.097 > 0.05). However, there was a strong positive correlation between the alert dwell times of different professional groups, namely physician vs. nurse (ρ = 0.739, p < 0.001), physician vs. other (ρ = 0.811, p < 0.001), and nurse vs. other (ρ = 0.733, p < 0.001). Alert dwell times were strongly correlated between different professional groups. There was a weak positive correlation between alert message length and the number of alerts (ρ = 0.277, p = 0.005 < 0.01).

Discussion
In this study, we successfully developed a dominant window detector through the Golang programming language, which was utilized to retrieve the alert dwell times within the CPOE system of an academic medical center in Taiwan. Few studies to date have reported alert dwell times [21][22][23]. We showed that alert dwell times can be used to assess which categories of administrative and clinical alerts are most time-consuming, and/or are being ignored. By looking for potentially resolvable problems of the CPOE alert system in this way, the clinical workflow can ultimately be improved.
Most CPOE systems use alerts to prevent clinical errors and to optimize the safety and quality of clinical decisions [29], such as improving physician compliance with guidelines [30] and reducing unintentional duplicate orders [31]. Previous studies have shown that it takes physicians 8 to 10 s to completely read alerts. However, in our study, the most common alert dwell time was around one second, suggesting that our physicians had become victims of alert fatigue, or that they had learned to reflexively disregard messages of low or no clinical importance [32]. One indication of alert fatigue is when clinicians spend less time per alert, sometimes going as far as not reading the alert at all [33]. The most frequent alerts in our CPOE alert system were administrative, highlighting the need to optimize them preferentially.
Intuitively, the most frequent alerts have the advantage of clinician familiarity. Our study showed that more frequent alerts have lower alert dwell times, which may be suggestive of the phenomenon of alert fatigue existing in relation to our hospital's current CPOE system. Previous studies have indicated that alert relevance (i.e., accuracy of automation) had no impact on alert dwell time [21]. Interestingly, we found that there was also no association between alert message length and alert dwell time. We hypothesize that this was because physicians did not read the alert message completely no matter how long the content. Since physicians frequently received clinically irrelevant alerts, they probably learned to override them expeditiously to optimize clinical workflow.
Our results also showed that the alert dwell times among different professional groups remained consistent. One possible contributor to this observation is that clinical processes affecting different professional groups in the hospital are analogous to one another. Another potential explanation is that the different professional groups receive alerts with a similar distribution of clinical importance. By identifying alerts with low dwell times owing to negligible clinical importance, dwell times can provide additional support for changing or even removing them [34,35].
Numerous studies of physician behavior have employed a wide array of analytical methods, including questionnaires, interviews, and observations [36][37][38]. The eventual goal of these studies is to understand the features of alerts that contribute to alert fatigue, and thus help to optimize the overall clinical workflow. The dominant window detector that we developed for this study not only retrieves the alert dwell times, but also analyzes the time spent by clinicians in different program windows during the consultation. This opens up the opportunity to analyze various elements of the clinical workflow, such as the time spent on different programs, and the identification of the most time-consuming software. Furthermore, by visualizing how computers are being used during various digitized clinical processes, any bottlenecks in the computerized elements of clinical workflow potentially become easier to discover and improve. By using a dominant window detector like ours in studies of clinician behavior on CPOE systems, the medical prescribing process may be ameliorated by decreasing alert fatigue, with potentially beneficial consequences for patient safety [39][40][41].

Limitations
This study has several limitations. First, although the dominant window detector is convenient to implement, additional data combination procedures were needed to calculate the alert dwell times. An automated, AI-based alert classification method will be applied in the next version of the dominant window detector. Second, the present study only showed the impact of fundamental alert characteristics on the alert dwell time; more variables such as medical specialties, individual variables (age, gender, clinical experience), categories, and clinical importance of alerts should be explored. Third, we focused on the alert dwell time, although alert acceptance is also an efficiency indicator of CPOE alert system performance and could not be recorded through our current program. Lastly, although we have collected and analyzed a large number of alerts, our data were collected from a single academic medical center in Taiwan. This may limit the generalizability of our findings. To address this, datasets of alerts collected in other hospitals should be used for external validation.

Conclusions
We have successfully developed a novel method to retrieve the alert dwell times from a homegrown CPOE system, and demonstrated its ability to help analyze the impact of several alert characteristics on alert dwell time, including categories of alert content and alert message length. Additionally, our study suggested that alert fatigue might occur for higher-frequency alerts with lower alert dwell times, and that the alert dwell times remained consistent across different clinical professional groups. Our dominant window detector can potentially expand the functionality of the hospital's homegrown CPOE system, providing important insights for system designers and hospital administrators looking to optimize clinical workflow.
Supplementary Materials: The following are available online at www.mdpi.com/article/10.3390/app112412004/s1, Figure S1: Framework of the dominant window detector (HWND= Handle to the window), Table S1: The example of alert title and alert content in predefined pairs, Table S2: Normality test result, Table S3: The example of less frequent alert categories with longer alert dwell times, Table S4: The example of alert categories with highly clinically relevant. Informed Consent Statement: Not applicable.

Data Availability Statement:
The data that support the findings of this study are available from Wan Fang Hospital Institutional Data Access but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of Wan Fang Hospital Institutional Data Access. If readers are interested in the original dataset, please contact yianyulin@w.tmu.edu.tw (Y.-Y.L.).