Process Mining Dashboard in Operating Rooms: Analysis of Staff Expectations with Analytic Hierarchy Process

The widespread adoption of real-time location systems is boosting the development of software applications to track persons and assets in hospitals. Among the vast amount of applications, real-time location systems in operating rooms have the advantage of grounding advanced data analysis techniques to improve surgical processes, such as process mining. However, such applications still find entrance barriers in the clinical context. In this paper, we aim to evaluate the preferred features of a process mining-based dashboard deployed in the operating rooms of a hospital equipped with a real-time location system. The dashboard allows to discover and enhance flows of patients based on the location data of patients undergoing an intervention. Analytic hierarchy process was applied to quantify the prioritization of the dashboard features (filtering data, enhancement, node selection, statistics, etc.), distinguishing the priorities that each of the different roles in the operating room service assigned to each feature. The staff in the operating rooms (n = 10) was classified into three groups: Technical, clinical, and managerial staff according to their responsibilities. Results showed different weights for the features in the process mining dashboard for each group, suggesting that a flexible process mining dashboard is needed to boost its potential in the management of clinical interventions in operating rooms. This paper is an extension of a communication presented in the Process-Oriented Data Science for Health Workshop in the Business Process Management Conference 2018.


Introduction
Operating rooms (ORs) are an essential and central element in modern hospitals [1]. Cost estimations of ORs are around 16% and 20% of total hospital budget, due to the amount and heterogeneity of human resources involved and the high technology used [2,3]. Efficiency in the management and interventions in ORs aims to reduce the non-occupation time of the surgical blocks and the way of organizing the different types of interventions to optimize medical teams (surgeons and nurses). To achieve a good efficient rate in ORs, it is crucial to have a protocol and a highly skilled staff to take over daily decision making for scheduled and unscheduled interventions [3].

Related Work
Real-time location systems (RTLS) monitor the position of a moving element with a given sampling frequency. In a hospital, moving elements are equipped with an active or passive element (tag), which is identified when it is nearby a beacon. In our study, patients were the moving elements who wore a wristband with the tag before entering the operating room service.
Lean principles present a condensed primer of RTLS in a health care environment [15]. Throughput is a key performance indicator for a patient pathway across a facility, which, linked to RTLS, could provide valuable information, such as waiting times and resource utilization [16].
RTLS systems in 23 hospitals in the US were analyzed from a qualitative perspective in Reference [17]. In this work, researchers observed the systems in use and conducted 80 semistructured interviews with hospital personnel and vendors. Authors found asset tracking the best feature and identified several obstacles related to the technical set-up and organizational context. According to the authors, RTLS must go beyond the simple deployment of technology to be effective.
Specifically, an OR service consists of several spaces, each of them equipped with a beacon to identify a patient whenever he/she goes to that specific area ( Figure 1). The regular flow of a patient starts in the preparation area, continues in the pre-surgery room or in the operating room, and ends in the recovery area. The application of process mining techniques in combination with RTL systems provided an easy-to-use and unobtrusive way to achieve a view of the deployed process and assess the time that patients, professionals, and assets are in each stage or location. In this paper, we analyze the web-based dashboard to perform process mining discovery and enhancement analytics, which has already been done by Fernandez-Llatas et al. [11] (Figure 2).
Analyzing RTLS data from a discovery perspective and enhancing these work flows with information related to the average time and overload, it is possible to create pre-programmed contingency plans for the management and allocation of resources of the operating rooms. However, why are these promising applications still not widely used in the clinical context? Instead of using a semistructured interview, we used the analytic hierarchy process [18] to quantify the features of a dashboard for discovering and enhancing processes based on RTLS data in an OR service.

Materials and Methods
It is necessary to identify the characteristics of the dashboards and relate them with the preferences and how and why those could impact clinical use.
The classic managerial structure of a hospital consists of a compartmental service-oriented organization [19]. This structure does not consider the flow of patients between services and how the decisions in one service could affect the following from the perspective of a process. Process-based management is focused on an organizational understanding of the medical and administrative processes in a hospital by collecting and analyzing collaborative information from professionals and their organizational units.
The dashboard analyzed in this study provides functionalities to aggregate data from an RTLS in an OR and analyze the flow of patients across the process of a surgical intervention. The dashboard allows to discover and enhance the patients' trajectories and the time they are at each stage and extract insights from processes. The information obtained from the dashboard is based on trustworthy data automatically collected by the RTLS and, thus, the information extracted depicts the actual performance and deployment of the processes in the OR. RTLS data are enriched with metadata of the clinical intervention (e.g., type of surgery). Our aim is to discover which of the characteristics of the dashboard are perceived as useful by OR staff to help them with their daily clinical and administrative work. In this section, we describe the characteristics of the process mining dashboard (Section 3.1), the methodology applied to perform the feature prioritization by the OR staff (Section 3.2), the questionnaires and tools used to do the feature prioritization (Sections 3.3 and 3.4), and the characteristics of the participants in the study, composed of the full team of ORs in Hospital General of Valencia (Section 3.5).

Analyzing Operating Rooms Processes with PALIA
PALIA consists of a web-based dashboard, which allows to perform process mining analysis on a given dataset. The software ( Figure 3) is composed of three major areas: Filters, for the selection of the data; mining, for the configuration of the visualization of work flows; and information, to show the information about the operation tool and the selected tracks. These three major areas are divided into five functional areas: There is also a list with all the samples selected with the filters (1-2). 3. Information (5), which shows details about how the process mining algorithm was applied and also features of the selected samples: Information on the number of merged branches, a log on how the process mining algorithm analyzed the events and infers work-flows, errors on data selection, and statistics on the transitions and states in the work-flow.
PALIA works in the following way: First, a comma-separated value file containing the RTLS data is loaded using a default file dialog window. Then, the user is able to filter the data and apply a discovery topological algorithm. The inferred work-flow appears in the screen with nodes and arrows, which represent the track followed by the patients across the surgical process (Preparation-Surgery-Recovery-Intensive care unit-Locker room-Adaptation).

Analytic Hierarchy Process to Determine Priorities
AHP is a methodology for decision-making which aims at solving complex problems [20]. It allows to quantify opinions and transforms them into a coherent decision model. The process is derived from a pairwise comparison using a numerical scale. AHP has found its widest applications in multifactor decision-making, planning, and resource allocation, and in conflict resolution [21,22]. AHP is a method which incorporates benefits and risks explicitly by combining the importance of differences in probabilities of outcomes related to alternatives and the weighting of the importance of those outcomes [23]. Unlike other methods for feature selection, such as conjoint analysis [24], AHP is specially concerned with the consistency of choices and their measurement and dependencies between the groups of elements [25]. Some key and basic steps of AHP were introduced by Pecchia et al. [18]: Broaden the objectives of the problem or consider all actors, objectives, and its outcome.

•
Identify the criteria that influence the behavior.

•
Structure the problem in a hierarchy of different levels constituting the goal, criteria, subcriteria and alternatives.
Our aim was to develop a hierarchy of elements grouped into categories to describe the functionalities of the PM tool used to manage OR processes. These categories were ranked using questionnaires to extract the relative importance of each need per category (local weights, LW), the relative importance of each category (category weights, CW), and the importance of each need compared to all the others (global weights, GW) [26].

Applying AHP to PALIA
AHP was applied to this study because of its inherent capability of prioritizing characteristics in a qualitative and quantitative way. The application of AHP involves six factors [21]: The hospital was motivated to adopt PALIA to support OR process management and was committed to implementing the decision and involved staff from the OR department. Stakeholders were active participants in the entire decision process from development to implementation. In order to identify the elements and the categories of the hierarchy, we used the different functionalities of PALIA. The hierarchy is composed of four levels, which have a 1:n relationship with the three functional areas described in Section 3.1. The first hierarchy level is generic and only describes the functional area (filters, miner, and log). The second hierarchy level describes the main features of the functional area (for filters, it contains dates, times, duration, type of intervention, type of OR, etc.). The third hierarchy level contains details of the main features within the functional area (in filters, for the type of intervention, it contains details of the medical service, the type of program, the surgical process, the surgeon in chief, etc.). The fourth and final hierarchy level contains low granularity details. To create the hierarchy tree and collect the answers, we used the web application BPMSG AHP On line System (https://bpmsg.com/academic/ ahp.php). In this study, we focused on two hierarchy priority levels: 1. Hierarchy Level 1, functional Area: Filter, miner, and information. 2. Hierarchy Level 2, features of the functional level (Tables 1-3). To select a specific range of dates (e.g., a day, a week, a month).

L1.3 Duration Filter
Allows to filter traces that have a specific minimum and maximum duration (e.g., ≥ 60 min)

L1.4 Level Filter
The real-time location system (RTLS) aggregates operating rooms (ORs) into different levels. There are levels that contain the small areas and levels that group different areas, generating more global location zones, for example, on one level, we can find "Operating Room 1", "Operating Room 2", etc. and on another level, these areas are grouped within the "Operating Rooms Level 0". This filter allows users to define the granularity of the area for the process mining analysis. More than one level can be selected at the same time.

L1.5 Name of the nodes Filter
This filter shows the names of the nodes or areas within the location of the system (e.g., Figure 1). This filter allows to indicate which areas or nodes the user wants to include in the mining (by removing the checkbox mark the node is excluded from the data corpus). This filter also allows to rename the area of location, being able to assign a new name to an area or even group two areas, giving them the same name (node aggregation).

L1.6 Features Filter
This filter allows to select and display only the data that comply with all the characteristics that are selected (e.g., type of surgical procedure, surgeon). Such data are typically available as metadata of the RTLS.

L1.7 Stretch Filter
This filter allows to select samples/traces which go through a given node or which go from one given source node to a given target node. It is also possible to limit the number nodes in the trace (e.g., samples with more than 3 nodes).

L1.8 Dis-gregation Filter
This filter allows to divide the corpus of data into several corpus, grouping the samples by their percentage of similarity. Samples not matching the similarity index are grouped into an outliers pool.

L1.9 Statistics Filter
This filters allows to display the percentage of samples of the flows that meet a given characteristic. In addition, these percentages can be grouped if they are less than a given value.

L3.3 Evolution of process Information
A log of actions executed when the inference engine starts (e.g., selecting data, filtering dates, filtering times, grouping samples).

L3.4 Error Messages Information
Error messages while the inference engine is being executed (e.g., the configuration of filters entails the in-existence of samples).

L3.5 Extra information
Information Extra information about the entire process (e.g., console messages, warnings).

Qualitative Study
In addition to quantitative prioritization and selection of the features in the dashboard with AHP, the study included an interview using a usability questionnaire implemented on the basis of the system usability scale (SUS) methodology. SUS is a balanced questionnaire comprising 10 alternating positive and negative statements [27], which is robust against extreme response bias [28]. To gain more insights about the dashboard usability, we adapted SUS to include seven closed items (to be rated in a 5-point Likert scale) and three open questions. The questionnaire is described in Table 4 .  Table 5 shows the profiles of the involved participants and their specific role into the OR service. The project identification code is 812386, under European Union's Horizon 2020 research and innovation program. Only one of the initial 11 participants (complete OR staff) was unable to fill the AHP questionnaire successfully and was discarded for the analysis. The respondents, who signed the informed consent of this study, were employers of the Hospital General de Valencia, one of the four hospitals of reference at the city, covering a population of 350,000 inhabitants. It has 27 operating rooms and in 2014, it registered 26,497 surgeries [29].

Participants
The study was organized in a session lasting 3 h. First, the dashboard was introduced to the study participants in the Hospital General premises. The first part consisted of a walkthrough of the application, in which we analyzed a simple dataset of one-day RTLS data with two types of surgical interventions. The second part consisted of analyzing a complete dataset containing three months of RTLS data with multiple surgical interventions. At this point, participants were familiar with the tool and were ready to prioritize features. The third and final part consisted of the fulfillment of questionnaires for the quantitative and qualitative study.

Results
A total of 10 questionnaires were collected after the session held in the University General Hospital of Valencia with professionals working in the OR service. For the analysis of the responses, participants were grouped into three categories depending of their roles within the OR service. The roles were defined according to the type of tasks and responsibilities each professional had in the OR service, so each participant was assigned to a unique category. The category of managers was composed of professionals with a high-level organization profile, who are responsible for the general performance of the service, human resource management, and reporting to the board of the hospital. The category of clinical staff was composed of three general surgeons and three nurses. The Technical category was composed of engineers who are responsible of the information technology systems in the ORs.
The AHP questionnaire allows to assign priorities for each of the features contained in the defined hierarchy levels, each of which correspond to a particular functional area of the PALIA web-tool.
The overall analysis of priorities shows a 57% level of consensus (relative homogeneity β = 77%), in which the miner component achieves the higher priority rates for the features of visualization (heat maps) and views selection (maximum occupation and current occupation). Collecting the relative priorities within each of the three main categories, we could extract the importance that each group of users assign to each of the PALIA functional areas. Figure 4 depicts the relative weights (%) assigned to each of the modules. The manager and clinical user groups showed a similar consensus on the prioritization of functional areas, whereas the group composed of the technical staff provided more priority to the mining component. The stack in the right part of Figure 4 is a weighted mean of the relative priority assigned by participants, in which we can see that the mining component is still the most important, and the two other components (filters and information) share a similar, lower importance.  Regarding the filters ( Figure 5), we can see a similar consensus between the managers and the clinicians, whereas the technical staff is weighting two features which were not that relevant for the former groups: Date and level selection. The answers for the filter component achieved a 65.7% group consensus. Regarding the miner (Figure 6), which was the most weighted component overall, we can see that clinicians were more interested in knowing the occupation of the rooms flow, but with respect to the frequency, there are similar priorities between them and managers. This figure also shows that only the technical staff is interested in having the list of samples which comprised the work-flow. The answers for the miner component achieved a 78.5% group consensus. Regarding information (Figure 7), we can see that the priorities assigned by the clinicians are the same for each feature, which could indicate a strong consensus or that the features are not relevant to this group. For the other two groups, there are two different features which received a significant different prioritization. Managers are more likely to have information about how the process evolved over time, and technical staff is more prone to having information about the sample cleaning (which, moreover, has a similar weight to other related features, such as the wrong selection of samples and extra information).The answers for the Information component achieved an 80.8% group consensus.   Table 4). Sixty-seven percent of the users would like to use the dashboard on a daily basis to organize and supervise ORs, and the same percentage think they could use the dashboard without the support of a technical person. Between 44% and 33% of the respondents indicated a strong agreement to the questions related to simplicity, consistency, and intuitiveness. Only two questions had a 10% of strong disagreement and 22% of strong agreement, and these were related to the ease of use and integration of the functionalities. Regarding to the open-ended questions, respondents highlighted the need to be able to identify patients through their number of clinical history instead of by tags ID, as it is now (from the RTLS). In addition, several of them also highlighted the potential usefulness of being able to introduce contextual factors and information related to incidences or intervention cancellations. One manager also pointed out the suitability of this framework in the management of bed occupancy.

Discussion
In this paper, we evaluated a dashboard for executing process mining analytics involving end users working in the OR service of a reference hospital. The dashboard used data from RTLS to discover, enhance and assess work flows of patients. The evaluation was two-fold, using first the AHP for prioritizing features in the dashboard and, subsequently, a SUS-based questionnaire.
AHP questionnaires allowed us to extract valuable and quantifiable information about the use of a dashboard for the exploitation of process mining on RTLS samples in the ORs of a hospital. The sample size of the questionnaire could be considered small (n = 10) to provide statistically significant findings. Nevertheless, this population contains all professional staff working in the operating room services in the hospital, and our results should be considered as a starting point to perform large-scale evaluations.
The assessment of PALIA features allow to enhance the communication with the clinical environment to create a powerful and usable tool for the application of process discovery on RTLS data. The distinction between the different groups according to their roles allowed us to analyze how to assign priorities to each of the stages of the application of process discovery. The group of clinicians always shows a high variability, which can be explained because they have fewer management tasks and their opinions vary more depending of the specific work they do [30]. Increasing the sample size of the study with new OR services from other hospitals would allow us to identify interesting preferences on clinical staff subcategories, such as surgeons, anesthesiologists, and nurses.
Another relevant finding is how the group of managers gives a high priority to the feature of comparing the process evolution over time. This finding is aligned with a previous study by Van der Aals et al., in which the feasibility of process mining tools (ProM) was analyzed in industrial contexts [31]. Technical staff assigns prioritization to the management of timestamps (date filters and occupation frequency) and information of the data cleaning process.
The quantitative prioritization of features allowed us to know the priorities each role assigned to the dashboard features. This information is of utmost importance for improving the application to provide end-users with specific tools to perform the type of analytics they can benefit the most from on a daily basis. New features (not assessed with AHP questionnaire) were also assessed in a semistructured discussion with experts. This questionnaire was based on the de facto SUS standard tool, in which we evaluated the suitability of the dashboard to fit in the daily routine of OR workers. The questionnaire was adapted, turning three closed-item questions into open-response. Respondents used these questions to indicate the possibility of creating a specific application to perform specific tasks (data filtering, process discovery, process enhancement, etc.), integrating the clinical identification of the patients instead of using the RTLS tags and using the dashboard in other contexts.
Based on the results of the quantitative and qualitative study of the features, in order to shorten the learning curve of such tools, we propose to prepare a specific dashboard interface for clinical and managerial staff, which contains only the top-ranked functionalities they have identified: Occupation maps and evolution of the processes. The remaining needed functionalities, such as data loading and filters, could be automated by the technical staff of the ORs, who were the category who assigned a higher mark to these characteristics.
Future work will focus on using the dashboard to perform advanced tasks, for instance, how errors and high noise problems (e.g., patient safety and medication issues) can be resolved using process mining as one possible application in an operating room environment.

Conclusions
The analysis of a process mining dashboard with quantitative and qualitative methodologies, such as AHP and SUS, allowed us to identify different preferences with respect to the functionalities and the role each professional has in the operating room. Despite the reduced size of the sample, our study included the complete staff of the operating rooms services of a reference hospital. Therefore, our results must be used with caution, as other findings may emerge when applying the same materials and methods in another hospital. This paper has demonstrated the ability of these techniques to identify unmet needs and user preferences when using tools for the application of process mining.