Process Mining in Clinical Practice: Model Evaluations in the Central Venous Catheter Installation Training

Abstract

An acknowledgment of feedback is extremely helpful in medical training, as it may improve student skill development and provide accurate, unbiased feedback. Data are generated by hundreds of complicated and variable processes within healthcare including treatments, lab results, and internal logistics. Additionally, it is crucial to analyze medical training data to improve operational processes and eliminate bottlenecks. Therefore, the use of process mining (PM) along with conformance checking allows healthcare trainees to gain knowledge about instructor training. Researchers find it challenging to analyze the conformance between observations from event logs and predictions from models with artifacts from the training process. To address this conformance check, we modeled student activities and performance patterns in the training of Central Venous Catheter (CVC) installation. This work aims to provide medical trainees with activities with easy and interpretable outcomes. The two independent techniques for mining process models were fuzzy (i.e., for visualizing major activities) and inductive (i.e., for conformance checking at low threshold noise levels). A set of 20 discrete activity traces was used to validate conformance checks. Results show that 97.8% of the fitness of the model and the movement of the model occurred among the nine activities.

1. Introduction

Medical practices have extraordinary importance for the procedural skills of medical students, health caretakers, maternity specialists, and paramedics [1]. To keep up proficiency in certain procedural skills that may be inconsistently utilized, remote health specialists should standardize practices. Any research-related practical skills can highlight the operational repetition, and reputation of clinical practice of medication, nursing, birthing assistance, and paramedic practice [2]. By undertaking this task, we gathered information on the doctor’s opinions about how often and by what means they expected to practice these abilities, with an emphasis on the use of reproduction as an educational strategy.

Most health disciplines teach a wide range of practical skills and clinical experience leads to further refinement, mastery, and dominance of these skills [3]. The execution of processes reasonably and competently is essential for providing better hospital services. As part of healthcare processes, individuals are involved in diagnosing, treating, and preventing diseases [4]. Evidence-Based Medicine (EBM) is a process of analyzing hospital processes based on evidence [5]. The right facilities with high-quality lab skills create a powerful and safe learning environment for medical students. The ability to gain skills, knowledge, and practices through additional feedback in the laboratory has moderate effects on patient outcomes compared to standard clinical training [6]. Consequently, medical trainees can perform technical and communication skills faster and more precisely on patients.

Simulated-Based Medical Education (SBME) studied feedback efficiency and found that terminal feedback was more effective than concurrent feedback [7]. During the questioning period following re-enactments, studies have shown that significant learning occurs [8,9]. In older days, feedback was primarily based on tutor observations and outcome evolution. With the advent of e-learning, telemedicine, video recordings, and virtual patient monitoring, opportunities for database feedback have increased [10]. Surgical training emphasizes the importance of following the process sequentially and spotting deviations in execution, rework, and transition times [11].

Medical process management experts rely heavily on process mining (PM) techniques for the evaluation of process models. It consists of three approaches, namely process discovery, checking, and enhancement [12]. Information about process execution is recorded in event logs and is accessible through existing data frameworks. Medical trainees benefit greatly from process mining during surgical operations since the structure is process-oriented. With the help of a case study of Central Venous Catheter (CVC) installation, for surgical training performances, students had to identify deviations from the model and only one student only achieved 100% conformance [13].

The process model from both students’ and instructor perspectives was generated to identify and improve the medical training. This is reported by authors from [14], who calculated the average of overall process performance to support the instructor during teaching [14]. In [15], the authors that generated the actual CCC19 dataset have validated both qualitatively and quantitatively from students’ perspectives in favor of their learning strategies [15]. However, these works do not clarify the deviation happening in the training activities. To do that, this work presents the structural view of activities performed during CVC installation by process mining techniques. The three research objectives (RO) of the presented work are:

RO1:

Display major training activities in CVC installation with Pertinet modeling;

RO2:

Present a novel framework with fuzzy mining analysis;

RO3:

Perform a conformance check to understand the activity deviation with prescribed CVC training.

In the remainder of the paper, Section 2 provides the literature review of healthcare process and applied knowledge; Section 3 provides a brief description of the training dataset and study framework; Section 4 describes experimental outcomes, and Section 5 summarizes the results. In the final Section 6, we present the conclusions to our work.

2. Research Background

Healthcare can be made more affordable, efficient, and effective with innovations. The advancement of technologies and business models has contributed to the advancement of healthcare [16]. Furthermore, healthcare systems in different parts of the world face unprecedented challenges, such as constant and rapid changes in clinical processes in response to new scientific knowledge, and the provision of high-quality care with limited resources. In order to provide quality hospital services, processes must be executed properly and efficiently. A healthcare process is a series of activities designed to diagnose, treat, and prevent diseases to improve the health of patients [16,17]. Clinical and nonclinical activities support these processes, which are carried out by qualified personnel (physicians, nurses, technicians, dentists, clerks), and can vary between organizations. There is no doubt that health processes are highly dynamic, complex, ad hoc, and increasingly multidisciplinary, which makes them an interesting area for analysis and improvement [18]. The quality of life of patients can be greatly enhanced by improving healthcare processes. The task of improving them is never easy and there are always a number of challenges involved. In order to meet the demand for services, reduce patient waiting times, increase resource productivity, and increase transparency, it is always necessary to reduce costs and increase capabilities.

Similarly, computer researchers and practitioners are becoming increasingly interested in the Internet of Things (IoT), a technology that has become more popular in recent years [19]. The IoT has transformed aspects of traditional healthcare paradigms because of its multidisciplinary approach [20]. Likewise, traditional health care systems can no longer satisfy the demands of a society that is constantly growing and developing. In order to efficiently and productively deliver healthcare applications and services, IoT is considered a bridging platform that connects the physical world and cyberspace. However, more research efforts are needed on the system, network, and architecture of IoT for efficient large-scale deployment to close the gap between fulfilling quality of service requirements and ensuring cost-effective implementations and operations of IoT [21,22]. With IoT technologies, data are collected automatically, in real time, during the execution of the process, resulting in massive amounts of sensor data. By using a model-based or AI-based approach to generate medical training events, the information relevant to the process can be extracted. A process-mining technique is then applied to analyze and identify deviations or optimization opportunities within the operational processes. Action recommendations are then provided to the student. Automatic action planning based on objective data is then accomplished.

A number of case studies have used PM in the healthcare domain with promising results [18,23]. In PM, the information extraction from data generated and stored in corporate information systems to analyze how processes are performed. In order for CGs to spread and be adopted, Medical Informatics will be crucial. Unfortunately, the pioneering approaches from years ago, often fragmented across a wide range of disciplines, did not lead to solutions that are sometimes exploited in hospitals today. It is reported that EBM has transformed clinical guidelines into an indispensable tool for physicians to use every day in their clinical practices [24].

In process-oriented medical education (POME) approaches, research in medical training and practice is considered as a sequence of steps [25]. This will create a description of the stages and steps provided. In this work, we provided a concentrated discussion on the basic concepts in medical training methods with PM knowledge. As mentioned, the research related to PM in evidence-based medicine related to clinical guidelines is minimal.

To overcome that, the CVC installation approach is taken as an example to develop the process modeling along with its applications in PM. Among the key issues related to the application of PM in healthcare, it is possible to identify certain key attributes. In general, control flow is the primary perspective from which process mining is applied. In this work, techniques or algorithms that are most commonly used, namely trace clustering, fuzzy mining, and heuristic mining, were applied because they are able to handle noise and incompleteness. These provide models for less-structured processes, which are common in healthcare. They provide the ability to group similar cases together, as is possible with trace clustering.

3. Methods

The majority of the process mining data is derived from event logs. In an event log, each event represents a single activity, also known as a collection of activities. How these logs were created and a dataset visualization is described in this section. Data preparation, model discovery, conformance check, performance evaluation (monitoring), and performance enhancement are the five steps shown in Figure 1.

3.1. CCC19 Dataset

Dataset CCC19 was derived from the interdisciplinary research project Process-Oriented Medical Education (POME) of the School of Medicine and the School of Engineering at the Pontificia Universidad Católica de Chile (PUC) [15]. There are three types of logs included in the challenge: XES, CSV, and XLSX. Medical students at PUC are trained in four stages to install CVCs that are mentioned below [13,15]:

• Learning: Students have to learn the initial procedure;
• PRE recording: Students have to perform the first pre-test (PRE);
• Practice: Students need to practice without barriers to get a good idea of the procedure;
• POST recording: Finally, they need to conduct the final post-test (POST) to make sure of the acquisition of enough knowledge.

Event logs generated from the video tagged dataset are comma-separated value (CSV) files that contain a row for every tagged video event. Consequently, an event log will be generated with the following information: case ID, resource ID, round type (POST and PRE), event ID, activity ID, stage, and the start and end time of the activity. Contrary to more conventional process mining datasets, event logs are not generated autonomously, but rather from observations of process execution. As an alternative, users used an observer-based approach to monitor student recordings through the use of special software video tags. An example of video tagging performed during CVC installation can be seen in Figure 2. Event logs are created by tagging PRE and POST training execution videos, which include 20 cases for 10 different students (resources), and 29 different activities. These events occurred between 11 October 2018 and 20 January 2019. The timestamp is recorded as a date (MM: DD: YYYY) and time (HH: MM: SS).

3.2. Process Stages

In the CVC installation process, there are six stages (operation and patient preparation, ultrasound preparation, locating structures, inserting guidewire, and inserting catheter), and each stage involves 29 steps. Color coding of these activities is shown in Figure 3 during POST and PRE training.

To define a Business Process Model Notation (BPMN) model for the installation of CVC with ultrasound, a Delphi method was adopted [26] and BPMN 2.0 is considered the standard for process modeling [27]. In process automation, it is a graphical notation for describing business processes that provide a standard notation widely understood by all users, and each arrow represents a time interval. Events, activities, and gateways start and end with core graphical components. To define the CVC process, 13 medical experts from eight different institutions were recruited. They represented three different specialities, and a prior model of business process with sequential activities conducted at various stages is presented in this panel.

3.3. Model Preparation

The data analysis and model preparation were conducted by the ProM 6.10 framework that provides a wide range of process mining techniques. It is public domain software is free of charge and implemented in Java [28]. The model discovery was done with inductive mining techniques for model soundness [29]. Inductive mining is an algorithm based on Petri nets that assists in model visualization. A perspective classifier is used to categorize event logs when preparing log sheets. A threshold value is provided by the user to filter out low-frequency activities during activity filtration.

Fuzzy mining concepts are relatively new in-process discovery algorithms and they are the principal method for describing problems related to large numbers of unstructured activities. By using this method, we simplified the training activity model in an interdisciplinary way. Correlation metrics are provided by this algorithm to interact with a process model at a given level of abstraction. By clustering unnecessary activities, it leaves them out of other mining algorithms [30]. We used Inductive Miner-Infrequent (IMi) for the process discovery, and IMi considered noise filtering applied to paths as input parameters and created the process tree. A model aligns the traces (i.e., a group of event logs) to best reflect what is currently being run. Model alignment with executed event logs provides the necessary information for model enhancement.

3.4. Model Fitness Calculation

The fit of the model or replay fitness is determined by how accurately the activities from the event logs fit into the trace. The process model determines how events are aligned optimally. The model fitness can be calculated by replaying the log events in the model and analyzing the model mismatches [31]. The start activity of the model and transitions in the trace that belongs to the log events are replayed sequentially. Token numbers are created artificially during this process and they indicate that the process is incomplete. Assume that x is the trace number in the aggregated log of the student log video dataset. The replay fitness is calculated as follows:

$$f=\frac{1}{2} \left(1-\frac{{{\displaystyle \sum }}_{i=1}^{x}nimi}{{{\displaystyle \sum }}_{i=1}^{x}nici}\right)+ \frac{1}{2} \left(1-\frac{{{\displaystyle \sum }}_{i=1}^{x}niri}{{{\displaystyle \sum }}_{i=1}^{x}nipi}\right)$$

For each log trace i (1 ≤ i ≤ x), where n_i is process examples added to the present trace, m_i is the missing token number, c_i is the user token number, r_i is the missing token number, and p_i is produced token number while a log replay of the present trace.

3.5. Model Analysis

An integrated fuzzy mining algorithm is used to visualize processes in ProM. Process discovery, according to experts, is most useful when identifying activities (e.g., tracking whether students repeat similar activities during a CVC procedure) and evaluating student performance, including activity completion and transition time between students and experts [31].

Table 1. Activities and their percentages of occurrences in the model.

Activities	Occurrences (Absolute)	Occurrence (Relative)
Prepare implements	126	9.039%
Get in sterile clothes	74	5.308%
Ultrasound configuration	66	4.735%
Check wire in the long axis	62	4.448%
Advance catheter	60	4.304%
Puncture	60	4.304%
Blood return	58	4.161%
Drop probe	56	4.017%
Guidewire install	56	4.017%
Check wire in short axis	54	3.874%
Remove syringe	52	3.73%
Anatomic identification	50	3.587%
Position probe	50	3.587%
Remove trocar	48	3.443%
Hand washing	44	3.156%
Put sterile gel	44	3.156%
Remove guidewire	42	3.013%
Compression identification	42	3.013%
Widen pathway	42	3.013%
Drap puncture area	40	2.869%
Cover probe	40	2.869%
Clean puncture area	38	2.726%
Check flow and reflow	38	2.726%
Anaesthetize	38	2.726%
Gel in probe	38	2.726%
Check catheter position	34	2.439%
Wire in a good position	20	1.435%
Position patient	12	0.861%
Doppler identification	10	0.717%

shows different activities as well as their occurrence in the model and their relative percentages.

Additionally, conformance checking compares a model of a business process to the same event log process, and validates the model execution as it relates to the event logs, and vice versa. Student activities for medical training were completed by ten different students and recorded in 1394 event logs. One can identify potential mismatches between the actual model and these event logs by comparing them with a developed model that specifies requirements. Accordingly, a conformance check allows medical trainers or experts to detect, explain, and measure student performance.

4. Results

A process model is generated as part of the process discovery check. There are two ways to observe student performance, including the process discovery check and the conformance check.

4.1. Process Discovery with Fuzzy Modeling

Fuzzy mining presents process models as process graphs that limit the activity items displayed in an aggregated manner. We used the ‘ProM Mine for Fuzzy Model’ algorithm to mine the activity interaction process between ten student groups to investigate the coherence between operational activities. Use a node edge filter with a cut-off of 40% significance to display the highly correlated activities. Approximately 40% of significant behavior appears as bright square activities in Figure 4. The less significant activities but with high correlation are hidden clusters that are represented by dark blue octagonal nodes. All relevant activities are grouped into clusters with their aggregate significance displayed. By obscuring less important activities, fuzzy process models enable medical experts to focus on the common features of a process. In general, darker edges indicate more significant relationships.

4.2. Petri Net (P-Net) Modeling of Student Activities

Fuzzy mining has the main limitation of not being able to validate models; instead, it provides a quick graphical view of the main structures of a process. Model validation depends on fitting the given data into the model (soundness), as well as process discovery. When dealing with medical datasets, it is recommended to have models that have a generalization of observations and pattern recognition of low noise. As a result, we considered using an inductive miner to develop a process model in the form of Petri nets.

A threshold value of 20% was used to ensure low model noise, which is less than our previous work [32]. The P-net outcome model (circles and boxes present the places and transitions) generated with inductive mining is presented in Figure 5. Black boxes are anonymous transactions to ensure the soundness of the model, while white boxes represent operational activities. In the patient preparation stage, the P-net model began with three parallel activities (get sterile cloths in, prepare implants, wash hands). It means that students can typically choose all three activities at this stage since there is no priority among these activities, and then follow two other sequential events (clean and drap puncture area).

4.3. Conformance Check

Clinicians must understand which operational activities are performing accurately during the CVC installation in order to improve student performance. Thus, any mismatch between the developed process model and the expert panel model must be addressed. Twenty trace activities were checked for conformance with the model. Assuming most of the traces were replayed in the model without any alignment, the aggregated model fitness is equal to 97.8%. Figure 6 shows the model fitness graph for operations in the CVC installation.

Nine out of 29 activities are reported as model moves, and the remaining 22 activities are aligned accurately. A summary of activity-based conformance checking is shown in

Table 2. Activity-based conformance check.

Activities	Total Log (Absolute)	Accurately Fitting Logs	Model Moves (In All Traces)
Gel in probe	20	15	5
Check catheter position	20	16	4
Anaesthetize	20	17	3
Check wire in the long axis	30	28	2
Compression identification	20	18	2
Widen pathway	20	18	2
Check flow and reflow	20	18	2
Position probe	27	25	2
Anatomic identification	26	25	1

. There were 20 instances of gel in probe activity in the trace, 15 times this activity executed correctly, and five times there was a model move. This indicates that the gel did not appear five times in the trace of the probe event and did not follow this activity during the CVC installation. Anesthesia possesses three times model move compared to catheter position check, which possesses four times model move. A total of five activities such as check wire along the long axis, compression identification, widening the pathway, checking flow and reflow, and anatomic identification are then introduced with the two-time model move, and anatomic identification does not take part in a model move for more than one time. It is imperative that these activities are synchronous with the expert opinion model.

With Petri-net alignments, we can also visualize the average time delay between activities. Figure 7 shows that the transition from gel in the probe to wire in a suitable position takes a medical student the longest time—635 (3.53 h) minutes. Light-yellow boxes indicate the shorter delay, while dark red boxes indicate the longest. The next transition from probe position to implant preparation is to add 623 min (3.46 h) and gel to the probe to prepare implants in 550 min (3.05 h).

5. Discussion

In this paper, we adopted the PM knowledge in medical training activities in order to provide medical feedback in the CVC installation. This proves that PM collects, links, analyzes, and uses routinely collected data automatically. With its use, effective resource allocation will be enhanced through improved alignment between data analysis and decision-making processes.

Origin support of procedural skills may not be perfect in some cases, and the issue is evident in places with limited medical facilities [33]. As a result, training may not be permitted. A decrease in the competency levels and confidence of nurses or medical trainees leads to a reduced ability of medical doctors to perform procedural expertise when needed. By analyzing real data and by responding to questions related to performance and compliance, this technology provides fact-based data with objective results. A process model created from event logs can be used for several applications including predictive analytics, performance analysis, and compliance analysis.

Training activities and knowledge acquisition have been deemphasized in clinical practice in order to achieve learning outcomes and enable doctors to meet and patient healthcare needs [34]. The current state of process mining in healthcare has a major focus on extracting knowledge from the medical data generated and stored in hospital systems. This knowledge is used to analyze the executed process models. Thus, we have incorporated process mining knowledge into medical training programs for future doctors in the specialty of CVC installation with ultrasound [35]. By effectively identifying non-presented activities to complement the tailored feedback of surgical procedures, process mining has been demonstrated to provide process-oriented feedback to clinical practitioners that are learning procedural skills.

As part of our approach, we have incorporated both fuzzy and inductive mining algorithms. From two different algorithms and patterns, the process-oriented feedback given to a medical student who wants to become a professional surgeon explains the performance of desired behavior. Here, we present the preliminary analysis of operational activities with low conformance using the model presented in this study. There was both a parallel pattern and a sequential pattern of activities during the CVC installation process. It suggested that conclusions be drawn cautiously because overperforming similar activities can limit the model’s performance.

During the installation of the guidewire, the drop probe is an optional procedure, and other activities follow panel suggestions. When the model maintains a reasonably structured flow, we can take advantage of its capabilities. The BPMN model showed a sequential pattern of activities as the catheter was installed, whereas our P-net model had some sequential elements and others appeared as choice activities with no priority. Unlike the expert model, our developed inductive mining model mismatches in two instances. For this reason, we extend this to a conformance check in Section 4.3. In the report given by Geyer-Klingeberg et al., the insights of event logs and deviations between actual and expected process models and easy-to-use and meaningful dashboards from both student and instructive perspectives are provided [36]. However, this study did not highlight overall model moves observed with the given activities.

If we observe Figure 5, two simultaneous events in the patient positioning and the ultrasound configuration can be viewed. In addition, there is a mismatch between the student outcome model and the expert model in this situation. The process flow moves sequentially in preparation for an ultrasound. In the expert model, three activities in the locating structures stage appear simultaneously in parallel, a pattern that is equally evident in the student outcome model. Anesthesia must be administered during the venous puncture stage, and the XOR gateway can be observed on the expert model (the only possible path). However, in the student model, the activities with exclusive gateways also appear as parallel events, which indicates that the student can select from all of these options.

By fuzzy mining and inductive mining algorithms, the study proposes the right patterns of activities with a high conformance value. Future medical students should be able to increase their awareness of finishing these activities in less time by observing transition times for moving from one activity to another. This is especially true for activities called gel in the probe, wire in a proper position, positioning probe, and implant preparation (Figure 7). By performing these two algorithms, students can be able to avoid similar mistakes in the future. Additionally, it provides feedback regarding simple and independent activities that are deliberate.

Fuzzy model results show that catheter placement and catheter position checking are highly significant activities while a positioning probe is an activity with low significance. A student starts the process by either preparing implants or putting on sterile clothing. Before implant preparation, it is advised to wear sterile clothing. A few clusters with low-frequency activities were formed after the preparation of the implants. Clustering was triggered by an ultrasound positioning probe. The next step is to make the hole to insert the probe and then proceed with sequential activities such as blood return, probe dropping, removing the syringe, checking the wire on the long axis, and finally verifying the catheter position. Clusters of activities with low significance can also be observed between these two activities.

This work is in line with the research by [37], as they proposed Fuzzy Discrete Event System (Fuzzy-DEVS) specification models which are constructed using a similar model that applied in this work. However, the system entity structure method can be applied to the event logs to prepare and structure them so that a Fuzzy-DEVS model is automatically generated from the transition system [38]. These models produce to simulate different stages of heating, ventilating, and air conditioning (HVAC) installation and predict the behaviors without having to replicate new experiments.

6. Conclusions

In this study, we have successfully implemented the process of mining medical feedback on CVC surgical procedures. In contrast to previous studies on this challenge, we are looking at patterns associated with activity deviations by integrating fuzzy and inductive techniques. To do this, we used the ProM simulation tool, and the dataset was validated with a 20% noise threshold. As a result of the low noise level, all the activities involved in the process can be represented in the model. It is a novel approach that can be replicated in other training programs.

However, the current case study work with a CCC19 challenge provides the opportunity with the necessity for additional training for feedback improvement; this study has some limitations. Primarily, it is observed that some student activities are not in good order. At the same time, we can find a lot of waiting time between the tasks, which indicates that students are unaware of the follow-up activities to perform. Additionally, a low process instance rate can infect the supposition of outcomes for medical students overall. Having a limited number of tagged videos may limit the model outcomes compared to the student population. The same problem arises when medical training consists of complex patterns, as more advanced conformance steps must be applied to detect non-relevant activities.

It will be possible to obtain encouraging results in the future if similar methods are replicated in other approaches. A comparison of different process mining and simulation techniques must also be made to evaluate tailored feedback in medical training, as well as conventional data mining methods. There is so much more to this exciting use of process mining technology to help improve medical procedure training. Currently, we are exploring how advanced process mining techniques can be applied. We will be comparing execution data with a normative model of procedure requirements as part of the CCC19.