The DDMUV is a tertiary referral centre that offers a long-term surveillance program for melanoma patients based on the European guideline on melanoma treatment [
2]. An example for the follow-up sub process in the European guideline on melanoma treatment modelled in the Business Process Modeling and Notation (BPMN) can be found in [
9]. The melanoma registry at the DDMUV contains data of baseline and follow-up visits of melanoma patients. Excisions are documented way back to the early 1990s, a continuous documentation of the follow-up visits started 2010. In 2017, the melanoma registry covered about 2200 patients. In this study we included all 1023 patients (43% females, mean age
years) with baseline visit (i.e., excisions) after January 2010 and at least one follow-up visit since patients without a single follow-up visit only had the excision at the DDMUV and no data is available in the melanoma registry. Besides the demographic data, different characteristics of the identified melanoma are documented. For the baseline visit this includes among others, (1) melanoma subtype (superficial spreading melanoma, nodular melanoma, lentigo maligna melanoma, acral lentiginous melanoma and others), (2) anatomic site (i.e., abdomen, hand, foot, head, etc.), (3) depth of invasion, (4) date of surgery for the primary tumor, and (5) staging information. More than one primary tumor can be documented. Only melanoma staging is used for conformance checking. We extracted five different event logs from our real world data, one including all patients (i.e., I-IV), and four for each AJCC stage separately (i.e., I, II, III, IV) based on the highest AJCC stage of the patient. If a patient initially started with AJCC stage I and then moved to AJCC stage II the patient is represented in the AJCC II log file.
Table 1 lists the number of patients and number of mean events per case in each log.
The number of patients per AJCC stage decreases with higher AJCC stage, which corresponds to the fact that most melanomas in Austria are diagnosed in early stages [
20]. Most patients (
) were in AJCC stage I. This group also had the highest number of patients lost to follow-up (
, 78%). The ratio of patients IN_FUP (i.e., in follow-up) was the highest in AJCC stage IV with 58% (
). There is no difference between proportion of individuals lost to follow-up (LTFU) between men and women. Men were generally older than women and there was no significant difference between the LTFU and IN_FUP in respect to the age. Patients in lower AJCC stages were generally younger (I: Mean age
years; II: Mean age
years; III: Mean age
years; IV: Mean age
years).
4.1. Conformance Checking of Melanoma Surveillance
The results of the conformance checking in the form of fitness and precision indicators can be seen in
Table 2. Our measurements show that the guideline models have an overall comparable and good fitness value, i.e., the model generally explains the behaviour seen in the log. This originates from three facts. (1) The relabelling and clustering of activities was done based on the terminology that was also used for the guideline model. (2) The time boxing method presented in
Section 3.1 leads to an ordered sequence of events, where loops and duplicates cannot occur. (3) The only
wrong events (i.e., events present only in the log, not in the model) are the LTFU events.
The precision of the model for stage I is 75.1% and declines to 63.1% for stage IV. The ratio between observed and possible behaviour indicates under-fitting for low values. The explanation for the generally lower precision values is that the guideline models include the whole time period of ten years of follow-up visits, while the event logs only cover a maximum of seven and a half years. Thus, modelled events like I_F_08_1Q (i.e., stage I, eighth year, first quarter) will never be reached during replay, leading to a lower precision. The explanation for the declining values of precision is that the guideline models for higher stages allow for all the lower stages’ events too, since a patient can start in stage I and be re-evaluated to stages II, III, or IV during their follow-up visits. The number of possible behaviour is thus higher while the number of actual patients in the stages is similar (II) or significantly lower (III and IV) than in stage I.
Figure 3 shows the most frequent trace recorded in the complete log. One hundred and forty-eight of the 1023 patients follow this trace where they (1) start with the excision (Start), (2) are staged in AJCC I (StageChange), (3) go to their first follow-up (I_F_00_3Q), and (4) are afterwards lost to follow-up (wrong event LTFU). The following missing event (IN_FUP) is in the guideline model but was not present for those traces in the log. Finally, the End event concludes the trace.
Figure 4 shows a patient that started in stage I and was re-evaluated to stage II and later to stages III and IV. All in all just 1 follow-up visit during stage II was missed and the fitness is very high. The trace spans over the whole observation period, with the start in 2010 and the last follow-up in late 2015. Thus, the patient was identified as in follow-up (IN_FUP).
In
Figure 5 the patient classified in stage II skipped multiple follow-up visits and left the monitoring entirely after four years. The low fitness value correlates with the low guideline compliance.
4.2. Applied Process Mining
In addition to conformance checking we applied several techniques and tools to the data at hand in an explorative manner, to try and find interesting trends in the data and to validate our results with domain experts.
Figure 6 shows the dotted chart analysis of the stage I event log. For stages II–IV the number of distinct event types becomes higher and additional stage changes speckle the diagram so the rainbow pattern becomes less noticeable.
The recording period of 7.5 years, from January 2010 to June 2017, can be seen on the y-axis. Observations made based on this dotted chart:
A clear ‘rainbow’ pattern is visible since the sequence of events recorded for stage I is linear and follow-up visits occur at generally regular intervals.
The outcome indicator (IN_FUP/LTFU event) is not visible since the End event occurs at the same timestamp +20 h and the latter dot overlaps the former one.
The most frequent trace (i.e., ending the follow-up after the first visit in stage I—see
Figure 3) can be observed due to the high number of End events that occur shortly after the Start and I_F_00_3Q events.
Figure 7 shows a part of the model depicting the flow of patients in stage I. It also includes the lost to follow-up step (LTFU) that marks an early dropout. Observations that can be made on this model section:
All 401 patients in stage I start with the ‘I_F_00_3Q’ event. One hundred and forty-eight patients drop out (i.e., LTFU) after this step. This corresponds to the 148 patients of the most frequent trace in
Figure 3.
The frequency of following events declines steadily. There are fewer patients in later steps than at the beginning. There are two reasons for that. (1) Not all patients started at the same time, thus not all patients can reach the 7th years’ follow-up visits in the fixed time interval. (2) After each step some of the patients drop out.
The sequence of follow-up events is not linear but makes a ‘braided’ impression due to the skipping of single follow-up visits (see also
Figure 5).
4.3. Medical Implications
In [
21] the prognosis among patients with thin melanomas depending on the surveillance compliance was analyzed. Patients were considered to be compliant with the follow-up regimen if they had at least one annual follow-up examination and non-compliant if they had follow-up intervals of more than one year. They showed that compliant patients before the onset of recurrence had a significantly better prognosis than non-compliant patients.
When using our calculated fitness instead of the fixed time-intervals to evaluate the survival, the same effect can be observed in our data as seen in
Figure 8. We sampled all 246 patients that stayed in follow-up for more than two years (a subset of
Table 1, Total, IN_FUP, I-IV) based on their fitness value into three equal-sized groups and used the Kaplan–Meier estimator for survival analysis. The survival probability of patients with a high guideline compliance after five years is about 5% higher compared to the least compliant group. However, adding the patients that stayed for less than 2 years to the estimator, looking at all 358 patients in follow-up (
Table 1, Total, IN_FUP, I-IV), showed a reversed effect. The main reason was that higher fitness is easier to achieve with a shorter stay and many with a short stay died early, e.g., after being staged in IV and the first follow-up visit.