Evaluating a Recurrent Neural Network Model for Predicting Readmission to Cardiovascular ICUs Based on Clinical Time Series Data ^†

Abstract

Unexpected readmission to intensive care units (ICUs) endangers patients’ lives due to premature patient transfers or prolonged stays at the care units. This can be mitigated by stratification of the readmission risk at discharge times using state-of-the-art machine learning (ML) methods. We fitted two alternative recurrent neural network (RNN) models based on long short-term memory (LSTM) on the Medical Information Mart for Intensive Care (MIMIC-III) dataset and evaluated them with an independent cohort from our hospital’s ICU (UKD). The first model processed all the available time series data from each patient’s ICU stay, whereas the second model focused on the data from the last 48 hours of the ICU stay prior to transfer. Our readmission prediction on MIMIC data reached an area under the curve of receiver operating characteristic (AUC-ROC) of 0.82. Furthermore, the model with the 48 h time frame outperformed the other model, as both models were applied to the independent test cohort. The results suggest that the RNN model for time series forecasting holds promise for future use as a clinical decision support tool, although follow-up studies with larger cohorts as well as user studies should be conducted to assess the generalizability and usability of the methods, respectively.

1. Introduction

According to different retrospective and review studies [1,2,3], the rates of readmission to ICUs in hospitals in developed countries are quantified inconsistently in the range from 0.14 to 14.5 percent. Regardless of the inconsistent readmission rates, patients who are readmitted to ICUs due to inappropriate discharge time are subject to life-threatening risks, with respiratory and cardiac complications as the most common causes of readmission [1,4]. At the same time, prolonged stays at ICUs can also lead to increased mortality or poor long-term prognosis for patients [5]. In particular, patients who undergo cardiac surgery are prone to longer ICU stays due to the invasive nature of heart surgery and the resulting increased risk of postoperative complications [6]. Therefore, it is important to find the optimal point in time for transfer, for this particular vulnerable group of patients. Intensivists need to quantify patients’ readmission risks in order to determine this point in time and avoid unplanned readmissions. This implies an increasing need for clinical decision support tools which complement current discharge routines through the use of predictive models taking advantage of artificial intelligence (AI).

AI in general and ML in particular have been widely used in many diagnostic and prognostic domains, such as oncology [7,8] and computational neuroscience [9,10], to provide clinical decision support tools, which either replace or complement established diagnostic and prognostic routines. More specifically, time series analysis has been successfully applied in many medical domains, including management of type 2 diabetes [11], hospital admission prediction [12], age-related death prediction [13] and ICU readmission prediction [3,14].

From a technical point of view, as related works suggest, a variety of techniques ranging from simple probabilistic predictors such as logistic regression (LR) and ML-based classifiers such as support vector machines (SVMs) to more sophisticated models such as convolutional neural networks (CNNs) have shown their potential for the prediction of readmission to ICUs [3,15,16].

In a previous study [17], currently under review, we trained a model with a recurrent architecture on a subset of the MIMIC-III dataset, an open access dataset of time series data from more than 50,000 care unit stays with different health conditions at a single center [18], without any limitation on the time frames of the input time series data, and showed its superiority to logistic regression and a feed forward network in predicting readmission. In the current study, we analyze the performance of a recurrent neural network (RNN) model processing health-related times series data, taking advantage of the long short-term memory (LSTM) method with slight architectural enhancements compared to the previous model, considering time series data from the last 48 h of patients’ stays at ICUs, sampled in 60-minute intervals.

The main contribution of our work is to propose and evaluate two alternative ML-based models, which are trained and fit using a publicly available dataset of health-related time series data, with and without concrete time windows (in compliance with state-of-the-art metrics in the domain) prior to patients’ discharge from ICUs. Moreover, as a further contribution, the proposed models were evaluated using a real-world hospital cohort with data from our own cardiovascular intensive care unit at UKD to assess how they would generalize against new sets of unseen data.

In the next sections, first, a detailed overview of the patient data and methods is given, and then the results of model training and evaluations are presented and discussed. Finally, the relevance of the findings and possible future follow-ups will be discussed.

2. Materials and Methods

This section explains the methodology of the study. First, the details of the dataset curation for training, validation and test steps will be elaborated, including the selection criteria and cohort statistics, which will be followed by the description of the integrated times series data. Then, the details of the training and validation step using 5-fold cross-validation on the MIMIC-III dataset are discussed. Finally, the details of the test step on an unseen set of data from a held-out MIMIC cohort as well as the UKD database are presented. The entire in-house-developed analysis pipeline is developed in the Python (V3.6.) programming language. The LSTM models for time series data analysis are implemented using the PyTorch library [19].

The inclusion criteria of subjects from both MIMIC and UKD datasets have been defined as follows:

• Patients who were transferred from the ICU to a normal station and then returned to the ICU within 48 h. Due to possible logistical reasons, patients whose first ICU stay was less than 24 h are excluded. These patients are labeled as ’readmitted’.
• Patients who died during the hospital stay. The dead subjects are also labeled as ’readmitted’.
• Patients who were transferred from the ICU to a normal station and then were not returned to the ICU within 30 days. Those who did not return within 30 days after transfer from ICU are labeled as ’non-readmitted’.

2.1. Patient Cohorts

For the training, we focused on the cases of patients from the MIMIC-III dataset who visited cardiovascular care units. As a result, the training subject cohort consisted of 11,513 patients, out of which 966 patients were labeled as readmitted and 10,547 were labeled as non-readmitted. From the MIMIC dataset, the train subjects’ ages ranged from 17 to 89 years (mean(M) = 66, standard deviation(SD) = 13.51). For anonymization purposes, the real admission times are not attainable from the MIMIC dataset.

For test purposes, a cohort of 502 patients who visited the cardiovascular ICU at the UKD were retrospectively analyzed. Among these, 100 patients were returned to the ICU within the 48 h after discharge (labeled as ‘readmitted’) and 402 patients visited the ICU just once, without being readmitted within 30 days after their discharge (labeled as ‘non-readmitted’). The ages of the subjects from the UKD database were quantified in the range from 19 to 95 years (M = 67, SD = 12.25). The cohort corresponds to cardiovascular ICU stays in the time period from January 2017 to December 2020. Due to the retrospective character of the study, patient consent was waived.

2.2. Time Series and Patient Data

For each subject in the train or test cohort, 10 vital or laboratory variables (selected by highly qualified cardiovascular surgeons) were captured during the patient’s stay at the corresponding intensive care unit. To form the time series data for all the subjects, 60 min time intervals have been taken into account. The missing values were extrapolated using the mean value of the corresponding variable throughout the entire corresponding cohort. Moreover, from the patients’ files, the age and weight values were extracted. Finally, the length of stay (LoS) of each subject was quantified from the admission time entries from the corresponding database (MIMIC or UKD) and added to the rest of the features to end up with feature vectors of size 13.

Table 1. The input parameters including time series data from laboratory or vital values and information from patient files (ABP: ambulatory blood pressure, LoS: length of stay). The table is adapted from [17].

Lab Values	Vital Signs	Patient Information
Creatinine	Body Temperature	Age
Blood PH	ABP	Weight
Sodium	Heart Rate	LoS
Potassium	Oxygenation
Hematocrit
Bilirubin

provides the list of the 10 variables used for the time series data analysis and the 3 extra patient variables.

2.3. Preprocessing

Preprocessing of the input data is applied, similarly to our previous work [17], consisting of the following steps: First, the time series features and patient variables are obtained from the corresponding dataset (MIMIC or UKD). Then, all values are standardized and missing data are replaced with the corresponding distribution’s mean value. In the last step, the data are both re-sampled and oversampled before being processed by the corresponding model. Figure 1 gives an overview of the preprocessing pipeline.

In the feature extraction step, the case IDs of the subjects who met our selection criteria have been processed to acquire the feature values from the corresponding database using appropriate queries. Then, the normalization is applied at two different levels. First, the features which might have been stored in different units (such as weight) have been unified and converted accordingly and cases with extreme and outlier values have been removed from the cohorts. In the second step, we applied standardization around the mean value to end up with normalized feature vectors. The standardization of each variable is applied as defined in Equation (1):

$$\widehat{s_i}=\frac{s_i-{\mu }_x}{{\sigma }_x},$$

(1)

where $\widehat{s_i}$ is the standardized value, $s_i$ is the original value, ${\mu }_x$ is the mean of the variable over the whole cohort x and ${\sigma }_x$ is the standard deviation over x.

As required by the LSTM method, all the input feature vectors must be of equal size before being analyzed. Thus, for all the timestamps without an actual measurement, the missing value was filled in with zero (which is the mean value after standardization). Moreover, as the vital and lab values of patients are usually measured in fragmented timestamps, the feature vectors extracted from the databases were non-uniformly distributed along subject cohorts. Thus, to unify the entire cohorts, the time series parameters are resampled to a frequency of one entry per hour. In case there exist more than one measurement per hour, the mean of each hour is used as the value. Alternatively, if there is no record for a time step, the mean over the whole cohort is used. Furthermore, the imbalance in the label distribution (there are more non-readmitted cases than readmitted ones in both cohorts) might result in the training of a deep learning model that is not able to predict the minority class labels accurately [20]. Therefore, the data for training the LSTM models were oversampled in order to have evenly distributed classes.

2.4. Model Training

In a previous study [17], we have shown the superiority of an RNN model based on LSTM architecture to logistic regression and a feed forward network using the MIMIC-III dataset. Therefore, in the current study, two alternative models with LSTM architecture have been analyzed, compared and further evaluated on unseen sets of data from open access and available in-house data. Both models are trained and validated using a subset of the MIMIC-III dataset, which was selected randomly. The first model, which will be denoted as the ’cropped’ model from now on, applies a fixed window of the last 48 h to each ICU stay. The second model, which will be denoted as ’uncropped’ in the next parts of the text, applies no limits on the available time series data from each subject.

As a special kind of recurrent neural network (RNN), long short-term memory (LSTM) is able to learn long-term dependencies in time series data. The input to an LSTM is a batch of arrays containing time series data. Although the LSTM model can handle inputs with varying lengths, the input length has to be equal inside each batch. Thus, to pad each time series with zeros until they have equal length, a PyTorch object is used.

To be able to capture non-linear relations between input variables and target labels, a hidden layer with the size of 50 neurons is applied after the LSTM pipeline. The input to the models are the time series data. For the cropped model, a fixed window of 48 h is used, while, for the uncropped model, no constraint on the length of time series data is applied. After processing the input time series batches, the three patient variables (age, weight and LoS) are concatenated to the feature vectors. Then, a rectified linear unit (ReLU) is applied as the activation function, while a dropout layer with a value of $0.3$ is used for regularization purposes. The final fully connected layer consists of 2 neurons, which are then passed through to the softmax function, which quantifies the final class probabilities (one for each label).

Figure 2 illustrates the architecture of the LSTM-based models. The $x_i$ input refers to the i-th time step of all time series.

Table 2. The hyperparameters of the LSTM-based model. The table is adapted from [17].

Hyperparameters	Values
optimization algorithm	Adam
learning rate	0.003
loss function	cross-entropy
batch size	32
epochs	early stopping

summarizes the hyperparameters for model fitting.

2.5. Model Validation

The training cohort from the MIMIC dataset is first split into separate train and validation cohorts with similar ratios of target labels. Then, to validate the model internally, 5-fold cross-validation is applied with random folds. As a result, the hyperparameters of the best performing models are stored to be used for the corresponding test steps. The training of each model continued until the area under the curve of precision recall (AUC-PR) stopped increasing after several epochs. Subsequently, the model that achieved the highest AUC-PR in the cross-validation was applied to the separate validation set. This procedure was repeated 10 times and the mean and standard deviation of the performance metrics were computed accordingly.

2.6. Model Evaluation

Both of the trained models are evaluated against unseen cohorts from the MIMIC-III and UKD datasets. To this end, two independent test cohorts have acted as representatives of unseen data. To quantify the model performance on each of the held-out test cohorts, each of the cropped and uncropped models, which were trained and fit in the validation step, are applied to the independent test cohorts. Then, the performance of the models is quantified and compared in terms of balanced accuracy, AUC-PR, precision, recall and AUC-ROC.

3. Results

In this section, the results of the model validation and evaluation are presented. First, the results obtained by different models in the training and validation phase will be elaborated. Then, the models’ performance on the test cohorts will be quantified and compared.

3.1. Model Training and Validation

The results of the model training with five-fold cross-validation on the MIMIC dataset are summarized in

Table 3. Evaluation metrics for the validation set from MIMIC database.

	Balanced Accuracy	Recall	Precision	AUC-PR	AUC-ROC
cropped to 48 h	0.799	0.694	0.389	0.666	0.877
uncropped	0.778	0.649	0.389	0.636	0.859

. Both cropped and uncropped models performed well on held-out validation subsets within the training cohort, with AUC-ROCs of 0.877 and 0.859, respectively.

3.2. Model Evaluation

The results of the evaluation steps on held-out cohorts from the MIMIC and UKD databases are illustrated in

Table 4. Evaluation metrics for the test sets from UKD and MIMIC databases.

UKD
	Balanced Accuracy	Recall	Precision	AUC-PR	AUC-ROC
cropped to 48 h	0.514	0.06	0.318	0.223	0.554
uncropped	0.498	0.22	0.196	0.221	0.517
MIMIC
	Balanced Accuracy	Recall	Precision	AUC-PR	AUC-ROC
cropped to 48 h	0.74	0.575	0.35	0.538	0.796
uncropped	0.766	0.628	0.477	0.571	0.828

and Figure 3 and Figure 4, comparing the models’ performance together and to an unskilled classifier. In general, we observe a remarkable gap in the performance of the models as applied to the two groups of held-out data. Both the cropped and uncrpopped models perform reasonably well on the held-out set from MIMIC, with AUC-ROCs of 0.796 and 0.828, respectively. From the test results of the UKD data, the cropped model (AUC-ROC = 0.554) performed better than the uncropped model (AUC-ROC = 0.517).

4. Discussion

Predicting the risk of readmission to intensive care units at discharge times through the use of artificial intelligence can support healthcare systems to manage their resources in a more efficient way. As unplanned readmissions might result in longer stays as well as health hazards for patients [1,4], it is critical to provide quantification of the readmission risks as an additional asset for decision making for the intensivists. Related works suggest that different vital and laboratory values of patients sampled in the last 48 h prior to the patient’s discharge from the intensive care unit contribute the most to the fact of whether the patient will return to the care unit [21]. Thus, state-of-the-art time series forecasting methods on health-related data from ICU stays could be of great importance for the analysis of readmission risk for patients. Therefore, the motivation behind this study has been to propose and evaluate two alternative models, based on time series forecasting, for readmission prediction for patients visiting cardiovascular ICUs after heart surgery.

The first model analyzes all available vital and lab values for all the participating subjects during their stays at the corresponding ICU. Alternatively, the second model takes into account only the values in the last 48 h prior to discharge from the care unit. As elaborated in the results, both of the alternate models performed reasonably well in predicting patients who would be readmitted to the ICU when they were applied to an unseen cohort from the MIMIC dataset. Furthermore, we showed that the model with the 48 h time limit outperformed the other model in predicting readmissions for a cohort of unseen cases from UKD, which conforms with the findings from related work [21]. The superiority of the model with a persistent number of timestamps can be justified, as, typically, data-driven ML-based methods perform more efficiently on data cohorts with higher consistency [22]. Moreover, it signifies the relevance of the patients’ clinical factors measured in the most recent timestamps prior to transfer for predicting readmission risks.

Overall, the findings reveal the predictive potential of the AI-based approach for health forecasting after evaluating its performance on unseen datasets. However, there are some drawbacks to the currently applied methodology, which should be mitigated in future follow-ups. As for any other time series model, the methodology towards health forecasting as described in this paper would also be affected by the issue of missing data. For this study, we replaced missing measurements with the corresponding mean value throughout the whole cohort. For the future follow-ups, we will consider alternative AI-based imputation methods such as bidirectional recurrent imputation for time series (BRITS) [23], which would most likely help to further improve the robustness of our methods. Furthermore, most of the state-of-the-art ML-based models with deep architectures are less interpretable due to their ’black box’ nature. Therefore, one of our future works will be to provide an ML-based approach towards ICU readmission prediction in an explainable manner, i.e., by following explainable AI (XAI) best practices. This will further help to give the medical experts higher levels of interpretability and confidence in AI analyses.

The results of the evaluation step reveal the inconsistency between the time series measurements from the two databases (MIMIC-III and UKD), also known as the dataset shift problem [24]. This might reflect different protocols and standards for patient care and discharge in the corresponding care units and should be diminished in the future using appropriate transfer learning techniques or by training new models on larger cohorts which better fit in protocols. Thus, we are gathering more annotated data to prepare a suitable cohort of ICU stays at UKD for further follow-ups. Nonetheless, even these relatively poor preliminary results on the data from UKD are slightly better than the performance of an unskilled classifier. These findings identify room for improvement in readmission prediction accuracy by the AI model. Although the AI performance might be considered far from perfection, even highly experienced intensivists will not predict ICU readmissions quite perfectly. Therefore, to further quantify how the proposed AI approach would compete with experienced domain experts, we intend to conduct experiments in which the performance of the AI model for ICU readmission prediction is compared to that of a group of experienced intensivists. Nevertheless, it should be noted that the proposed methods are designed as decision support tools for medical experts and therefore should not be used as an autonomous system for decision making.

Another important aspect of any clinical decision support tool is usability. In this regard, as part of the upcoming follow-ups, we intend to conduct proper empirical studies to assess how a provided AI assistant user interface would satisfy user needs in terms of experience and performance. Moreover, to account for better generalizability, the integration of datasets from other hospitals and intensive care units would be a useful next step to take.

5. Conclusions

The aim of the study has been to train and evaluate a deep model with a recurrent architecture to predict the readmission of patients who visited cardiovascular ICUs based on times series data of vital and lab values. To this end, the RNN model based on the LSTM method was trained with open access data and tested using independent hospital data. Our findings revealed the potential of the proposed AI-based methodology to assist domain experts at cardiovascular intensive care units for the better quantification of readmission risks at discharge times. To further address the usability and generalizability aspects of the methods, empirical user studies and experiments with data from other care units should be conducted.