Machine-Learning-Based COVID-19 and Dyspnoea Prediction Systems for the Emergency Department

Featured Application

Machine Learning-based diagnostic tool to predict SARS-CoV-2 positivity and the need of hospitalized patients for oxygen therapy when managing constrained resources in emergency departments in contingency periods.

Abstract

The COVID-19 pandemic highlighted an urgent need for reliable diagnostic tools to minimize viral spreading. It is mandatory to avoid cross-contamination between patients and detect COVID-19 positive individuals to cluster people by prognosis and manage the emergency department’s resources. Fondazione IRCCS Policlinico San Matteo Hospital’s Emergency Department (ED) of Pavia let us evaluate the exploitation of machine learning algorithms on a clinical dataset gathered from laboratory-confirmed rRT-PCR test patients, collected from 1 March to 30 June 2020. Physicians examined routine blood tests, clinical history, symptoms, arterial blood gas (ABG) analysis, and lung ultrasound quantitative examination. We developed two diagnostic tools for COVID-19 detection and oxygen therapy prediction, namely, the need for ventilation support due to lung involvement. We obtained promising classification results with F1 score levels meeting 92%, and we also engineered a user-friendly interface for healthcare providers during daily screening operations. This research proved machine learning models as a potential screening methodology during contingency times.

1. Introduction

On 11 March 2020, the World Health Organization (WHO) declared the SARS-CoV-2 (i.e., COVID-19) pandemic [1]. The virus causes bilateral multi-focal interstitial pneumonia, which can quickly evolve into acute respiratory distress syndrome (ARDS). Infected subjects present a heterogeneous and changing clinical picture, ranging from asymptomatic cases to either focal or multi-focal lung patterns, commonly referred to as the white lung pattern [2]. Symptoms include fever, dry cough, fatigue, and shortness of breath [3]. Epidemiological data report droplets spread with face-to-face exposure as the most probable cause of virus transmission. It is mandatory to live upon social distancing rules, rapidly identify and isolate infected subjects, and track the most recently infected people to prevent the virus from scattering [4]. Therefore, it is fundamental to promptly diagnose SARS-CoV-2 infection to appropriately treat the patients according to the highest healthcare standards and quickly operate contact tracing. The gold standards to diagnose the SARS-CoV-2 condition are the amplification of viral RNA by real-time reverse transcription-polymerase chain reaction (rRT-PCR), IgM–IgG coupled immunoglobulin and rapid antigenic analyses. However, all suffer from several drawbacks, thus having the need for expensive equipment, trained personnel, long processing times, and both preanalytical and analytical vulnerabilities, namely poor sensitivity and overall detection accuracy [5,6].

The need for quick and reliable tools to diagnose and stratify patients’ prognoses to optimize resource allocation is of the utmost importance. Splitting the emergency department (ED) into clean and dirty areas during a pandemic is fundamental to preventing patient-to-patient spreading.

During the last decade, the number of research articles on artificial intelligence (AI) as a resource for all kinds of medical specialties highly increased, proving machine learning (ML) algorithms to be successful [7]. Notably, several studies report medical instruments supported by AI concerning applications ranging from stroke management to cancer detection [8,9,10,11,12,13]. AI is reshaping how biomarkers and biosignals are defined and managed [8,9]. AI-enabled support systems might aid clinicians in decision making, especially during ED triage. Indeed, most studies aim to develop models to schedule patients according to their triage acuity level, assessing the severity of their conditions and deciding upon hospital admission [14]. SARS-CoV-2 studies comprise algorithms exploiting computed tomography (CT), lung ultrasound (LUS), and X-ray imaging techniques to diagnose and examine COVID-19 evolving patterns [2,15,16,17,18]. On the other hand, researchers exploited ML- and AI-based techniques to tackle contact tracing, the prediction and forecasting of epidemiological measurements, and SARS-CoV-2 drug development [19,20,21].

During the first heavy pandemic waves, several governments limited swab testing due to the unfeasible number of them to be carried out. Therefore, from that period onward, research has focused on machine learning algorithms to quickly assess patients for COVID-19 positivity and mortality [20,21,22,23,24,25]. The studies confirmed the feasibility of the statistical learning-based approach. Nonetheless, only 8% of the studies observed by literature reviews focused on blood test analyses [21]. Researchers produced statistical models having different goals ranging from COVID-19 to mortality prediction, with classification results having sensitivity levels of approximately 80–89% [21,26].

Fondazione IRCCS Policlinico San Matteo Hospital’s Emergency Department (ED) of Pavia let us evaluate the exploitation of machine learning algorithms on a clinical dataset gathered from laboratory-confirmed rRT-PCR test patients, collected from 1 March to 30 June 2020. The main goal was to quickly stratify patients and employ cross-contamination procedures, avoiding extensive swab testing and leveraging physicians’ workload. We gathered patients’ data based on two primary principles. First, we engaged features readily available in every ED triage, such as anamnesis, symptoms, and vital signs. Moreover, we collected data concerning patients’ respiratory failures, routine blood tests, arterial blood gas (ABG) analysis, and lung ultrasound quantitative evaluations [2,27]. We adopted support vector machine (SVMs) and random forest (RF) [28] algorithms to assess patients’ COVID-19 positivity. Furthermore, we predicted whether a patient would need oxygen therapy, such as continuous positive airway pressure (CPAP) or invasive ventilation. Indeed, we must organize and wisely engage limited resources during contingency times. The diagnostic tools allowed to meet promising results with F1 score levels at 92% and sensitivity—i.e., recall—ones at 96%.

We believe the novelty of our designed methodology, summarized in Figure 1, stands in the following lines:

• A careful clinical features collection: we based our classifiers on the features that physicians employed during triaging and daily clinical operations;
• Extensive and robust data analysis before ML clustering;
• Exploitation blood tests to assess patients, rather than imaging data;
• Assessment of patients’ need for oxygen therapy to carefully engage limited resources in contingency scenarios [21];
• A quantitative lung involvement examination to produce robust results: studies report lung ultrasound examination as a fast, cheap, and agile tool to assess patients’ lung involvement [2].

2. Materials and Methods

This section provides a detailed description of the data collection, cleaning processes and exploration. Furthermore, we describe the selection and the design of the machine learning methodologies to diagnose SARS-CoV-2 and predict the need for assisted ventilation.

2.1. Data Collection

Fondazione IRCCS Policlinico San Matteo’s Emergency Department of Pavia established a strict protocol during triage operations to diagnose patients who might have potentially been infected by SARS-CoV-2. The procedure was mandatory to stratify patients and avoid cross-contamination during hospital operations. To apply ML methodologies and aid physicians during the pandemic, we mainly looked at clinical characteristics available in any ED triage, such as history taking, symptoms, and vital signs. Furthermore, we gathered information that is not only associated with patients’ respiratory malfunctions, but also satisfies the constraint of being promptly available and cheap, such as routine blood tests, arterial blood gas examination (ABG), and lung ultrasound quantitative evaluation [2,27,29]. We collected data from patients complaining about potential SARS-CoV-2 symptoms, whom the physicians swabbed to diagnose the disease. The collection process lasted from March 1 to 30 June 2020. In the end, we acquired the list of features shown in Figure 2 for 1355 patients, where we report the correlation coefficient between each feature and the outcome to be predicted, namely both COVID-19 positivity assessment and oxygen therapy potential need.

Several studies reported the presence of typical SARS-CoV-2 pneumonia patterns during LUS examination. We collected quantitative information regarding lung injuries due to their usefulness in the diagnosis and the prognosis of patients affected by COVID-19 [2,27,29,30,31,32,33]. LUS presents advantages compared to other tools: a fast, non-ionizing, radiation-free, cheap, and bedside technique. Not only can we perform an ultrasound for unstable patients, but also it is easily repeatable. Therefore, we extensively exploited a strict acquisition protocol [34] and assigned a standardized LUS Score to get a quantitative analysis of patients’ lung health conditions. Hence, we studied each lung in six different areas with a high-frequency linear probe (5 MHz) and scored each of the lung portions [34]. Furthermore, we defined the modified LUS score as the sum of the scores assigned to each lung’s region, thus ranging from 0, namely when the lung is well aerated, to 36, describing a lung in which all the regions present a tissue-like consolidated pattern. Hence, as reported in Figure 2, we included the 12 scores associated with each lung’s region, ranging from 0 to 3, and the overall modified LUS score. Figure 2 reports that quantitative lung evaluation positively correlates with the estimation targets.

We performed data cleaning and pre-processing operations, comprising categorical features transformation into dummy variables, namely the conversion of textual features into discrete and numerical values and the handling of missing values. Values could have been missing due to a variety of reasons. The first reason is machinery malfunctioning: the device performing the test needed might not have written certain information well. Secondly, the physicians’ workload and pace. If a subject was visibly affected by SARS-CoV-2, and there was not enough time to assist other patients according to the highest healthcare standards and complete the data acquisition protocol, the personnel interrupted the process to respect the hard time constraints demanded by the pandemic. Unfortunately, there is no quick way to fix missing values.

Hence, we adjusted the dataset to 443 patients by excluding all the patients whose values were missing.

In conclusion, we split 90% of the data into the training set and the remaining 10% into the test set. Experts usually suggest these percentages to be 70% and 30% [35]. However, because our dataset size was insufficient to keep relevant information in the training set at the end of the data-cleaning process, we decided to opt for this kind of subdivision. Moreover, to meet reliable results free from overfitting problems, we performed a 10-fold cross validation. At each learning step, we randomly split the training set into 10 subsets such that the 10th one was used to validate our data, whilst the remaining four optimized the models’ weights. K-fold cross validation is a common practice used by data scientists facing dataset size problems [35].

2.2. Data Exploration

We conducted the study in the ED outline. It is usually highly recommended to exhibit the patients’ statistical characteristics of the dataset to explain what features a study stands upon and to compare the dataset to others [24,25]. Not only did we consider in

Table 1. Dataset statistics. Bp stands for blood pressure, pp means partial pressure, and A-a-g is the arterial–alveolar gradient.

	Negative (172 Patients)	Positive (271 Patients)	Positive and Negative (443 Patients)	[36]. 838 Negative. 786 Positive
	25–75 Percentile (Median)	25–75 Percentile (Median)	25–75 Percentile (Median)	25–75 Percentile (Median)
Age (years)	37.0–67.5 (54)	51.0–75.0 (63)	47.0–73.0 (60)	48.0–77.0 (62)
Systolic bp (mmHg)	125.0–150.0 (135)	115.5–144.0 (130)	120.0–145.0 (130)
Diastolic bp (mmHg)	70.0–90.0 (80)	70.0–85.8 (80)	70.0–90.0 (80)
Mean bp (mmHg)	91.6–106.7 (100)	88.3–104.3 (95)	90.0–105.0 (96.7)
Respiratory rate	16.0–22.0 (20)	16.0–26.0 (20)	16.0–24.0 (20)
Oxygen saturation (%)	94.0–98.0 (97)	90.0–97.0 (94)	91.0–98.0 (95)
Body temperature (°C)	36.2–37.6 (36.7)	36.5–38.0 (37.1)	36.3–37.9 (37)
Hemoglobin (g/dL)	12.2–14.9 (13.5)	12.8–14.9 (13.9)	12.6–14.9 (13.7)	11.8–14.6 (13.4)
White blood cell (109/L)	6.3–11.5 (8.2)	4.8–8.1 (6.3)	5.2–9.2 (6.9)	5.7–10.7 (7.8)
Neutrophils (109/L)	3.7–9.2 (5.6)	3.2–6.6 (4.6)	3.4–7.1 (4.9)	3.7–8.1 (5.3)
Lymphocytes (109/L)	0.9–2.2 (1.6)	0.6–1.1 (0.8)	0.7–1.6 (1)	0.8–1.7 (1.2)
Platelets (109/L)	179.5–272.5 (224.5)	146.0–239.0 (184)	157.0–256.8 (204)	171.0–282.0 (222.3)
C-reactive protein (mg/dL)	0.1–10.5 (1.3)	2.6–15.2 (7.9)	0.9–14.4 (5.3)	0.6–10.2 (3.9)
Lactate dehydrogenase (U/L)	182.0–290.0 (222)	243.5–428.0 (326)	211.3–399.8 (286)	218.4–388.8 (278)
Creatine phosphokinase (U/L)	51.0–143.0 (86)	68.0–293.5 (113)	62.0–217.8 (99)	53.0–162.5 (86)
PH	7.4–7.5 (7.4)	7.4–7.5 (7.5)	7.4–7.5 (7.4)
O2 pp (mmHg)	69.5–102.0 (84.8)	58.7–82.2 (68.7)	62.6–92.8 (74.1)
CO2 pp (mmHg)	31.4–38.6 (35.7)	29.6–35.8 (33)	30.3–37.0 (33.9)
PaO2/FiO2	317.6–462.9 (392.2)	226.4–352.7 (299.5)	256.1–405.8 (323.8)
A-a-g of O2 (mmHg)	9.6–42.6 (22.5)	33.6–93.1 (47.3)	20.8–60.9 (40.4)
LUS Score	0.0–7.5 (2)	6.0–16.0 (11)	2.0–13.0 (7)

the median value of the features in Figure 2, but we also divided the statistics for the people who resulted as either positive or negative from the RT-PCR swab. Furthermore, we also dealt with dyspneic patients. Namely, we predicted if the considered subjects would need aided ventilation, either with CPAP or something more invasive. However, since the dyspnoea we look at comes from SARS-CoV-2, there is no need to stratify the displayed statistics further. In Table 1, we reported some statistical characteristics of one of the datasets used by our colleagues to highlight the similarities, except for the percentage of positive patients, 61.2% in our case vs. 48.4% in that of Cabitza et al. [36]. Concerning oxygen therapy, only 20.8% of subjects needed CPAP or invasive ventilation, whereas 79.2% of patients required either an oxygen mask, nasal cannula, or no oxygen therapy. We evaluated the correlation coefficients between each input feature and either target to examine the elements and the machine learning models. The data exploration procedure, reported in Figure 3, helped us to extract the clinical picture of people affected by COVID-19. Indeed, we can cluster SARS-CoV-2 positive subjects by age and comorbidities, particularly hypertension, diabetes mellitus and cardiovascular diseases. They clinically present themselves with fever, dry cough, and dyspnea, along with an increase in respiratory rate and a reduction in hemoglobin and oxygen saturation. Concerning the blood analyses, we report COVID-19 positivity related to lymphopenias, an increase in white blood cells, neutrophils, lactate dehydrogenase, cardiac troponin, and C-reactive protein. Furthermore, ABG test results present alterations, such as an elevated oxygen alveolar–arterial gradient (A-a) and reduced pO2, pCO2 and P/F ratio [27].

We further validated such clustering by examining the scatter plot matrix of the features, studied both by coronavirus positivity and patients who needed aiding ventilation. For simplicity, we only expose the selected patterns in Figure 3. Overall, we can see that none of the scatter plots show a clear separation between patients affected by COVID-19 and those with other diseases, but only a smooth transition. Nonetheless, there are some recognizable patterns. COVID-19 patients with severe conditions are older than the ones presenting healthier patterns, have higher LUS scores and have lower P/F ratios. We observed the same but a more pronounced pattern comparing patients who needed CPAP or invasive ventilation with those who did not. To decide whether we should consider all the features to cluster the patients for both classification scenarios, we performed principal component analysis (PCA) [37]. We computed the number of input values needed to maintain the 95% statistical variance in our dataset. We report that to keep 95% of information while reducing the number of input features, we should keep 48 principal components instead of 58 (Figure 4). The reduction was not significant enough to justify a further level of complexity. Therefore, the models described in the next section do not perform any feature selection process prior to prediction. Additionally, physicians asked for a fast answer. Hence, we kept all the input features without a further pre-processing step beside the classical ones, which comprise feature rescaling and cleaning.

2.3. Machine Learning Models

We chose random forest (RF) and support vector machines (SVMs) [38] to advance the state-of-the-art results [23,24,25]. The exploration process we reported in the previous sections encouraged this choice.

Random forest is an ensemble learning methodology to classify by designing a cluster of decision trees throughout the training. The hyperparameters describing an RF algorithm are the number of estimators composing the forest, the tree’s maximum depth, the highest number of levels we let each tree reach, and the estimator’s minimum number of data points placed in a node before splitting it. Moreover, we usually tune the maximum number of features to be considered for splitting a node and the minimum number of data points allowed in a leaf. Finally, we also choose whether to bootstrap our data; namely, we can choose to either resample data points or not. Data scientists usually recommend exploiting bootstrap when the dataset size is small. In

Table 2. RF and SVM hyperparameters for each classification task.

Model	Hyperparameter	COVID-19 Prediction	Dyspnea Prediction
RF	n_estimators	550	500
	max_depth	2	2
	min_samples_split	1	1
	max_features	50	None
	min_samples_leaf	Auto	Auto
	bootstrap	True	True
SVM	C	1	1
	γ	0.01	0.01
	kernel	RBF	Sigmoid

, we present such parameters as we find them in the Python Scikit-Learn library [39], respectively: n_estimators, max-depth, min_samples_split, max_features, min_samples_leaf and bootstrap.

SVM is also a supervised learning methodology, representing one of the most robust prediction algorithms. An SVM acts by representing the data points in a p-dimensional feature space and dividing them such that there exists a hyperplane clustering the points. The gap between the hyperplane and the data points belonging to each class must be the widest. There are fewer hyperparameters compared to the RF algorithm. First, we need to determine the kernel function, the nonlinear function enabling us to fit the maximum-margin hyperplane, mapping the data points in the p-dimensional space. Data points cannot be linearly separable. Therefore, we transform them through a kernel function acting as a degree of closeness measure, mapping the data into another feature space. One of the most used functions is the radial basis function (RBF), featuring the hyperparameter γ. The hyperparameter γ controls the distance of influence of each training point. The lower its value, the higher the similarity radius, resulting in more points grouped. Similarly, it exists also for other kernels. Moreover, we choose the C hyperparameter, a penalty we assign for each misclassified data point. The smaller C is, the lower the error penalty. Typically, we look for γ ⊂ [1 × 10⁻⁴, 10] and for C ⊂ [1 × 10⁻¹, 100].

Both models exploit hyperparameters to classify the data—Table 2. We adopt hyperparameter tuning procedures to boost our classification performance, evaluated in terms of ROC-AUC, accuracy, precision, recall and F1-score. The first one is called grid search cross validation. We list the values of the hyperparameters which we would like to test our models with, and we evaluate every combination. At the end of the process, we choose the values attaining the best classification performance on the K-fold cross-validation. The second one is called random search cross-validation. The process is like grid search. However, we choose the hyperparameters utilizing a heuristic search over random values. We present the best-identified hyperparameters chosen after either of these methodologies in Table 2. The hyperparameter tuning processes rely upon pseudo-random number generation, such as selecting the data points belonging to the training and test sets or the K-fold cross validation. We set the random seed on 19 for all the experiments to make the experiments reproducible and to look at the improvements derived from tuning the hyperparameters. The test systems used to conduct our experiments, both for the training phase and the hyperparameters tuning, are equipped with an Intel-i9-9900X CPU, working at 3.5 GHz, 128 GB of RAM and two 2944-cores NVIDIA RTX 2080 GPU. We wrote the code to attain our classification goals using Python and the latest version of the Scikit-Learn library.

3. Results and Discussion

Several studies and reviews highlighted the importance and the role of AI-based medical instruments to aid physicians and engage limited resources [12,21,22]. Researchers are producing novel methodologies to define biomarkers and process signals, reshaping how we address clinical tools [9,10,20,21,22]. Here, we propose two diagnostic tools for COVID-19 detection and oxygen therapy prediction, namely, the need for ventilation support due to lung involvement. The models chosen for the two classification tasks steadily seized convergence throughout optimization concerning the hyperparameters displayed in Table 2. We assessed the gathered results by evaluating accuracy, precision, sensitivity—i.e., recall—and the F1-score [28]. The first metric informs the reader about how good we are at diagnosing the absence of SARS-CoV-2, whilst the latter is the degree of accuracy we have over an unbalanced dataset, measured employing precision and recall. Indeed, the consequences of incorrectly diagnosing a patient as healthy are the inappropriate lack of treatments and cross-contamination among subjects presenting other pathologies.

Considering both clustering scenarios, we found area under the curve levels (Figure 5 and Figure 6) exceeding 93%. At the same time, we measured 96% recall when considering COVID-19 detection. We found an overall F1-score of 92% for the first task and 83% for the second one, and we also measured precision continuously above 80% (

Table 3. Test set classification results.

Classification Task	Model	AUC	Accuracy	Precision	Recall	F1 Score
COVID-19	RF	93.0%	91.0%	89.0%	96.0%	92.0%
	SVM	95.0%	91.0%	89.0%	96.0%	92.0%
Oxygen Therapy	RF	96.0%	91.0%	83.0%	83.0%	83.0%
	SVM	93.0%	87.0%	80.0%	67.0%	73.0%

). These results are particularly worthy of notice when compared to the rRT-PCR test. Indeed, the nasopharyngeal swab attains 73.3% sensitivity (95% CI 68.1–78.0%).

Furthermore, currently, no metric exists to determine if the ED will need other resources [40]. Therefore, we provided a valuable tool to wisely manage the hospital’s limited resources by predicting whether the considered patient will need CPAP or invasive ventilation. Physicians usually make decisions by looking at the patients’ continuous vital signs, whilst we predict future needs concerning a particular time, namely when the patient arrives at the ED. This process implies that the clinical picture we consider could change abruptly at any moment. Nonetheless, we measured an 83% F1-score and ROC-AUC values above 90%, reported in Figure 5.

Comparing our classification performance with our colleagues, which is reported in

Table 4. State-of-the-art classification results from [23,26,36,41].

	Models	Features	Patients	AUC.	Accuracy	Specificity	Recall	F1 Score
Cabitza et al.	Knn, Naïve Bayes, RF, SVM, Logistic Regression	72	1624	76.0%	78.0%	82.0%	74.0%	-
Goodman-Meza et al.	Multilayer Perceptron, Naïve Bayes, RF, SVM, Logistic Regression, XGBoost, ADABoost	12	1455	91.0%	-	64.0%	93.0%	-
Plante et al.	XGBoost	15	192,779	91.0%	-	42.0%	96.0%	-

[23,24,26,41], we not only managed to improve their results while considering a more significant number of features, but also handled a smaller and particularly unbalanced dataset. Indeed, concerning COVID-19 detection, we reached 96% of recall, whilst others could exceed 90% only using a three-way model [36]. Namely, a model abstains from prediction when the confidence score is below 75%. Other researchers reached 95.9% sensitivity with 41.7% specificity [41]. Their model represents a valuable screening tool to rule out COVID-19 infection. However, such a low specificity can be dangerous in facing an infectious disease. Indeed, it is more important to identify positive patients and isolate them than to rule out negative ones.

On the one hand, we collected a dataset whose features represent the daily clinical scenario that the physicians are given. Namely, we inserted the quantitative LUS examination together with blood and ABG tests. We believe that this process allowed the robust classification performance obtained. On the other hand, our dataset is smaller than that of our colleagues [23,26,36,41], although we managed to handle both its size and class imbalances, and the LUS examination requires trained personnel to be performed.

Finally, we also designed a graphical user interface (GUI) shown in Figure 7, targeting assistance to the medical personnel during contingency times at the emergency department. The GUI presents five sections for each data group: anamnesis, vital signs, blood gas analyses (BGA), blood tests and LUS score. Completing them with patient’s data, we obtain the probability of being a COVID-19-positive patient according to our classification model.

4. Conclusions

Concerning the data collected from the routine hospital operations between 1 March and 30 June 2020, our research proved the feasibility of developing reliable algorithms to diagnose SARS-CoV-2 with high classification performance. Furthermore, in addition to what our colleagues had already studied, we demonstrated how to estimate dangerous dyspneic scenarios. Namely, whether the patients need CPAP or invasive aided ventilation, this prediction is noteworthy to manage resources in contingency times. On the grounds of a close and stable collaboration with IRCCS Policlinico San Matteo’s Emergency Department of Pavia, we based the study on highly reliable clinical data to develop two artificially intelligent systems, one of which is equipped with a GUI and was tested by the medical personnel as a supporting decision-making device in a real-world clinical scenario.