Developing and Validating Ensemble Classifiers for At-Home Sleep Apnea Screening ^†

Abstract

In this paper, we developed ensemble classifiers with SpO2 signals for sleep apnea screening. The ensemble classifiers (eclf) were built on top of five base classifiers, including logistic regression (LR), random forest (RF), support vector machine (SVM), linear discrimination analysis (LDA), and light gradient boosting machine (LGMB). Performance evaluation showed that when heavier weights were assigned to the LR and SVM classifiers, the ECLF achieved a better balance between sensitivity (0.81 ± 0.02) and specificity (0.80 ± 0.02) while maintaining the overall performance as measured by AUC (0.81 ± 0.01).

1. Introduction

Due to lifestyle changes, more and more people are suffering sleep disorders nowadays. Among all the different types of sleep disorders, sleep apnea is a most common one around the globe. Sleep apnea is characterized by brief stops of breathing during sleep. If that happens, on average, more than five times per hour throughout the night, then the person is clinically considered sleep apnea-positive. Sleep apnea is often accompanied by snoring during sleep and excessive daytime sleepiness. The long-term consequence of untreated sleep apnea includes increased risks of cardiovascular diseases, metabolic disorders, mental problems, and cognitive impairment [1]. Developing sleep apnea screening methods that are easy to use at home has been the objective of many studies. The main challenge of home sleep apnea screening is that most of the signal modalities in medical settings, such as EEG, ECG, and breath effort, are not readily measurable with home-use sensors. To address this challenge, several studies have attempted to develop apnea screening methods based solely on the blood oxygen levels, or SpO2, during sleep. This signal modality is ideal for home sleep apnea screening because it can be collected using consumer sleep tracking gadgets with reasonable accuracy [2]. This study aims to develop ensemble classifiers that takes the SpO2 signals as input to predict whether a user is sleep apnea-positive or -negative.

2. Related Work

Polysomnography (PSG) is the gold standard for sleep apnea diagnosis. A PSG test measures many physiological signals which allow doctors to calculate the apnea-hypopnea index (AHI), defined as the average number of obstructive respiratory events occurred per hour during sleep, for diagnosing sleep apnea. An AHI of larger than 5 events/h is considered out of the normal range and thus sleep apnea-positive. Other AHI cutoff thresholds, such as 15 and 30, are also widely used to determine the severity of sleep apnea [3]. The PSG test is expensive, invasive, and not always available. Many studies have attempted to develop alternative methods for sleep apnea screening using information stored in medical records. Instead of using physiological signals, those methods rely on information such as demographic characteristics, blood tests, and questionnaire scores to assess the likelihood of sleep apnea [4]. In a related vein, machine learning and deep learning models that use SpO2 signals for sleep apnea screening were also developed. While early works have used only small datasets for model development [5], a very recent publication trained and tested their models on several large datasets [6]. In this study, we developed and validated our models using one of the largest sleep datasets.

3. Methodology

In this section, we present the pipeline for developing the ensemble classifier for sleep apneas screening and the dataset used. Signal processing and machine learning were performed using Python 3.10.5. Several Python modules were used, including Numpy, Pandas, Matplotlib, Scikit-learn, and Scipy.

3.1. Dataset

The Sleep Heart Health Study (SHHS) dataset used in this study consists of multi-modal physiological signals collected from 5786 subjects during sleep [7]. Permission was granted from the National Sleep Research Resource (NSRR) to export the SpO2 signals as well as some demographic information (i.e., age, gender, BMI) from the dataset. Data usage in this study was compliant with the Data Assess and Use Agreement (DAUA). The SpO2 signals were preprocessed to remove readings of zeros, below 50% or above 100%. Sudden changes between consecutive readings (>4%) were also removed. The ‘nsrr_ahi_ph3r_aasm15′ variable was used as the ground truth values for AHI.

3.2. Ensemble Classifier

Ensemble is a method in machine learning that combines multiple base classifiers to improve the performance of the classifications. In this study, we used five base classifiers, including linear regression (LR), random forest (RF), support vector machine (SVM), linear discrimination analysis (LDA), and light gradient boosting machine (LGMB). The final output of an ensemble classifier is the most frequently occurring label (i.e., the mode) of the weighted outputs of the based classifiers. Let $f_{LR},f_{RF},f_{SVM},f_{LDA},f_{LGMB}$ be the 5 base classifiers, and the final output of the ensemble classifiers $f_{eclf}$ is calculated using Equation (1). We built two ensemble classifiers: $f_{eclf1}$ uses a weight vector of [1, 1, 1, 1, 1], while $f_{eclf2}$ uses a weight vector of [2, 1, 2, 1, 1]. We assigned heavier weights to the LR and SVM classifiers in the latter case because those two classifiers yielded better performance in our previous study [8].

$$f_{eclf}=\mathrm{mode}(w_{LR}·f_{LR},w_{RF}·f_{RF},w_{SVM}·f_{SVM},w_{LDA}·f_{LDA},w_{LGMB}·f_{LGMB}),$$

(1)

Following the common practice in machine learning, 80% of the dataset was used for model training and tuning, and the rest was used for model testing. Hyper-parameter tuning was achieved through 5-fold cross validation. Classifier performances were measured using AUC, sensitivity, specificity. We repeated the dataset split 50 times with different random seeds. Boxplots were generated to show the distribution of the performance measures over the 50 repetitions.

4. Result

The AUC, sensitivity, and specificity of the base classifiers and the ensemble classifiers are shown in Figure 1, Figure 2 and Figure 3. It is shown that among the five base classifiers, LR achieved the highest AUC and specificity. RF had the highest sensitivity but the lowest specificity, which translated to the lowest AUC among all base classifiers. Compared to ECLF, ECLF2 had similar AUC to LR but a better trade-off between sensitivity and specificity.

5. Discussion

We have presented the development and validation of ensemble classifiers with hard voting for sleep apnea screening. Performance evaluation analysis showed that when heavier weights were assigned to the LR and SVM classifiers, the ECLF achieved a better balance between sensitivity (0.81 ± 0.02) and specificity (0.80 ± 0.02) while maintaining the overall performance as measured by AUC (0.81 ± 0.01). RF and LDA achieved high sensitivity (>0.95) at the sacrifice of specificity, while LR and SVM achieved high specificity (>0.80) at the sacrifice of sensitivity. LGMB demonstrated mediocre performance on both sensitivity and specificity. In our future work, we plan to apply other ensemble techniques, such as soft voting, and to tune the weights given to the base classifiers.