Validation of the Modified Berlin Questionnaire for the Diagnosis of Obstructive Sleep Apnea in Patients with a History of COVID-19 Infection

Round 1

Reviewer 1 Report

In this manuscript, the authors investigated the efficacy of the newly revised questionnaire for obstructive sleep apnea (OSA). Modified Berlin Questionnaire (mBQ) has higher sensitivity and specificity than usual BQ. The study provides an important contribution to the treatment of OSA after COVID-19 era.

 

Major points

1)     The authors use 5, 15, and 30 as thresholds for AHI. Although I admit the results of the data by using these thresholds are beautiful, I think AHI=10 and 20 are also important thresholds in diagnoses of OSA. Please indicate whether similar results can be obtained with AHI=10 or 20.

 

Minor point

1)     none

Author Response

In this manuscript, the authors investigated the efficacy of the newly revised questionnaire for obstructive sleep apnea (OSA).

Modified Berlin Questionnaire (mBQ) has higher sensitivity and specificity than usual BQ.

The study provides an important contribution to the treatment of OSA after COVID-19 era.

• Thank you for the positive comments.

Major points

The authors use 5, 15, and 30 as thresholds for AHI. Although I admit the results of the data by using these thresholds are beautiful, I think AHI=10 and 20 are also important thresholds in diagnoses of OSA.  Please indicate whether similar results can be obtained with AHI=10 or 20.

• Additional analysis testing AHI cut-off 10 events/h lower sensitivity (72.7 [54.5-86.7]%) and specificity (91.9 [78.1-98.3]%) than the values for AHI cut-off 15 events/h, and the accuracy was also lower (82.9 [72.0-90.8]%). When testing the AHI cut-off 20 events/h, the sensitivity was higher (95.0 [75.1-99.9]%) but the specificity (84.0 [70.1-92.8]%) and the accuracy (87.1 [77.0-94.0]%) were lower than the values for AHI cut-off 15 events/h. These results are now included in the manuscript.

Minor point

none

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors have modified the Berlin questionnaire (BQ) to identify risk of OSA in patients with acute COVID. The modification removes the items for obesity and hypertension, leaving 3 categories for snoring, apneas and tiredness, where two out of 3 categories have to be positive to make a respondent high risk for OSA. An earlier study from the same group recruited 320 patients diagnosed with COVID in March to June 2020, and administered the modified BQ. This is a follow-on study, where a subset of 70 participants had an in-lab PSG, and because there has been some time elapsed (a year or more?)  since the original questionnaires, the modified BQ was repeated and it was this score that was compared with the PSG results.

 

I commend the authors comprehensive display of the key test performance metrics, including the 95% confidence limits. The performance of the mBQ (eg in terms of sensitivity and specificity) is much better than the performance of the standard BQ in many other studies. 

 

I understand that there would have been a delay with getting PSGs due to covid related disruptions to the healthcare system, and hence the decision to repeat the BQ, however to answer the research question (identifying OSA risk in those with COVID) the initial BQ would have been the one to assess against the PSG. It is possible that responses could be affected by acute the covid infection, for example they may have responded higher on the tiredness question or acute pharyngitis may have led to a greater tendency for upper airway obstruction. This is relevant because the simple agreement between mBQ administered initially and at the time of the PSG is very low at 0.43: this may also be due to the long time period between the first and second questionnaire administration. Perhaps this could be discussed further as a limitation of the research, and measures put in place in studies to replicate the findings, to minimise the gap between mBQ (still administered at time of COVID) and PSG.

 

I do not agree with how the ROC analyses have been done. The index and reference tests have been reversed: in effect the results shown compare different thresholds of lab PSG AHI in the detection of OSA risk defined by mBQ. The reference test (gold standard) really should be defined by the PSG AHI, for example an AHI of 15 or greater, and the BQ is diagnostic test being evaluated (the index test). 

 

There was a higher proportion of males in the study (73%) – does this reflect the gender distribution of covid infected individuals, or could it suggest a selection bias, with those at risk of OSA being more likely to volunteer?

 

While they are part of the process of validating a questionnaire, the factor analyses and the surveys of experts to determine construct validity have yielded unsurprising results. The items within each category are closely related to each other (snoring intensity and frequency), daytime and morning tiredness, and they are the main symptoms of OSA. 

Author Response

The authors have modified the Berlin questionnaire (BQ) to identify risk of OSA in patients with acute COVID. The modification removes the items for obesity and hypertension, leaving 3 categories for snoring, apneas and tiredness, where two out of 3 categories have to be positive to make a respondent high risk for OSA. An earlier study from the same group recruited 320 patients diagnosed with COVID in March to June 2020, and administered the modified BQ. This is a follow-on study, where a subset of 70 participants had an in-lab PSG, and because there has been some time elapsed (a year or more?)  since the original questionnaires, the modified BQ was repeated and it was this score that was compared with the PSG results.

I commend the authors comprehensive display of the key test performance metrics, including the 95% confidence limits. The performance of the mBQ (eg in terms of sensitivity and specificity) is much better than the performance of the standard BQ in many other studies.

• Many thanks for the positive comments.

I understand that there would have been a delay with getting PSGs due to covid related disruptions to the healthcare system, and hence the decision to repeat the BQ, however to answer the research question (identifying OSA risk in those with COVID) the initial BQ would have been the one to assess against the PSG. It is possible that responses could be affected by acute the covid infection, for example they may have responded higher on the tiredness question or acute pharyngitis may have led to a greater tendency for upper airway obstruction. This is relevant because the simple agreement between mBQ administered initially and at the time of the PSG is very low at 0.43: this may also be due to the long time period between the first and second questionnaire administration. Perhaps this could be discussed further as a limitation of the research, and measures put in place in studies to replicate the findings, to minimise the gap between mBQ (still administered at time of COVID) and PSG.

• As also acknowledged in the manuscript, there was a delay with getting PSGs due to the pandemic. The median time between the initial mBQ during the acute covid infection and the second mBQ for the PSG validation was 406 days (IQR 379-475 days). We have now presented the time delay, and changes in the subscales in the Results section, included a figure (Figure 4), and discussed these findings accordingly. It seems correct that the signs of airway obstruction (snoring and witnessed apneas) were more common (not significant) during the first occasion with acute upper airway infection whereas the tiredness as a sign of post-COVID syndrome was more prevalent at the follow-up survey administration. Since the current study did not aim to confirm the first mBQ results at the acute COVID-19 infection period but to validate the retest mBQ answers when PSG was conducted, the weak correlation between the first mBQ and the retest mBQ may not necessarily mean a weakness of the current protocol. These aspects are now included in the Discussion section.

 

I do not agree with how the ROC analyses have been done. The index and reference tests have been reversed: in effect the results shown compare different thresholds of lab PSG AHI in the detection of OSA risk defined by mBQ. The reference test (gold standard) really should be defined by the PSG AHI, for example an AHI of 15 or greater, and the BQ is diagnostic test being evaluated (the index test).

• Thank you for this important and careful observation. We have now conducted a reverse version, and a new figure, which is much better illustrating the difference between mBQ and BQ with regard ROC for OSA diagnosis defined by the reference PSG AHI cut-off 15 events/h.

 

There was a higher proportion of males in the study (73%) – does this reflect the gender distribution of covid infected individuals, or could it suggest a selection bias, with those at risk of OSA being more likely to volunteer?

• The gender distribution in the first study among 320 covid infected patients was almost equal, 54% males vs 46% females. We had already stated: ‘’ Certain limitations should also be acknowledged. First, the patients who participated in the current overnight PSG study were more likely to be the ones who had sleep related problems, thus, a sample selection bias cannot be excluded. Second, the majority of the current cohort were males, which limits the generalizability of the current findings to the general population’’. We have now revised the statement to better reflect the risk of selection bias for the volunteers, as the Reviewer suggests. “…. The gender distribution in the first study among 320 covid infected patients was almost equal, 54% men vs 46% women but there was a higher proportion of men in the current study (73%), which may reflect that the patients who participated in the current overnight PSG study were more likely to be the ones who had sleep related problems, and thus, a sample selection bias cannot be excluded. The gender imbalance also limits the generalizability of the current findings to the general population.’’

 

While they are part of the process of validating a questionnaire, the factor analyses and the surveys of experts to determine construct validity have yielded unsurprising results. The items within each category are closely related to each other (snoring intensity and frequency), daytime and morning tiredness, and they are the main symptoms of OSA.

• It’s true. Even if the items are related to each other, positive responses to and the proportion of each item differ between the participants. We have now included these comments in Discussion.

Author Response File: Author Response.pdf

Reviewer 3 Report

minor corrections to improve the overall quality:

- in all the subanalyses in the results section and in the tables add p value as appropriate

- Search for the latest and adapt guidelines in the equator database

-  Many of the risk factors and co-morbidities associated for OSA which include obesity, hypertension and diabetes mellitus are associated with poor COVID-19 outcomes. There are plausible mechanisms by which OSA may independently increase one's risk of morbidity and mortality associated with COVID-19 and data from the newly published CORONADO study suggests that OSA treated patients may be at increased risk of death from COVID-19. It is clear that the pandemic has had a major effect on the treatment management and diagnosis of OSA and moving forward it may be necessary to explore new diagnosis and treatment pathways for these individuals. please discuss and cite doi: 10.1016/j.smrv.2020.101382.

- OSA symptoms are usually related to complaints as tinnitus,  vertigo and hypoacusia. Moreover, the vertigo patients have a higher rate of REM-related OSA, and the acceptance rate to CPAP use is low. Further research is needed to explore novel therapeutic approaches, or combination of currently available non-CPAP therapies, in patients with REM OSA. please discuss and cite doi:10.3389/fmed.2021.790931 and doi:10.1371/journal.pone.0252844.

- The individual prognostic factors for coronavirus disease 2019 (COVID-19) are unclear. 263 meta-analyses examining the association of 91 unique prognostic factors with 11 outcomes. Angiotensin-converting enzyme inhibitors, obstructive sleep apnoea, pharyngalgia, history of venous thromboembolism, sex, coronary heart disease, cancer, chronic liver disease, COPD, dementia, any immunosuppressive medication, peripheral arterial disease, rheumatological disease and smoking were associated with at least one outcome and had >1000 events, p<0.005, I²<50%, 95% prediction interval excluding the null value, and absence of small-study effects in the respective meta-analysis. The risk of bias assessment using the Quality in Prognosis Studies tool indicated high risk of bias in 302 out of 428 articles for study participation, 389 articles for adjustment for other prognostic factors and 396 articles for statistical analysis and reporting. please discuss and cite doi:10.1183/13993003.02964-2020. and doi:10.3390/children8100921.

-  An interesting paper discussed the Berlin questionnaire is useful as a clinical screening test and epidemiological tool in the sleep clinic population. Despite limited evidence, it likely has potential clinical and research utility in other populations. Adopting more consistent methodological definitions and focussing more on the general population and specific clinical populations to determine its usefulness as a clinical or epidemiological screening tool are recommended. please discuss and cite doi:10.1016/j.smrv.2017.04.001.

- an observational study consisted of 64 patients recruited for the polysomnography diagnostics of sleep respiratory disturbances. The anthropometric assessment included body weight, height, and body mass index (BMI), all related to the risk of OSA. The BQ consisted of the following three categories: 1 - snoring, 2 - daytime somnolence, and 3 - hypertension. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were evaluated. Likelihood ratio was used to assess the diagnostic accuracy. We found that patients were, on average obese; the mean BMI amounted to 31.9 ± 6.0 kg/m². Polysomnography identified OSA in 73.4% of patients (AHI >5), where the BQ categorized 87.5% of patients at high risk of OSA. Sensitivity of the BQ was 87.2%, specificity 11.8%, PPV 73.2%, and NPV 25.0%. Diagnostic accuracy assessed by the likelihood ratio had a value of 1.00. The BQ had a false discovery rate of 31.2% and a misclassification rate of 32.8%. , please discuss and cite doi:10.1007/5584_2017_7.

 

Author Response

The minor corrections to improve the overall quality:

• Thank you for the positive suggestions.

- In all the subanalyses in the results section and in the tables add p value as appropriate

• All p values are now added accordingly.

- Search for the latest and adapt guidelines in the equator database

• The statement “The manuscript was prepared according to the STARD (Standards for Reporting Diagnostic accuracy studies) 2015 guidelines” is added in the Materials and Methods section.

-  Many of the risk factors and co-morbidities associated for OSA which include obesity, hypertension and diabetes mellitus are associated with poor COVID-19 outcomes. There are plausible mechanisms by which OSA may independently increase one's risk of morbidity and mortality associated with COVID-19 and data from the newly published CORONADO study suggests that OSA treated patients may be at increased risk of death from COVID-19. It is clear that the pandemic has had a major effect on the treatment management and diagnosis of OSA and moving forward it may be necessary to explore new diagnosis and treatment pathways for these individuals. please discuss and cite doi: 10.1016/j.smrv.2020.101382.

• We have already cited both the CORONADO study (ref 4) in Introduction, and the recent systematic review (ref 37) in Discussion. We have now added the statement “The Coronavirus SARS-CoV-2 and Diabetes Outcomes (CORONADO) study suggested that patients with diabetes who were hospitalized for COVID-19 had an almost 3-fold increased risk of death on day 7 (ref 4)” in the first paragraph. We have also added the following statement in the Discussion section “Hence, it is clear that the COVID-19 pandemic has had a major effect on the treatment management and diagnosis of OSA, and it is crucial to explore new diagnosis and treatment pathways for these individuals, who are at high risk of increased morbidity and mortality (ref 37).”

- OSA symptoms are usually related to complaints as tinnitus,  vertigo and hypoacusia. Moreover, the vertigo patients have a higher rate of REM-related OSA, and the acceptance rate to CPAP use is low. Further research is needed to explore novel therapeutic approaches, or combination of currently available non-CPAP therapies, in patients with REM OSA. please discuss and cite doi:10.3389/fmed.2021.790931 and doi:10.1371/journal.pone.0252844.

• We thank the Reviewer for this suggestion, but our paper is a mBQ survey validation study, and these detailed comments/discussions are out of the scope.

- The individual prognostic factors for coronavirus disease 2019 (COVID-19) are unclear. 263 meta-analyses examining the association of 91 unique prognostic factors with 11 outcomes. Angiotensin-converting enzyme inhibitors, obstructive sleep apnoea, pharyngalgia, history of venous thromboembolism, sex, coronary heart disease, cancer, chronic liver disease, COPD, dementia, any immunosuppressive medication, peripheral arterial disease, rheumatological disease and smoking were associated with at least one outcome and had >1000 events, p<0.005, I2<50%, 95% prediction interval excluding the null value, and absence of small-study effects in the respective meta-analysis. The risk of bias assessment using the Quality in Prognosis Studies tool indicated high risk of bias in 302 out of 428 articles for study participation, 389 articles for adjustment for other prognostic factors and 396 articles for statistical analysis and reporting. please discuss and cite doi:10.1183/13993003.02964-2020. and doi:10.3390/children8100921.

• The same reply applies to these suggestions; interesting comments but we do not think that it’s appropriate to discuss these aspects in the current paper.

-  An interesting paper discussed the Berlin questionnaire is useful as a clinical screening test and epidemiological tool in the sleep clinic population. Despite limited evidence, it likely has potential clinical and research utility in other populations. Adopting more consistent methodological definitions and focusing more on the general population and specific clinical populations to determine its usefulness as a clinical or epidemiological screening tool are recommended. please discuss and cite doi:10.1016/j.smrv.2017.04.001.

• This reference was already cited (ref 20). We have now added the statement: “It has been suggested that the BQ is useful as a clinical screening test and epidemiological tool in the sleep clinic population. Adopting more consistent methodological definitions and focusing more on the general population and specific clinical populations to determine its usefulness as a clinical or epidemiological screening tool are also recommended [20].”

- An observational study consisted of 64 patients recruited for the polysomnography diagnostics of sleep respiratory disturbances. The anthropometric assessment included body weight, height, and body mass index (BMI), all related to the risk of OSA. The BQ consisted of the following three categories: 1 - snoring, 2 - daytime somnolence, and 3 - hypertension. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were evaluated. Likelihood ratio was used to assess the diagnostic accuracy. We found that patients were, on average obese; the mean BMI amounted to 31.9 ± 6.0 kg/m2. Polysomnography identified OSA in 73.4% of patients (AHI >5), where the BQ categorized 87.5% of patients at high risk of OSA. Sensitivity of the BQ was 87.2%, specificity 11.8%, PPV 73.2%, and NPV 25.0%. Diagnostic accuracy assessed by the likelihood ratio had a value of 1.00. The BQ had a false discovery rate of 31.2% and a misclassification rate of 32.8%. , please discuss and cite doi:10.1007/5584_2017_7.

• We appreciate the suggestion, but this study with a very low specificity would not add much to the already cited manuscripts validating BQ against an AHI cut-off 5 events/h.

Author Response File: Author Response.pdf