Inter-Rater Variability in the Evaluation of Lung Ultrasound in Videos Acquired from COVID-19 Patients

Abstract

Lung ultrasound (LUS) allows for the detection of a series of manifestations of COVID-19, such as B-lines and consolidations. The objective of this work was to study the inter-rater reliability (IRR) when detecting signs associated with COVID-19 in the LUS, as well as the performance of the test in a longitudinal or transverse orientation. Thirty-three physicians with advanced experience in LUS independently evaluated ultrasound videos previously acquired using the ULTRACOV system on 20 patients with confirmed COVID-19. For each patient, 24 videos of 3 s were acquired (using 12 positions with the probe in longitudinal and transverse orientations). The physicians had no information about the patients or other previous evaluations. The score assigned to each acquisition followed the convention applied in previous studies. A substantial IRR was found in the cases of normal LUS (κ = 0.74), with only a fair IRR for the presence of individual B-lines (κ = 0.36) and for confluent B-lines occupying < 50% (κ = 0.26) and a moderate IRR in consolidations and B-lines > 50% (κ = 0.50). No statistically significant differences between the longitudinal and transverse scans were found. The IRR for LUS of COVID-19 patients may benefit from more standardized clinical protocols.

1. Introduction

Lung ultrasound (LUS) is used to quickly and precisely differentiate between the most common causes of respiratory problems. It has been extensively studied as a bedside diagnostic tool and is now universally included in point-of-care ultrasound (PoCUS) guidelines with high-quality supporting evidence [1]. LUS has the potential to refashion healthcare delivery as it enables an augmented clinical interpretation of a patient’s status in real time, which could have an immediate impact on clinical decisions, and even be used to monitor response to therapy and evolution [2,3,4]. Moreover, LUS imaging is typically less expensive than conventional chest X-ray or computed tomography (CT), making it convenient for locations with limited access to these resources [5,6].

LUS has demonstrated an ability to provide immediate information on the condition of COVID-19 patients [4,7]. There are multiple pulmonary manifestations of COVID-19 that can be observed with LUS, such as the presence of pleural effusion, B-line artifacts, or consolidations [8,9,10].

LUS allows physicians to perform a complete bedside chest exam on both mild and severe COVID-19 patients. It is a useful imaging technique for detecting and monitoring the lung involvement as well as prognosis of the disease, and predicting admission to an intensive care unit (ICU) and mortality [3,4,7,11,12]. Furthermore, LUS reduces the risk of environment cross-infection compared to other imaging modalities as these devices can be more easily cleaned and disinfected after use [6,13].

However, LUS is an operator-dependent imaging technique, and its utility depends on accurate acquisition and interpretation by bedside physicians [14,15,16,17]. Poor image acquisition and incorrect identification and interpretation of artifacts are potential sources of error in its clinical application [1]. In previous studies (most of which were conducted before the pandemic), LUS findings showed moderate to fair inter-rater agreement. However, as the observed agreement in the interpretation of frequently occurring events may be due to some extent to chance, more studies with a controlled environment are required to determine the accuracy with which physicians can interpret LUS acquisitions.

Furthermore, to the best of our knowledge, there are no published studies to date that specifically evaluate the best orientation of the transducer (i.e., transverse or longitudinal) in an LUS acquisition in COVID-19 patients (see Figure 1).

Our aim was to first characterize the inter-rater agreement of LUS experts when evaluating the main findings for COVID-19. Our hypothesis was that kappa agreement in ultrasound artifacts and diagnostic interpretation would be substantial, based on the high agreement in other clinical scenarios. We also evaluated the impact of the transducer orientation in LUS acquisitions in COVID-19 patients on the observed findings.

2. Materials and Methods

In this study, a total of 33 physicians (internal medicine n = 16; intensivist n = 4; family physician n = 5; pneumology = 1; pediatrics = 1; and emergency medicine, n = 6) with advanced experience in performing and interpreting LUS, from 29 different healthcare centers in Spain, independently evaluated previously acquired ultrasound videos of 20 patients. All had more than 3 years’ experience performing and interpreting LUS.

The acquisitions corresponded to patients with COVID-19 diagnosed by nasopharyngeal RT-PCR for SARS-CoV-2 obtained in the internal medicine service of two different hospitals in Madrid collected during the summer of 2021 [8]. All studies were collected by two physicians (YT-C, AT-V), who followed a 12-areas LUS protocol and a 0–3 point per finding score system [18] (Figure 1). Specifically, each area was scored from 0 to 3 according to the observed patterns (Figure 2). Score 0 is associated with the physiological horizontal artifacts, A-lines. Score 1 is assigned when isolated vertical artifacts appear (B-lines). Score 2 represents confluent B-lines in less than 50% of the pleural line. Score 3 is associated with confluent B-lines extending more than 50% of the pleural line, as well as subpleural or lobar consolidation or pleural effusion. In each zone, the ultrasound probe was used in longitudinal and transverse positions, and a 3 s video of 20 fps was recorded. In total, 24 videos of 3 s each were acquired per patient. No patient had more than one scan in the database. Each physician assigned the highest score to the 3 s video.

In all cases, the data were obtained with a ULTRACOV ultrasound scanner prototype using a 3.5 MHz convex probe with 128-channel ultrasound electronics [8]. This resulted in a total of 480 videos (28,800 frames). The imaging depth in all cases was set to 13 cm. More details of the data acquisition can be found in a recently published work, which focused on the automatic calculation of LUS score [8].

The data were acquired from a study conducted at a tertiary academic hospital and an emergency field hospital that investigated a reduction in the exploration time per patient when using an ultrasound system developed specifically for LUS. The study was reviewed by our institutional review board (IRB) and approved at both participating sites. Informed consent was obtained for each patient.

Several LUS protocols have been proposed for the lung assessment of COVID-19 patients based on the number of areas or points to explore. We adopted a 12-zone scanning protocol, which was previously validated and shown to be consistent with higher ICC and a higher degree of concordance with CT [18].

The selected patients for this study (n = 20) were selected from the total acquired dataset from that study (n = 28) so that half of the cases (n = 10) corresponded to patients in relatively good condition (with a total score between 1 and 7 based on the in situ assessment of the LUS expert), while the other half (n = 10) corresponded to patients who had a moderate condition (with a score between 8 and 18 based on the in situ assessment of the LUS expert). As the LUS device was not located within the ICU, no severe cases were present in the database.

The physicians had no information about the patients in the survey and were blinded to their history and clinical information. They also did not have access to the characteristics of the scanner and the evaluations performed by the other physicians. All the videos from the same patient were supplied together before moving on to the next patient, as would happen in a regular patient examination.

The sonographer expert who collected the videos also conducted the survey, so that a comparison between the findings obtained during the examination and those observed in the surveyed videos was also undertaken. As the survey was performed 9 months after the scans, there was no recollection of each patient’s status at that time.

2.1. Preparation

All participant physicians were instructed to evaluate the de-identified studies and provide their interpretation using a web survey. They received instructions at the beginning of the study, which included the scanning protocol and definition of the orientation of the probe in each case. No other information on the interpretation or definition of the LUS findings was provided during the evaluation. The physicians were blinded to any clinical or imaging information. Furthermore, they had no prior experience with the system used to collect the videos.

In previous studies [4,14], physicians met before performing the evaluations for the inter-rater study to review some sample videos to discuss their interpretation. In this case, no previous calibration session was conducted.

2.2. Data Analysis

We performed a series of statistical analyses comparing the interpretation of the presence of ultrasound artifacts and the ultrasound diagnosis performed by the physicians. All were performed with Python using NumPy and Scikit-learn libraries.

First, we evaluated the agreement between raters of the individual scores (0, 1, 2, or 3) assigned by each observer to each of the 480 videos in the study. These videos correspond to the 20 patients, with 12 zones and two probe orientations each. Cohen’s kappa was used to quantify the inter-rater agreement between each pair of physicians [19]. The coefficient ranges from −1 to +1, with 0 representing random chance and 1 representing perfect agreement.

Based on the total score from the evaluation of the 12 zones, patients were classified into four subgroups: A. total score 0; B. total score between 1 and 7; C. total score between 8 and 18; D. total score between 19 and 36. These subgroups have been used in previous studies to obtain a fair indication of the severity of their condition. Similar to previous cases, Cohen’s kappa between this four-class classification was obtained. In this case, analysis was performed separately for the longitudinal and transverse examinations. The results are shown in Figure 3.

Agreement in the interpretation of each ultrasound artifact (A-lines, isolated B-lines, confluent B-lines, and consolidations) was also assessed separately. The degree of inter-rater agreement was evaluated using Fleiss’ kappa statistics (k) [19,20]. Kappa values close to 1 imply strong agreement beyond chance in LUS diagnosis [19,20]. We interpreted the scaled kappa statistics as follows: k ≤ 0, less than chance agreement; k 0.01–0.20, slight agreement; k 0.21–0.40, fair agreement; k 0.41–0.60, moderate agreement; k 0.61–0.80, substantial agreement; and k ≥ 0.81, near-perfect agreement [19,20].

Table 1. Fleiss kappa analysis of the inter-rater agreement in the findings in all the videos.

Score	Finding	Fleiss Kappa (k and 95% CI)		Agreement
0	Normal/A-lines	0.74	[0.71–0.76]	Substantial
1	Individual B-lines	0.36	[0.33–0.39]	Fair
2	Confluent B-lines < 50%	0.26	[0.24–0.29]	Fair
3	Confluent B-lines > 50% & Consolidations	0.50	[0.47–0.53]	Moderate

contains the results of this analysis.

As an alternative means of visualizing agreement in the findings, Figure 5 shows a matrix of the scores assigned to each video with respect to the most voted score (among the 33 evaluations), which can be considered as a surrogate for the ground truth. This provides a quick view of the most challenging scores. Agreement between the comparison of the ultrasound diagnosis performed in situ with the recorded videos was also undertaken utilizing k values adjusted for maximum attainable agreement.

3. Results

3.1. Patients

The selected patients in this study (n = 20) corresponded to COVID-19 admissions to hospital. The mean age was 53.2 years (standard deviation (SD) 11.9) and 45% was female. Five patients (25%) had hypertension, two (10%) were diabetic, and none had cardiovascular disease. They had an average of 19.1 days (SD 20.6) after symptom onset, consisting of fever (95%), shortness of breath (75%), and weakness (85%). The mean lymphocyte count was 1.81 × 109 (SD 1.00), C-reactive protein was 29.4 mg/dL (SD 33.3), and D-dimer 536.47 ng/mL (SD 315.7) at admission. No patients died at follow-up. The LUS exams were performed within 2–3 days of their hospital admission after obtaining consent. None ended up in ICU and all were discharged after several days/weeks in hospital.

3.2. Overall Agreement between Raters

The overall Cohen’s kappa statistics between each pair of raters of the 480 videos are shown in Figure 4. Raters are sorted from left to right based on their overall Cohen’s kappa with their peers (0.45 to 0.78 variation).

Comparison of the evaluations of the sonographer who collected the videos performed during the examination with respect to those observed in the surveyed videos indicates a Cohen’s kappa value of 0.68 (moderate agreement).

The most relevant outcome of the patient LUS evaluation is their classification into four subgroups based on the severity of their lung condition. Therefore, we evaluated Cohen’s kappa between the classification of patients in each subgroup performed by each physician considering longitudinal and transversal directions (Figure 4). The agreement was slightly higher with the studies performed in the longitudinal direction (Figure 4).

3.3. Agreement in Specific Findings

Regarding the degree of agreement between physicians with respect to the specific findings, Table 1 summarizes Fleiss’ kappa analysis. There was good agreement in determining (normal) A-lines (κ = 0.74) and fair agreement in determining the presence of individual B-lines (κ = 0.36), as well as on the presence of confluent B-lines occupying less than 50% of the ultrasound image (κ = 0.26). Moderate agreement was found for confluent B-lines occupying more than 50% and consolidations (κ = 0.50).

Figure 5 shows a matrix with information on how the scores were assigned to each video with respect to the most voted score (mode) in each case. The most voted score (mode) may be considered a good estimation of the ground truth. The largest differences were found for score = 2.

3.4. Agreement in Specific Findings

Regarding the impact of the probe orientation (longitudinal or transversal, as shown in Figure 1) when performing the study, the total score assigned to each patient in both cases is shown in Figure 6. A scatter plot shows very good correlation between both types of examination (R² = 0.87) and the Bland–Altman plot of longitudinal minus transversal scores indicates that, on average, the longitudinal view indicates slightly lower scores (−1.12).

4. Discussion

Easy-to-access and reliable diagnostic methods, which can accurately guide the management of COVID-19, are vital in non-hospital settings and areas with limited resources [5]. Some studies have noted that LUS could be a first-line diagnostic alternative to conventional chest X-ray and CT scans since there is no exposure to ionizing radiation [4,6,13] and should be encouraged to avoid transporting patients and reduce the risk of environmental contamination [21,22,23]. Moreover, it could be considered in vulnerable populations, such as pregnant women and children.

Previous research has shown that COVID-19 has notable LUS characteristics, such as B-lines or consolidations [11]. These findings correlate well with COVID-19 CT findings, such as ground-glass consolidations and septal thickening [9]. As a result, given that LUS may be able to predict outcomes in COVID-19 patients, it is crucial to ascertain whether clinicians can correctly interpret these results.

In this study, several LUS findings demonstrated moderate agreement (e.g., consolidations) and others fair agreement (e.g., individual B-lines and confluent B-lines <50%). Therefore, LUS could signify a reliable COVID-19 diagnostic and prognostic tool. Moreover, there was good agreement as to whether an LUS scan was interpreted as normal. In addition, beyond COVID-19, an abnormal LUS scan has prognostic implications for multiple diseases. This study represents the first study to assess inter-observer agreement in LUS findings in COVID-19 obtained with the same device and including practitioners from multiple specialties and centers, who commonly use different portable devices.

Our results are similar to other previous studies on inter-rater reliability for LUS outside of COVID-19. Previous investigations have demonstrated moderate to substantial agreement for B-lines [15,16,17], while this research shows only moderate to fair agreement for consolidations. This is similar to the results obtained in a previous LUS study with COVID-19 patients [14].

This work shows the importance of working toward a more standardized interpretation protocol. Among the possible solutions, the following options should be considered:

(1)

Standardization of the terminology to describe artifacts and signs in LUS is essential. Several definitions of each LUS abnormality can be found in the literature, especially for consolidations, but also regarding pleural abnormalities [1], which were not considered in this study. This group believes that the reliability of findings such as consolidations might improve with a more specific and consensus-based definition.

(2)

The use of automatic tools to quickly analyze the acquisitions and obtain some quantitative values, such as the percentage of affected pleura (B-lines < 50% or >50%) and the size of the consolidations, may be helpful to obtain more consistent results among raters.

(3)

The length of the acquired videos (3 s in this study) could be extended to provide more information in some cases.

(4)

Access to additional clinical data about the patients may also help in their evaluation.

There are several limitations to this study. Due to its dynamic nature, the use of LUS fundamentally differs from traditional medical imaging practices where an exam is performed by a technologist and interpreted remotely by a physician with limited clinical knowledge of the patient. The same provider performs and interprets the study, immediately integrates the findings into the clinical setting, and repeats the study as needed to identify changes associated with bedside interventions. In this case, the raters did not have the opportunity to explore the patients or adapt the ultrasound exploration according to their preferences and findings. Therefore, despite allowing us to evaluate the scans in a very controlled setting (same device, same image quality, etc.), this type of patient observation is not realistic. This fact may have caused some errors in the interpretation of some particular cases. The impact of this was evaluated by performing a comparison of the evaluations the sonographer who collected the videos during the examination with respect to those observed in the surveyed videos. The moderate agreement found (Cohen’s kappa 0.68) in this case is a good indication of the differences that might be expected between in situ evaluations and those performed with a recorded video.

Furthermore, in this study, there were no patients who suffered extremely severe conditions. This reduced the number and size of the consolidations (if present), making them more difficult to identify (see Figure 7). As shown in Figure 5 and Table 1, the cases with score = 2 were those with significantly higher disagreements.

In terms of which is the best way to perform LUS, i.e., whether to use longitudinal or transversal view, our results show that there is a very good correlation between both types of examination (R² = 0.87), although, on average, the transversal view provides slightly higher scores (1.12). This was expected as avoiding the ribs provides a larger field of view of the lungs and, therefore, a higher probability of detecting pneumonia-related artifacts. The difference is small and does not impact the classification of patients into subgroups for most patients. However, in our case, 4 out of 20 patients changed their subgroup, with 3 increasing their subgroup classification with a transversal view and one decreasing the subgroup (Figure 6). This does not necessarily mean the two orientations are similarly useful, especially since we only examined a type of interstitial lung disease. In certain pathologies, such as pneumothorax, the visualization of the ribs provides a depth landmark and helps to better identify the pleural line. Consequently, our group believes that each patient might benefit most from a different approach, which has been adapted to a flexible scanning protocol subject to the clinical scenario. We would like to acknowledge that, as in all cases, the 12 videos of the transversal view of each patient were evaluated right after the 12 videos of the longitudinal view, this may have created some undesirable correlation between both evaluations. This was chosen to mimic the original in situ study, but a more randomized order of the videos could have been a better choice.

Furthermore, pathological findings such as B-lines may have been better represented than others (consolidations and pleural effusions). Despite these limitations, this study represents one of the most controlled studies into the inter-observer agreement of LUS findings for COVID-19.

Other studies could be conducted with the gathered data. For instance, a study of variability by region (i.e., anterior vs lateral vs posterior), upper and lower, left and right, etc., could be conducted. Furthermore, we did not include AI tools, which are able to evaluate the acquired videos and compare them with human observers. In this work, the AI tool used in [8,16] was not compared but will be part of future work.

5. Conclusions

The most reliable LUS findings with COVID-19 were the presence of B-lines or determining whether a scan is normal. We did not observe statistically significant differences between the longitudinal and transverse scans. The IRR in LUS of COVID-19 patients may benefit from more standardized clinical protocols.