Evaluation of the Usefulness of CO-RADS for Chest CT in Patients Suspected of Having COVID-19

The purpose of this study was to use the Coronavirus Disease 2019 (COVID-19) Reporting and Data System (CO-RADS) to evaluate the chest computed tomography (CT) images of patients suspected of having COVID-19, and to investigate its diagnostic performance and interobserver agreement. The Dutch Radiological Society developed CO-RADS as a diagnostic indicator for assessing suspicion of lung involvement of COVID-19 on a scale of 1 (very low) to 5 (very high). We investigated retrospectively 154 adult patients with clinically suspected COVID-19, between April and June 2020, who underwent chest CT and reverse transcription-polymerase chain reaction (RT-PCR). The patients’ average age was 61.3 years (range, 21–93), 101 were male, and 76 were RT-PCR positive. Using CO-RADS, four radiologists evaluated the chest CT images. Sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were calculated. Interobserver agreement was calculated using the intraclass correlation coefficient (ICC) by comparing the individual reader’s score to the median of the remaining three radiologists. The average sensitivity was 87.8% (range, 80.2–93.4%), specificity was 66.4% (range, 51.3–84.5%), and AUC was 0.859 (range, 0.847–0.881); there was no significant difference between the readers (p > 0.200). In 325 (52.8%) of 616 observations, there was absolute agreement among observers. The average ICC of readers was 0.840 (range, 0.800–0.874; p < 0.001). CO-RADS is a categorical taxonomic evaluation scheme for COVID-19 pneumonia, using chest CT images, that provides outstanding performance and from substantial to almost perfect interobserver agreement for predicting COVID-19.


Introduction
In the city of Wuhan, Hubei province, China, a highly contagious disease was first identified at the end of 2019, which was later proven to be caused by a novel coronavirus (2019-nCoV or severe acute respiratory syndrome coronavirus 2) [1]. As of 10 August 2020, more than 19,718,000 people worldwide have been infected with coronavirus disease 2019 (COVID- 19), and more than 728,000 have died [2].
Early diagnosis of COVID-19 is crucial for the isolation of patients and prevention of the spread of infection, as well as early patient intervention. The gold standard for COVID-19 diagnosis is the reverse transcription-polymerase chain reaction (RT-PCR) assay. However, its sensitivity is variable as it depends on the duration of symptoms, viral load, the rate of viral replication in the upper respiratory The CO-RADS categories are summarized as follows [8]: 1. CO-RADS 0 is chosen if none of the five categories can be assigned because of incomplete scans or poor quality. 2.
CO-RADS 1 means a very low level of suspicion for pulmonary lesion resulting from COVID-19, based on either a normal CT or CT findings of unequivocal non-infectious etiology. Mild or severe emphysema, perifissural nodules, lung tumors, or fibrosis characterize CO-RADS 1. 3.
CO-RADS 2 means a low level of suspicion for pulmonary lesion resulting from COVID-19, based on CT findings in the lungs typical of etiology indicative of infections other than COVID-19, including bronchitis, infectious bronchiolitis, bronchopneumonia, lobar pneumonia, and pulmonary abscess. Findings include a tree-in-bud sign, a centrilobular nodular pattern, lobar or segmental consolidation, and lung cavitation. 4.
CO-RADS 3 means CT findings equivocal for pulmonary involvement of COVID-19 that can also be seen in other types of viral pneumonia or non-infectious etiologies. Findings include perihilar ground-glass, homogenous extensive ground-glass with or without sparing of some secondary pulmonary lobules, or ground-glass together with smooth interlobular septal thickening with or without pleural effusion in the absence of other typical CT findings. 5.
CO-RADS 4 means a high level of suspicion for pulmonary lesion resulting from COVID-19, based on typical CT findings but exhibiting some overlap with other types of (viral) pneumonia. Findings are not located in contact with the visceral pleura or are located strictly unilaterally, have a predominantly peribronchial distribution, or overlap with severe diffuse preexisting pulmonary abnormalities. 6.
CO-RADS 5 means a very high level of suspicion for pulmonary lesion resulting from COVID-19, based on typical CT findings. Obligatory features are ground-glass opacities, with or without consolidations, close to visceral pleural surfaces, including the fissures, and a multifocal bilateral distribution.

Readout
Chest CT images were reviewed by four radiologists (Reader 1 with 1 year, Reader 2 with 6 years, Reader 3 with 7 years, and Reader 4 with 10 years of experience) using CO-RADS. Reader 1 and Reader 2 were board-certified radiologists, and the remaining two were radiology residents. They evaluated the images using the medical viewing system EV Insite R (PSP Co., Tokyo, Japan), with feature reading tools.
Although they knew the age and sex of the patients, they were blind to other clinical information. Readers categorized each patient as CO-RADS 0, 1, 2, 3, 4, or 5.

Statistical Analysis
All statistical analyses in this study were performed using SPSS for Windows version 22.0 (IBM Corp., New York, NY, USA).
We performed a Student's t-test or Chi-squared test to compare characteristics (age, sex, comorbidities) of patients with or without COVID-19. The sensitivity and specificity for differentiating COVID-19-positive from COVID-19-negative patients. Receiver operating characteristic (ROC) analyses were performed to calculate the area under the ROC curve (AUC) for evaluations of the four readers in distinguishing COVID-19-positive from COVID-19-negative patients using CO-RADS categorization. Then, the sensitivity and specificity were calculated using cut-off value ≥ Category 3. Interobserver agreement was assessed using the intraclass correlation coefficient (ICC). ICCs were calculated by comparing the CO-RADS score of each reader with the median of the remaining three readers and were interpreted as follows: <0.20, slight; 0.21-0.40, fair; 0.41-0.60, moderate; 0.61-0.80, substantial; and 0.81-1.0, almost perfect [11]. We considered a p-value of less than 0.05 to be statistically significant. Table 2 shows the characteristics of the 154 patients included in this study. The patients ranged in age from 21 to 93 years (mean ± SD; 61.3 ± 18.8), with 101 males and 53 females. All patients were clinically suspected of having COVID-19 and underwent one or more RT-PCR tests. Seventy-six of them tested positive with RT-PCR and were diagnosed with COVID-19. Among the COVID-19 patients, clinical presentations included fever (n = 67; 88.1%), cough (n = 32; 42.1%), dyspnea (n = 26; 34.2%), and fatigue (n = 13; 17.1%). There were no asymptomatic COVID-19 patients in this study. The time between symptom onset and the CT scan was between 0 and 27 days (mean ± SD; 8.6 ± 5.6). Among the 78 COVID-19-negative patients, 36 were diagnosed as having pneumonia other than COVID-19, which were community-acquired pneumonia (n = 29), interstitial pneumonia (n = 4), drug-induced pneumonia (n = 1), eosinophilic pneumonia (n = 1), and lung abscess (n = 1). Fifteen patients were diagnosed with diseases other than pneumonia, which were pulmonary edema (n = 5), atelectasis (n = 4), lung tumor (n = 3), pleuritis (n = 2), and emphysema (n = 1). Twenty-seven patients were diagnosed as normal (Table 3). Patients without COVID-19 were older and had more comorbidities of cancer and collagen disease than the patients with COVID-19 (p = 0.006, <0.001 and 0.022). However, there was no significant difference between the two groups for sex ratio, comorbidity of cardiovascular disease, lung disease, and diabetes.  Figure 1).     Although the interobserver agreement for predicting COVID-19 of Reader 1 was substantial, that of the remaining three readers was almost perfect. The ICC average of readers was 0.840 (range, 0.800-0.874; Table 4). Figure 3; Figure 4 show representative true-positive cases, and Figure 5 represents a true negative case. Table 5; Table 6 list the CO-RADS category and characteristics of the cases that were diagnosed as false positives (i.e., non-COVID-19 patients with the median of Category 4 or 5) or false negatives (i.e., COVID-19 patients with the median of Category 1 or 2 by readers). There were 10 false-positive and 6 false-negative cases. The final diagnoses of the false-positive cases were community-acquired pneumonia (n = 5), interstitial pneumonia (n = 1), drug-induced pneumonia (n = 1), eosinophilic pneumonia (n = 1), pulmonary edema (n = 1), and pleuritis. Many of the false-negative cases were young patients. One patient was in his 90s and had no comorbidities. The other five patients were in their 20 s or 30 s.             Diagnostics 2020, 10, 608 9 of 12 Figure 5. Representative CT images of a COVID-19-negative patient. Thirty-four-year-old female. The CT images show segmental consolidation with mucus plug (arrow). RT-PCR was negative for SARS-CoV-2 and she was diagnosed with community-acquired pneumonia. All readers diagnosed it as Category 2. COVID-19, coronavirus disease 2019; CT, computed tomography; RT-PCR, reverse transcription-polymerase chain reaction; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2.

Discussion
COVID-19 is an alarming infectious disease that is raging around the world. COVID-19 pneumonia, as has been reported, can cause severe respiratory failure and even death, especially in the elderly and patients with comorbidities [12,13].
The CO-RADS classification devised by the Dutch Radiological Society is a simple interpretation method with a high inspection accuracy and high matching rate. The widespread adoption of CO-RADS is expected to enable standardization of CT image interpretation worldwide but evidence for it is still insufficient. This is the first study, to our knowledge, to verify the effectiveness of CO-RADS with Japanese data.
The diagnostic performance of CT for COVID-19 had an average sensitivity of 87.8%, specificity of 66.4%, and AUC of 0.859. Although radiologists with varying years of experience assessed the images, there was no significant difference between the readers (p > 0.200). Moreover, there was substantial to almost perfect interobserver agreement among the readers. The average ICC of readers was 0.840 (range, 0.800-0.874).
It has been reported that in epidemic areas CT scans play an important role in diagnosing COVID-19 because of their high sensitivity, and combining RT-PCR with CT scans is expected to improve the diagnosis of COVID-19 [4,5]. This study was conducted in a population where approximately half of the patients were COVID-19 positive and all COVID-19 positive patients were symptomatic, and we were able to achieve high diagnostic performance results using CT scans. However, in areas with lower prevalence, it has been reported that their specificity and positive predictive value are lower. Further validation is needed as to whether CO-RADS is useful for diagnosing COVID-19 in non-endemic areas [14]. It is also known that young people without comorbidities are often asymptomatic or mildly symptomatic, even if infected with COVID-19 [15].
Despite the patients in this study being suspected of having pneumonia because of their symptoms and, thereby, being diagnosed with COVID-19, 6/76 (7.9%) patients had no CT findings of pneumonia. One patient was an older man with no comorbidities, and the other five patients were in their 20 s or 30 s. Because there is insufficient evidence for the effectiveness of screening patients who are asymptomatic or have mild symptoms, especially if they are young, it is necessary to thoroughly verify whether CT scans should be used for such patients.
This study also included 10 false-positive cases, consisting of community-acquired pneumonia, interstitial pneumonia, drug-induced pneumonia, eosinophilic pneumonia, pulmonary edema, and pleuritis. Because the image findings of these diseases overlap with that of COVID-19 pneumonia, it was difficult to make a perfect differential diagnosis, even with CO-RADS. When diagnosing patients suspected of having COVID-19 pneumonia, it is crucial to make a comprehensive assessment based on both CT images and clinical findings. In recent years, artificial intelligence, especially deep learning, has been greatly developed and applied to medical imaging [16][17][18][19]. Ni et al. utilized a deep

Discussion
COVID-19 is an alarming infectious disease that is raging around the world. COVID-19 pneumonia, as has been reported, can cause severe respiratory failure and even death, especially in the elderly and patients with comorbidities [12,13].
The CO-RADS classification devised by the Dutch Radiological Society is a simple interpretation method with a high inspection accuracy and high matching rate. The widespread adoption of CO-RADS is expected to enable standardization of CT image interpretation worldwide but evidence for it is still insufficient. This is the first study, to our knowledge, to verify the effectiveness of CO-RADS with Japanese data.
The diagnostic performance of CT for COVID-19 had an average sensitivity of 87.8%, specificity of 66.4%, and AUC of 0.859. Although radiologists with varying years of experience assessed the images, there was no significant difference between the readers (p > 0.200). Moreover, there was substantial to almost perfect interobserver agreement among the readers. The average ICC of readers was 0.840 (range, 0.800-0.874).
It has been reported that in epidemic areas CT scans play an important role in diagnosing COVID-19 because of their high sensitivity, and combining RT-PCR with CT scans is expected to improve the diagnosis of COVID-19 [4,5]. This study was conducted in a population where approximately half of the patients were COVID-19 positive and all COVID-19 positive patients were symptomatic, and we were able to achieve high diagnostic performance results using CT scans. However, in areas with lower prevalence, it has been reported that their specificity and positive predictive value are lower. Further validation is needed as to whether CO-RADS is useful for diagnosing COVID-19 in non-endemic areas [14]. It is also known that young people without comorbidities are often asymptomatic or mildly symptomatic, even if infected with COVID-19 [15].
Despite the patients in this study being suspected of having pneumonia because of their symptoms and, thereby, being diagnosed with COVID-19, 6/76 (7.9%) patients had no CT findings of pneumonia. One patient was an older man with no comorbidities, and the other five patients were in their 20 s or 30 s. Because there is insufficient evidence for the effectiveness of screening patients who are asymptomatic or have mild symptoms, especially if they are young, it is necessary to thoroughly verify whether CT scans should be used for such patients.
This study also included 10 false-positive cases, consisting of community-acquired pneumonia, interstitial pneumonia, drug-induced pneumonia, eosinophilic pneumonia, pulmonary edema, and pleuritis. Because the image findings of these diseases overlap with that of COVID-19 pneumonia, it was difficult to make a perfect differential diagnosis, even with CO-RADS. When diagnosing patients suspected of having COVID-19 pneumonia, it is crucial to make a comprehensive assessment based on both CT images and clinical findings. In recent years, artificial intelligence, especially deep learning, has been greatly developed and applied to medical imaging [16][17][18][19]. Ni et al. utilized a deep learning model for automatic detection of abnormalities in chest CT images of COVID-19 patients. They demonstrated the deep-learning model performed exceptionally in detecting COVID-19 pneumonia on chest CT and could assist radiologists in making quicker diagnoses with superior diagnostic performance [20]. Further research will be needed to compare the diagnostic performance of deep learning algorithms with human readers, and to verify whether deep learning can assist radiologists and improve their accuracy and efficiency in the diagnosis of COVID-19.
This study had several limitations. First, our retrospective study was conducted at a single institution. Thus, to validate the findings of this research, more extensive, multicenter studies are necessary. Second, we used several CT systems from different companies. This may have caused variations in image quality, which in turn may have affected the diagnostic performance results of our readers. Third, the diagnostic criteria for COVID-19 in this study were defined as the results of RT-PCR, but it is known that these diagnoses are not perfect. Some patients may have false negatives or false positive Finally, we did not examine the relationship between CT image findings and severity and prognosis of disease [21]. Moreover, the timing of execution of chest CT scan after the onset of symptoms was inhomogeneous, and this fact represents a limitation, as chest abnormalities change during the diseases. Future research is necessary to clarify these relationships.

Conclusions
CO-RADS is a categorical taxonomic evaluation scheme for COVID-19 pneumonia, using chest CT images, 1 provides outstanding performance and from substantial to almost perfect interobserver agreement for predicting COVID-19. Interpretation of chest CT images using CO-RADS may contribute to the diagnosis and management of COVID-19 pneumonia.