Next Article in Journal
Quantitative Blood Flow Measurements in the Common Carotid Artery: A Comparative Study of High-Frame-Rate Ultrasound Vector Flow Imaging, Pulsed Wave Doppler, and Phase Contrast Magnetic Resonance Imaging
Previous Article in Journal
Fast, Accurate, and Robust T2 Mapping of Articular Cartilage by Neural Networks
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Radiological Findings Increased the Successful of COVID-19 Diagnosis in Hospitalized Patients Suspected of Respiratory Viral Infection but with a Negative First SARS-CoV-2 RT-PCR Result

by
Margarita L Martinez-Fierro
1,*,
Carolina González-Fuentes
1,2,
Dagoberto Cid-Guerrero
2,
Samantha González Delgado
1,
Santiago Carrillo-Martínez
1,2,
Edgar Fernando Gutierrez-Vela
2,
Juan Yadid Calzada-Luévano
1,2,
Maria R. Rocha-Pizaña
3,
Jacqueline Martínez-Rendón
1,
Maria E. Castañeda-López
1 and
Idalia Garza-Veloz
1,*
1
Molecular Medicine Laboratory, Unidad Académica de Medicina Humana y Ciencias de la Salud, Universidad Autónoma de Zacatecas, Zacatecas 98160, Mexico
2
Hospital General “Luz González Cosío”, Circuito Ciudad Gobierno, Zacatecas 98160, Mexico
3
Escuela de Ingenieria y Ciencias, Tecnologico de Monterrey Campus Puebla, Puebla 72453, Mexico
*
Authors to whom correspondence should be addressed.
Diagnostics 2022, 12(3), 687; https://doi.org/10.3390/diagnostics12030687
Submission received: 24 February 2022 / Revised: 5 March 2022 / Accepted: 10 March 2022 / Published: 11 March 2022

Abstract

:
SARS-CoV-2 is the etiological agent of COVID-19 and may evolve from asymptomatic disease to fatal outcomes. Real-time reverse-transcription polymerase chain reaction (RT-PCR) screening is the gold standard to diagnose severe accurate respiratory syndrome coronavirus 2 (SARS-CoV-2) infection, but this test is not 100% accurate, as false negatives can occur. We aimed to evaluate the potential false-negative results in hospitalized patients suspected of viral respiratory disease but with a negative previous SARS-CoV-2 RT-PCR and analyze variables that may increase the success of COVID-19 diagnosis in this group of patients. A total of 55 hospitalized patients suspected of viral respiratory disease but with a previous negative RT-PCR result for SARS-CoV-2 were included. All the participants had clinical findings related to COVID-19 and underwent a second SARS-CoV-2 RT-PCR. Chest-computed axial tomography (CT) was used as an auxiliary tool for COVID-19 diagnosis. After the second test, 36 patients (65.5%) were positive for SARS-CoV-2 (COVID-19 group), and 19 patients (34.5%) were negative (controls). There were differences between the groups in the platelet count and the levels of D-dimer, procalcitonin, and glucose (p < 0.05). Chest CT scans categorized as COVID-19 Reporting and Data System 5 (CO-RADS 5) were more frequent in the COVID-19 group than in the control group (91.7% vs. 52.6%; p = 0.003). CO-RADS 5 remained an independent predictor of COVID-19 diagnosis in a second SARS-CoV-2 screening (p = 0.013; odds ratio = 7.0, 95% confidence interval 1.5–32.7). In conclusion, chest CT classified as CO-RADS 5 was an independent predictor of a positive second SARS-CoV-2 RT-PCR, increasing the odds of COVID-19 diagnosis by seven times. Based on our results, in hospitalized patients with a chest CT classified as CO-RADS 5, a second SARS-CoV-2 RT-PCR test should be mandatory when the first one is negative. This approach could increase SARS-CoV-2 detection up to 65% and could allow for isolation and treatment, thus improving the patient outcome and avoiding further contagion.

1. Introduction

Severe accurate respiratory syndrome coronavirus 2 (SARS-CoV-2) is an RNA virus belonging to the family of SARS-CoV and the Middle East respiratory syndrome coronavirus (MERS-CoV) [1]. SARS-CoV-2 was identified in January 2020 as the etiological agent of an atypical pneumonic epidemic that started in Wuhan, China [1]. This illness was later called coronavirus disease 2019 (COVID-19) and was declared a pandemic by the World Health Organization (WHO) in March 2020 [1]. Up to 26 December 2021, there had been 278 million cases and 5.4 million deaths [2,3]. In Mexico, up to 2 January 2022, there had been 3,990,587 confirmed cases and 299,544 accumulated deaths [4,5].
The SARS-CoV-2 genome encodes multiple non-structural proteins and four structural proteins: spike (S), envelope (E), membrane (M), and nucleocapsid (N). Infection starts with the transmission of aerosols with SARS-CoV-2 which binds to the epithelial nasal cells in the upper respiratory tract of the host [6]. SARS-CoV-2 binds to the cell membrane through the S protein [7] via angiotensin converting enzyme 2 (ACE2) and activates the protease TMPRSS2. After virus entry into the host cell, the viral genome is replicated via a replicase–protein complex encoded by the 20 kilobase (kb) replicase gene comprising two large open reading frames, designated ORF1a and ORF1b [3]. RNA-dependent RNA polymerase (RdRp) and helicase are encoded by ORF1b. The replicase complex includes both cellular proteins and up to 16 viral subunits [3]. After replication, this complex binds to the cell-host membrane and the genomic RNA is incorporated for the assembly of new virus particles, which are released via exocytosis to infect more cells [8]. The virus goes through replication and local propagation with infection of the ciliated cells in the airway. The immune response is limited in this phase, but individuals are highly infectious [6].
There has been a spectrum of COVID-19 severity reported in the literature [9]. While patients with mild COVID-19 can present fever, sore throat, cough without expectoration, general discomfort, nausea, vomiting, and abdominal pain, patients with moderate infection can present pneumonia with persistent fever and cough, but without hypoxemia [9]. Severe infections are characterized by the presence of hypoxemia with pneumonia; oxygen saturation < 92%; and a critical state that corresponds to acute respiratory distress syndrome, coagulation alterations, encephalopathy, cardiac failure, and acute kidney injury [6]. SARS-CoV-2 induces a multisystemic disease. The cytokine storm associated with COVID-19 involves an increase in proinflammatory factors, such as procalcitonin and cytokines [10,11]. The infection also causes a prothrombotic state, and, consequently, patients can present increased levels of reactive C protein, D-dimer, platelets, and fibrinogen [12]. Measuring the levels of these parameters could be used to predict disease severity [10].
COVID-19 diagnosis can be realized through molecular detection of SARS-CoV-2 antigens or RNA. The gold standard for SARS-CoV-2 detection is real-time reverse-transcription polymerase chain reaction (RT-PCR), which uses RNA extracted preferably from nasopharynx and oropharynx exudates [13]. In Mexico, an official national health network of Viral Respiratory Disease Monitoring Health Units is distributed throughout the country; the purpose is epidemiological surveillance. These health units have good laboratory facilities and qualified staff, and since the beginning of COVID-19 pandemic, they have been used to collect and report data about the disease [14]. All the institutions in the national network use the same standardized RT-PCR test based on the Berlin protocol, which detects the E and RdRp genes [15,16]. The E gene is used for screening to detect any beta-coronavirus associated with bats, while the RdRp gene is specific for SARS and SARS-like coronaviruses (including SARS-CoV-2) [15,16]. Other protocols have also been implemented, the most popular of which are to detect the N, E, or S genes and/or ORF1 of SARS-CoV-2 [15]. Nevertheless, a high number of false negatives for SARS-CoV-2 is one of the main challenges for the COVID-19 diagnosis. Indeed, while the accuracy of the RT-PCR method is 80–100%, the false-negative rate is highly variable, between 9.3% and 67% [3,17]. There could be several causes for the high number of false negatives: the method of detection, changes in the sample storage temperature or how long samples are stored, inadequate viral material, laboratory mistakes, and limitations in sample transportation [18].
Researchers have assessed diagnostic tools for COVID-19; the nasopharyngeal swab appears to have higher positivity rates than other samples, especially later in the course of the illness [19]. The evaluation of commercial RT-PCR kits for SARS-CoV-2 has allowed researchers to calculate a sensitivity between 82% and 100%, a specificity of 100% [20], and a precision near 59% [21]. Moreover, the use of alternative samples has been suggested to maintain appropriate precautions in patients suspected of having COVID-19 who had a negative RT-PCR result [22]. Of note, up to 93% of patients with an initial negative test but with a chest-computed axial tomography (CT) scan highly suggestive of COVID-19 had a positive second SARS-CoV-2 RT-PCR result, on average, 5.1 days after the first test [21], highlighting the necessity of a correct COVID-19 diagnosis that will have an impact on the prognosis, in the selection of the pharmacological therapy and clinical interventions, and, therefore, on the patient outcome [13]. In this study, we aimed to evaluate the potential false-negative results in hospitalized patients suspected of viral respiratory disease but with a negative previous SARS-CoV-2 RT-PCR. We also evaluated biochemical, clinical, and imaging variables that may increase the success of COVID-19 diagnosis in this group of patients.

2. Materials and Methods

2.1. Study Population

This case-control study was carried out from November 2020 to May 2021 and was approved by the Ethic and Investigation Committee of the General Hospital of Zacatecas Luz González Cosío (HGZ) in Zacatecas, Mexico (Protocol ID: 0227/2021/C). Patients who authorized their participation in this protocol by signing the informed consent were included in the study. The inclusion criteria were patients older than 18 years who met the hospitalization criteria according to standards established by the Ministry of Health of Zacatecas [23]. These patients were hospitalized in the area of adults with COVID-19 in the HGZ. Participants had clinical and/or radiological findings suggestive of COVID-19 and had had an initial negative SARS-CoV-2 RT-PCR test. Patients who did not agree to participate in the study and/or who had psychiatric illness were not included. All patients included in the study underwent a second nasopharynx and oropharynx swabs for RT-PCR test for SARS-CoV-2. Based on the result, they were divided into the COVID-19 group (positive SARS-CoV-2 RT-PCR) or the control group (negative SARS-CoV-2 RT-PCR). The COVID-19 group included 36 patients, and the control group included 19 participants.

2.2. Clinical and Biochemical Data at the Hospital Admission

The presence of comorbidities in each participant was obtained from their medical record. The clinical manifestations in these patients at the time of hospitalization considered for this study were the presence/absence of symptoms, such as non-productive cough, dyspnea, thoracic pain, fever, anosmia, dysgeusia, asthenia, arthralgias, diarrhea, headache, and nausea. Biochemical test results, including hemoglobin, platelet count, leucocytes, lymphocytes, neutrophils, prothrombin, activated thromboplastin time, glucose, creatinine, ureic nitrogen, direct bilirubin, and albumin, were obtained at the time of admission. Inflammatory markers considered for the study were fibrinogen, D-dimer, lactate dehydrogenase, ferritin, procalcitonin, C-reactive protein, troponin, and creatine kinase (Ck). The need for oxygen supplementation was also recorded.

2.3. Clinical Criteria Used to Classify of Patients According to COVID-19 Severity

For each patient, COVID-19 severity was established according to the Mexican Inter-Institutional consensus “Clinical guide for COVID-19 treatment in Mexico” created by the Mexican Government together with Mexican Health authorities [9]. According to this guideline, mild SARS-CoV-2 infection included patients with no evidence of pneumonia or hypoxia (oxygen saturation >94%); moderate infection was considered in cases with clinical evidence of pneumonia (oxygen saturation >90%); and severe infection included all cases with clinical evidence of pneumonia and one of the following conditions, namely respiratory frequency up to 30 breathes per minute, severe respiratory difficulty, and/or oxygen saturation <90%.

2.4. Chest CT Scan

At hospital admission, patients with suspicion of COVID-19 were subjected to chest CT as an auxiliary tool for the diagnosis. A radiologist evaluated CT scans by using the COVID-19 Reporting and Data System (CO-RADS). The scale is from 0 to 6, and each category represents the level of suspicion for pulmonary involvement in COVID-19. CO-RADS 0 was chosen if none of the categories from 1 to 5 could be assigned because scans were incomplete or of insufficient quality. CO-RADS 1 indicated findings with a very low level of suspicion (normal or non-infectious etiology). CO-RADS 2 indicated a low level of suspicion, with findings typical for infections other than COVID-19. CO-RADS 3 is an equivocal/unsure level of suspicion, representing features compatible with COVID-19, but also other diseases. CO-RADS 4 indicated a high level of suspicion for COVID-19. CO-RADS 5 was a very-high level of suspicion for pulmonary involvement, as is typical for COVID-19 findings; the mandatory features for CO-RADS 5 were ground-glass opacities with or without consolidations in lung regions close to visceral pleural surfaces, including the fissures and a multifocal bilateral distribution. Finally, CO-RADS 6 included those patients with a positive SARS-CoV-2 RT-PCR result.

2.5. SARS-CoV-2 RT-PCR Screening

At hospital admission, health workers identified patients with suspicion of COVID-19 based on clinical and/or radiological data, and a first RT-PCR was performed by using a nasopharynx or oropharynx swab. The first SARS-CoV-2 screening employed RT-PCR based on the Berlin protocol and was performed at the Health Public Laboratory of Zacatecas State. This laboratory is part of the National network of Viral Respiratory Disease Monitoring Health Units and is part of the Health Secretary of Zacatecas. If the first RT-PCR result was negative, a second nasopharynx swab was obtained and sent to the Molecular Medicine Laboratory from Academic Unit of Human Medicine and Health Sciences from the Autonomous University of Zacatecas Francisco García Salinas for viral screening. This laboratory has been authorized as a COVID-19 diagnosis laboratory by the Instituto de Diagnóstico y Referencia Epidemiológicos Dr. Manuel Martínez Báez, which is the institution of the Ministry of Health of Mexico in charge of diagnosis, control, referral, research, and technological development for the surveillance of epidemiological diseases [13]. Exudate samples were screened for this second time for SARS-CoV-2 with a one-step RT-PCR assay, using the TaqMan 2019nCoV assay (Thermo Fisher Scientific, Waltham, MA, USA) according to the manufacturer indications. This RT-PCR assay is based on detection of the SARS-CoV-2 genes N and S and the ORF1 region. Both laboratories that carried out the COVID-19 diagnosis for the protocol had 100% concordance in their results when they carried out a blinded independent COVID-19 diagnosis from the same panel of samples.

2.6. Statistical Analysis

The Shapiro–Wilk test was used to assess the normality of the variables. For normally distributed variables, Levene’s test was applied to check homoscedasticity. Student’s t-test was applied when the variances were homogenous. When the variances were not homogenous, Welch’s test was applied. For non-normally distributed variables, the Mann–Whitney–Wilcoxon test was applied for comparisons between study groups. Categorical variables were compared by using Fisher’s exact test or the chi-square test. The phi coefficient and Cramer’s V were used to measure the strength of association between a CO-RADS 5 chest CT and a positive second RT-PCR. Multivariate logistic regression was used to evaluate independent predictors of COVID-19 diagnosis during the second RT-PCR, with COVID-19 diagnosis as the dependent variable. Statistical analysis was performed with R Studio Cloud 4.0.3 (https://rstudio.cloud/project/2614648. Accessed on 6 January 2022), and p < 0.05 was considered statistically significant.

3. Results

During the period in which the study was carried out (November 2020 to May 2021), 709 patients were hospitalized with suspicion of respiratory viral infection and underwent a first SARS-CoV-2 screening. From these patients, 55 were negative, but they had clinical/radiological findings related to COVID-19 and therefore were included form the study. These patients underwent a second SARS-CoV-2 RT-PCR. Thirty-six (65.5%) of the participants were positive for SARS-CoV-2; these patients comprised the COVID-19 group. The remaining 19 (34.5%) patients were negative for SARS-CoV-2 and comprised the control group. Table 1 displays the characteristics of the general population and stratified into the COVID-19 and control groups.
The average age of the study population was 57.3 years; 31 (56.4%) patients were men. The presence of comorbidities at the time of inclusion in the study was as follows: 32.7% of patients had type 2 diabetes mellitus (T2DM), 40% had systemic arterial hypertension, and 34.5% presented obesity. At the time of admission, there were no differences between the study groups in age, sex, comorbidities, and symptoms (p > 0.05).
Considering the first SARS-CoV-2 screening and the beginning of symptoms, the mean COVID-19 testing time was 9.97 ± 5.10 days for the COVID-19 group and 10.84 ± 7.63 days for the control group (p = 0.94). The second SARS-CoV-2 screening was performed 13.72 ± 5.38 days after symptoms began for the COVID-19 group and 13.53 ± 7.43 days after symptoms began for the control group (p = 0.6). Considering the need for supplemental oxygen during hospitalization, 85.5% of patients used at least nasal cannulas with supplementary oxygen, up to 90.9% used a facemask, and 18.2% of patients required mechanical ventilation (Table 1). There were no differences in the type of supplemental oxygen used during the hospital stay between the COVID-19 and control groups (p > 0.05).
Table 2 displays the laboratory findings of the overall study population and for each study group. Considering the obtained data of hematic biometry, the platelet count differed between the COVID-19 and control groups (p = 0.01). The glucose levels were 167.58 ± 93.09 mg/dL and 124.16 ± 58.37 mg/dL in the COVID-19 and control groups, respectively (p = 0.03). The partial pressure of oxygen was lower in the COVID-19 group (68.50 ± 31.69 mmHg) than in the control group (81.42 ± 40.43 mmHg); this difference was significant (p = 0.04).
Regarding the biochemical parameters related to inflammation/prothrombotic events and compared with controls, D-dimer was slightly increased in the COVID-19 group, with an average value of 5.24 pg/mL (p = 0.003). There was a significant increase in procalcitonin of the control group (4.88 ± 14.26 pg/mL) compared with the COVID-19 group (0.84 ± 1.17 pg/mL) (p = 0.005). There were no differences in the other biochemical markers between the groups (p > 0.05).
Figure 1 and Figure 2 shows the results of the chest CT, evaluated by using the CO-RADS system. Of the study population, 78.18% of the patients presented CO-RADS 5. As expected, CO-RADS 5 findings were significantly more frequent in the COVID-19 group (91.66%) than in the control group (52.63%) (p = 0.003). In the COVID-19 group, only three patients had a CO-RADS of 1, 2, or 4 (Figure 2).
The phi coefficient and Cramer’s V were calculated to determine the strength of the association between reaching a specific CO-RADS score and having a positive second SARS-CoV-2 RT-PCR. The phi coefficient was 0.42, and Cramer’s V was 0.43. Considering the results of the second SARS-CoV-2 RT-PCR and chest CT classified as CO-RADS 5, 42 of 55 patients were correctly classified, a concordance of 76.36% between the methods.
After statistical modeling by multiple logistic regression and considering SARS-CoV-2 infection as the dependent variable, only radiological findings classified as CO-RADS 5 remained an independent predictor of COVID-19 for a second viral screening (odds ratio = 7.0, 95%; confidence interval, 1.5–32.7; p = 0.013).

4. Discussion

The timely and accurate diagnosis of COVID-19 in hospitalized patients is important to prevent the spread of the virus and to provide the patient with the correct treatment. Considering that there are reports of patients with a high clinical suspicion of COVID-19 but with negative SARS-CoV-2 RT-PCR, which demonstrates the presence of false negatives in COVID-19 molecular diagnostic methods, we evaluated potential false-negative results in hospitalized patients suspected of viral respiratory disease but with a negative previous SARS-CoV-2 RT-PCR result. We also evaluated variables that may increase the success of COVID-19 diagnosis in this group of patients.
In our study, the most frequent comorbidities in hospitalized patients were hypertension (40%), obesity (34.5%), and T2DM (32.7%). These findings are in agreement with other studies [24,25]. However, there were no differences between the COVID-19 and control groups in the proportions of these comorbidities, age, or sex. These data demonstrate that the presence of comorbidities and risk factors may worsen an infectious disease (including COVID-19), but they were not associated with a false-negative diagnostic-test result among the study population. Moreover, we did not observe differences in the proportions of symptoms between study groups. This result is not unexpected, considering that, in Mexico, a “COVID-19-suspicious patient” needs to fulfil the criteria of the operational definition for respiratory infection [26], and our participants fit this definition. The most prevalent symptoms were dyspnea, fever, and asthenia. Of note, even though there was no significant difference between the COVID-19 and control group, no patient with two negative SARS-CoV-2 RT-PCR results presented anosmia, which has been described as a characteristic symptom of COVID-19, mainly in mild illness or illness with few symptoms [27].
Our univariant analysis of biochemical parameters showed that both the platelet count and D-dimer levels were higher in the COVID-19 group compared with the control group. The increased platelet number and D-dimer levels correlate with COVID-19-associated coagulopathy, which includes platelet activation and leads to increased inflammation (involving monocytes, dendritic cells, neutrophils, and activated T cells, and proteases of coagulation) [28]. D-dimer, a product of fibrin degradation, correlates with the activity of tissue plasminogen activator (t-PA), plasmin, and other fibrinolytic pathways [29]. These results indicate a relationship between proinflammatory conditions of patients that is caused by COVID-19, and it is been associated with the gravity of the infection [28,29]. Procalcitonin is a glycoprotein precursor of calcitonin; it does not have hormonal activity, and its levels are usually undetectable, but it can increase with bacterial infections; hence, it has been proposed as a biomarker to distinguish between bacterial and viral infections [11]. In our study, the COVID-19 group presented lower procalcitonin levels compared with the control group; this could indicate the infectious etiology in the control group. The COVID-19 group showed a higher initial level of serum glucose and a decrease in the oxygen partial pressure compared with the control group. It is important to note that 32.7% of the patients had T2DM, but up to 78% of those patients had an altered glucose value, perhaps due to stress from the illness per se and possibly related to increased metabolism for viral replication. Compared with the control group, the oxygen partial pressure was lower in the COVID-19 group. This phenomenon may be explained because, in moderate and severe COVID-19, hypoxemic respiratory insufficiency due to diffuse alveolar damage could lead to acute respiratory syndrome, as is consistent with the Berlin criteria [30].
In our study, a positive second SARS-CoV-2 RT-PCR result after a negative first test improved the COVID-19 diagnosis by 65%. In this context, the chest CT at the time of hospital admission classified as CO-RADS 5 was a predictive factor for a positive second SARS-CoV-2 RT-PCR result. There was a strong correlation between these variables, with Cramer’s V of 0.43 and a phi correlation coefficient of 0.42. A chest CT result classified as CO-RADS-5 increased the odds of obtaining a positive result from a second SARS-CoV-2 RT-PCR by seven times in the studied population. The relationship between CT findings and a positive result in the second SARS-CoV-2 RT-PCR had already been addressed [31,32,33,34]. As an example Xie et al. evaluated five patients with a negative RT-PCR result but with a high suspicion of SARS-CoV-2 infection with the aim of describing radiologic findings based on an initial CT scan. Three of their patients who had a chest CT scan highly compatible with COVID-19 pneumonia and a negative first SARS-CoV-2 RT-PCR result were diagnosed with COVID-19 after a second screening carried out 2–8 days after the first screening [31]. In contrast to our study—where we used the CO-RADS scale—those authors proposed a scale with a maximum of 24 points, considering specific areas and the severity of the damage. Besides evaluating findings, the CO-RADS scale [35] correlates with the severity of the infection, has high interobserver correlation, and the false-positive rate for CO-RADS 5 has been reported to be as low as 0–3%, making chest CT a great tool in COVID-19 diagnostics after a negative initial RT-PCR result. Based on our data, CT findings compatible with COVID-19 in combination with a second SARS-CoV-2 RT-PCR that is positive could increase the percentage of COVID-19 diagnosis by up to 65.5%. In our study, despite the differences in biochemical variables between the COVID-19 and control groups—platelet count and D-dimer, procalcitonin, and/or glucose levels—they did not have an independent predictive value for COVID-19 diagnosis if a patient underwent a second SARS-CoV-2 RT-PCR after a false-negative result from the first RT-PCR. After progress in vaccination coverage and the changes in genetic variability of the virus, most of the COVID-19 cases are mild. Therefore, the use of imaging results remains as a very useful tool in moderate, severe, or unclear cases.
Considering the 709 patients hospitalized with suspicion of respiratory viral infection who underwent a first SARS-CoV-2 screening during the period in which the study was carried out, the false-negative rate of RT-PCR was 5%. A false-negative result could have several causes, including the sampling technique, sample storage, sample transportation and processing, and the correct time of sampling relative to the beginning of the symptoms. Sensitivity and specificity can be influenced by the amplification protocols and the detected target. A false-negative result during the first screening may also be because the screening protocol includes only one SARS-CoV-2 gene (RdRp, according to the guidelines of the Mexican health institutions). The second SARS-CoV-2 RT-PCR is more specific for the virus—the target genes are N and S, as well as the ORF1 region of SARS-CoV-2—and this difference may increase the accuracy of detection. Another factor to consider is that we selected our population based on the criteria for the severity of the infection, because, in Mexico, only patients with moderate, severe, and critical illness are hospitalized. Hence, the higher replication and viral count are probably at the bottom of the airway, so the sampling location might not have contained the minimum viral quantity required for detection. Even though RT-PCR kits are able to detect each SARS-CoV-2 variant, the Omicron variant has demonstrated significantly lower replication in lower airway in comparison to the Delta variant, which was the most prevalent SARS-CoV-2 variant in Mexico at the time the study was realized [36]. The average 3-day difference between the first and second SARS-CoV-2 test is due to time required to receive results from the first laboratory. This delay may also influence the differences in successful SARS-CoV-2 detection.

5. Conclusions

We showed that, in hospitalized patients with a negative first SARS-CoV-2 RT-PCR result but with clinical findings related to COVID-19, the platelet count; glucose, D-dimer, and procalcitonin levels; and/or a chest CT classified as CO-RADS 5 differed between the COVID-19 and control groups. However, only the chest CT classified as CO-RADS 5 was an independent predictor of a positive second SARS-CoV-2 RT-PCR, increasing the odds of COVID-19 diagnosis by seven times. Based on our results, in hospitalized patients with a chest CT classified as CO-RADS 5, a second SARS-CoV-2 RT-PCR test should be mandatory when the first one is negative. This approach could increase SARS-CoV-2 detection by up to 65.5% and could allow for the isolation of these patients and provision of correct treatment, improving the patient outcome and avoiding further contagion.

Author Contributions

Conceptualization, D.C.-G. and M.L.M.-F.; methodology, M.L.M.-F., C.G.-F., D.C.-G., S.C.-M., E.F.G.-V., J.Y.C.-L., J.M.-R. and I.G.-V.; formal analysis, M.L.M.-F. and C.G.-F.; investigation, S.G.D., M.R.R.-P., J.M.-R. and M.E.C.-L.; resources, M.L.M.-F.; data curation, M.L.M.-F. and C.G.-F.; writing—original draft preparation, M.L.M.-F., C.G.-F. and S.G.D.; writing—review and editing, M.L.M.-F., C.G.-F., S.G.D. and I.G.-V.; visualization, M.L.M.-F., C.G.-F. and M.R.R.-P.; supervision, M.L.M.-F. and I.G.-V.; project administration, M.L.M.-F.; funding acquisition, M.L.M.-F. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded in part by the Programa de Apoyos para el Fortalecimiento de Capacidades para el Diagnóstico de COVID-19-CONACYT. Grant ID: 314340 (A12 xii. 2020) to Margarita L Martinez-Fierro. This study did not receive additional external founding.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Ethic and Investigation Committee of the Hospital General de Zacatecas “Luz González Cosío” (Protocol ID: 0227/2021/C).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the patient(s) to publish this paper.

Data Availability Statement

Data that support the findings of this study are available from the corresponding authors (M.L.M.-F. and I.G.-V.) upon reasonable request.

Acknowledgments

The authors thank the clinical and laboratory staff of the Hospital General de Zacatecas Luz González Cosío for their support in conducting this study.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Pascarella, G.; Strumia, A.; Piliego, C.; Bruno, F.; Del Buono, R.; Costa, F.; Scarlata, S.; Agro, F.E. COVID-19 diagnosis and management: A comprehensive review. J. Intern. Med. 2020, 288, 192–206. [Google Scholar] [CrossRef] [PubMed]
  2. World Health Organization. Weekly Epidemiological Update on COVID-19. 28 December 2021. Available online: https://www.who.int/publications/m/item/weekly-epidemiological-update-on-covid-19---28-december-2021 (accessed on 1 December 2021).
  3. Rahimi, A.; Mirzazadeh, A.; Tavakolpour, S. Genetics and genomics of SARS-CoV-2: A review of the literature with the special focus on genetic diversity and SARS-CoV-2 genome detection. Genomics 2021, 113, 1221–1232. [Google Scholar] [CrossRef]
  4. Salud, S.D. Datos Abiertos Dirección General de Epidemiología. 2022. Available online: http://www.gob.mx/salud/documentos/datos-abiertos-152127 (accessed on 1 January 2022).
  5. COVID-19 Tablero México. 2020. Available online: https://datos.covid-19.conacyt.mx/ (accessed on 1 January 2022).
  6. Parasher, A. COVID-19: Current understanding of its Pathophysiology, Clinical presentation and Treatment. Postgrad. Med. J. 2021, 97, 312–320. [Google Scholar] [CrossRef]
  7. Kumar, M.; Al Khodor, S. Pathophysiology and treatment strategies for COVID-19. J. Transl. Med. 2020, 18, 353. [Google Scholar] [CrossRef] [PubMed]
  8. Mousavizadeh, L.; Ghasemi, S. Genotype and phenotype of COVID-19: Their roles in pathogenesis. J. Microbiol. Immunol. Infect. 2021, 54, 159–163. [Google Scholar] [CrossRef] [PubMed]
  9. Gobierno de México. Guía Clínica para el Tratamiento de la COVID-19 en Mexico. Consenso Interinstitucional. 2021. Available online: https://coronavirus.gob.mx/wp-content/uploads/2021/08/GuiaTx_COVID19_ConsensoInterinstitucional_2021.08.03.pdf (accessed on 1 January 2022).
  10. Ismail, A.A. SARS-CoV-2 (Covid-19): A short update on molecular biochemistry, pathology, diagnosis and therapeutic strategies. Ann. Clin. Biochem. 2022, 59, 59–64. [Google Scholar] [CrossRef] [PubMed]
  11. Liu, F.; Li, L.; Xu, M.; Wu, J.; Luo, D.; Zhu, Y.; Li, B.; Song, X.; Zhou, X. Prognostic value of interleukin-6, C-reactive protein, and procalcitonin in patients with COVID-19. J. Clin. Virol. Off. Publ. Pan Am. Soc. Clin. Virol. 2020, 127, 104370. [Google Scholar] [CrossRef] [PubMed]
  12. Lazzaroni, M.G.; Piantoni, S.; Masneri, S.; Garrafa, E.; Martini, G.; Tincani, A.; Andreoli, L.; Franceschini, F. Coagulation dysfunction in COVID-19: The interplay between inflammation, viral infection and the coagulation system. Blood Rev. 2021, 46, 100745. [Google Scholar] [CrossRef] [PubMed]
  13. Martinez-Fierro, M.L.; Rios-Jasso, J.; Garza-Veloz, I.; Reyes-Veyna, L.; Cerda-Luna, R.M.; Duque-Jara, I.; Galvan-Jimenez, M.; Ramirez-Hernandez, L.A.; Morales-Esquivel, A.; Ortiz-Castro, Y.; et al. The role of close contacts of COVID-19 patients in the SARS-CoV-2 transmission: An emphasis on the percentage of nonevaluated positivity in Mexico. Am. J. Infect. Control 2021, 49, 15–20. [Google Scholar] [CrossRef] [PubMed]
  14. Martinez-Fierro, M.L.; Diaz-Lozano, M.; Alvarez-Zuniga, C.; Ramirez-Hernandez, L.A.; Araujo-Espino, R.; Trejo-Ortiz, P.M.; Mollinedo-Montano, F.E.; Ortiz-Castro, Y.; Vazquez-Reyes, S.; Velasco-Elizondo, P.; et al. Population-Based COVID-19 Screening in Mexico: Assessment of Symptoms and Their Weighting in Predicting SARS-CoV-2 Infection. Medicina 2021, 57, 363. [Google Scholar] [CrossRef] [PubMed]
  15. Parra-Ortega, I.; Vilchis-Ordonez, A.; Lopez-Martinez, B.; Angeles-Floriano, T. Analytical recommendations for SARS-CoV-2 identification by RT-PCR in pediatric patients. Bol. Med. Del Hosp. Infant. De Mex. 2021, 78, 171–180. [Google Scholar] [CrossRef] [PubMed]
  16. Corman Victor, B.T.; Brünink, S.; Drosten, C.; Landt, O.; Koopmans, M.; Zambon, M. Diagnostic Detection of Wuhan Coronavirus 2019 by Real-Time RT-PCR; WHO: Geneva, Switzerland, 2020; Volume 12. [Google Scholar]
  17. Kanji, J.N.; Zelyas, N.; MacDonald, C.; Pabbaraju, K.; Khan, M.N.; Prasad, A.; Hu, J.; Diggle, M.; Berenger, B.M.; Tipples, G. False negative rate of COVID-19 PCR testing: A discordant testing analysis. Virol. J. 2021, 18, 13. [Google Scholar] [CrossRef] [PubMed]
  18. Ahmed, W.; Simpson, S.L.; Bertsch, P.M.; Bibby, K.; Bivins, A.; Blackall, L.L.; Bofill-Mas, S.; Bosch, A.; Brandao, J.; Choi, P.M.; et al. Minimizing errors in RT-PCR detection and quantification of SARS-CoV-2 RNA for wastewater surveillance. Sci. Total Environ. 2022, 805, 149877. [Google Scholar] [CrossRef]
  19. Martinez, R.M. Clinical Samples for SARS-CoV-2 Detection: Review of the Early Literature. Clin. Microbiol. Newsl. 2020, 42, 121–127. [Google Scholar] [CrossRef]
  20. Shen, L.; Cui, S.; Zhang, D.; Lin, C.; Chen, L.; Wang, Q. Comparison of four commercial RT-PCR diagnostic kits for COVID-19 in China. J. Clin. Lab. Anal. 2021, 35, e23605. [Google Scholar] [CrossRef]
  21. Lippi, G.; Simundic, A.M.; Plebani, M. Potential preanalytical and analytical vulnerabilities in the laboratory diagnosis of coronavirus disease 2019 (COVID-19). Clin. Chem. Lab. Med. 2020, 58, 1070–1076. [Google Scholar] [CrossRef] [Green Version]
  22. Wiseman, J.; D’Amico, T.A.; Zawadzka, S.; Anyimadu, H. False negative SARS-CoV-2 PCR—A case report and literature review. Respir. Med. Case Rep. 2020, 31, 101140. [Google Scholar] [CrossRef]
  23. Gobierno de México. Algoritmos Interinos para la Atención del COVID-19. 2020. Available online: http://educacionensalud.imss.gob.mx/es/system/files/Algoritmos_interinos_COVID19_CTEC.pdf (accessed on 1 January 2022).
  24. Guan, W.J.; Ni, Z.Y.; Hu, Y.; Liang, W.H.; Ou, C.Q.; He, J.X.; Liu, L.; Shan, H.; Lei, C.L.; Hui, D.S.C.; et al. Clinical Characteristics of Coronavirus Disease 2019 in China. N. Engl. J. Med. 2020, 382, 1708–1720. [Google Scholar] [CrossRef] [PubMed]
  25. Rodilla, E.; Saura, A.; Jimenez, I.; Mendizabal, A.; Pineda-Cantero, A.; Lorenzo-Hernandez, E.; Fidalgo-Montero, M.D.P.; Lopez-Cuervo, J.F.; Gil-Sanchez, R.; Rabadan-Pejenaute, E.; et al. Association of Hypertension with All-Cause Mortality among Hospitalized Patients with COVID-19. J. Clin. Med. 2020, 9, 3136. [Google Scholar] [CrossRef] [PubMed]
  26. Salud, S.d. Lineamientos para la Atención de Pacientes por COVID-19. 2020. Available online: http://cvoed.imss.gob.mx/wp-content/uploads/2020/02/Linemaineto-cl%C3%Adnico-COVID-19-CCINSHAE-14feb2020.pdf.pdf.pdf.pdf (accessed on 1 January 2022).
  27. Mao, L.; Wang, M.; Chen, S.; He, Q.; Chang, J.; Hong, C.; Zhou, Y.; Wang, D.; Li, Y.; Jin, H.; et al. Anosmia and Ageusia: Common Findings in COVID-19 Patients. Laryngoscope 2020, 130, 1787. [Google Scholar] [CrossRef] [Green Version]
  28. Le Joncour, A.; Biard, L.; Vautier, M.; Bugaut, H.; Mekinian, A.; Maalouf, G.; Vieira, M.; Marcelin, A.G.; Rosenzwajg, M.; Klatzmann, D.; et al. Neutrophil-Platelet and Monocyte-Platelet Aggregates in COVID-19 Patients. Thromb. Haemost. 2020, 120, 1733–1735. [Google Scholar] [CrossRef] [PubMed]
  29. Ghebrehiwet, B.; Peerschke, E.I. Complement and coagulation: Key triggers of COVID-19-induced multiorgan pathology. J. Clin. Investig. 2020, 130, 5674–5676. [Google Scholar] [CrossRef] [PubMed]
  30. Gibson, P.G.; Qin, L.; Puah, S.H. COVID-19 acute respiratory distress syndrome (ARDS): Clinical features and differences from typical pre-COVID-19 ARDS. Med. J. Aust. 2020, 213, 54–56 e51. [Google Scholar] [CrossRef] [PubMed]
  31. Xie, X.; Zhong, Z.; Zhao, W.; Zheng, C.; Wang, F.; Liu, J. Chest CT for Typical 2019-nCoV Pneumonia: Relationship to Negative RT-PCR Testing. Radiology 2020, 296, E41–E45. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  32. Sanchez-Oro, R.; Torres Nuez, J.; Martinez-Sanz, G. Radiological findings for diagnosis of SARS-CoV-2 pneumonia (COVID-19). Med. Clin. 2020, 155, 36–40. [Google Scholar] [CrossRef]
  33. Ye, Z.; Zhang, Y.; Wang, Y.; Huang, Z.; Song, B. Chest CT manifestations of new coronavirus disease 2019 (COVID-19): A pictorial review. Eur. Radiol. 2020, 30, 4381–4389. [Google Scholar] [CrossRef] [Green Version]
  34. Wong, H.Y.F.; Lam, H.Y.S.; Fong, A.H.; Leung, S.T.; Chin, T.W.; Lo, C.S.Y.; Lui, M.M.; Lee, J.C.Y.; Chiu, K.W.; Chung, T.W.; et al. Frequency and Distribution of Chest Radiographic Findings in Patients Positive for COVID-19. Radiology 2020, 296, E72–E78. [Google Scholar] [CrossRef] [Green Version]
  35. Prokop, M.; van Everdingen, W.; van Rees Vellinga, T.; Quarles van Ufford, H.; Stoger, L.; Beenen, L.; Geurts, B.; Gietema, H.; Krdzalic, J.; Schaefer-Prokop, C.; et al. CO-RADS: A Categorical CT Assessment Scheme for Patients Suspected of Having COVID-19-Definition and Evaluation. Radiology 2020, 296, E97–E104. [Google Scholar] [CrossRef]
  36. Gupta, R. SARS-CoV-2 Omicron spike mediated immune escape and tropism shift. Res. Sq. 2022. [Google Scholar] [CrossRef]
Figure 1. Representative chest-computed axial tomography (CT) slices. Each patient was screened based on anatomical abnormalities observed on chest CT scans. (A) Chest CT findings categorized as CO-RADS 5 and (B) chest CT findings classified as CO-RADS 3. CO-RADS, COVID-19 Reporting and Data System.
Figure 1. Representative chest-computed axial tomography (CT) slices. Each patient was screened based on anatomical abnormalities observed on chest CT scans. (A) Chest CT findings categorized as CO-RADS 5 and (B) chest CT findings classified as CO-RADS 3. CO-RADS, COVID-19 Reporting and Data System.
Diagnostics 12 00687 g001
Figure 2. Chest-computed axial tomography (CT) findings according to the COVID-19 Reporting and Data System (CO-RADS) classification. The study population with a negative first SARS-CoV-2 RT-PCR result (n = 55) underwent a second SARS-CoV-2 RT-PCR. According to this result, they were classified into the COVID-19 group (positive RT-PCR result) or the control group (negative RT-PCR result). At the time of admission, a chest CT was carried out, and the scans were categorized by using the CO-RADS system. The graph shows the number of patients in each group based on their CO-RADS score. Most of the patients in the COVID-19 group were classified as CO-RADS 5.
Figure 2. Chest-computed axial tomography (CT) findings according to the COVID-19 Reporting and Data System (CO-RADS) classification. The study population with a negative first SARS-CoV-2 RT-PCR result (n = 55) underwent a second SARS-CoV-2 RT-PCR. According to this result, they were classified into the COVID-19 group (positive RT-PCR result) or the control group (negative RT-PCR result). At the time of admission, a chest CT was carried out, and the scans were categorized by using the CO-RADS system. The graph shows the number of patients in each group based on their CO-RADS score. Most of the patients in the COVID-19 group were classified as CO-RADS 5.
Diagnostics 12 00687 g002
Table 1. General data of the study population classified into the COVID-19 and control groups according to the second SARS-CoV-2 RT-PCR.
Table 1. General data of the study population classified into the COVID-19 and control groups according to the second SARS-CoV-2 RT-PCR.
VariableGeneral Population
(n = 55)
Study Groupp-Value
COVID-19 (n = 36)Controls (n = 19)
Age (years)57.3 ± 16.7757.94 ± 15.9456.11 ± 18.640.69
Sex female 24 (43.6)14 (38.8)10 (52.6)0.39
Comorbidities (n, %)
  Hypertension 22 (40)12 (33.3)10 (52.6)0.17
  Diabetes 18 (32.72)13 (36.1)5 (26.3)0.46
  Obesity 19 (34.54)13 (36.1)6 (66.6)0.74
  Other comorbidity 26 (47.27)15 (41.7)11 (57.8)0.25
Symptoms * (n, %)
  Cough 33 (60)21 (58)12 (63)0.73
  Dyspnoea53 (96.36)36 (100)17 (89)0.12
  Chest pain14 (25.45)10 (27.7)4 (21)0.75
  Fever38 (69.09)26 (72.2)12 (63.1)0.49
  Anosmia4 (7.27)4 (11.1)00.29
  Dysgeusia5 (9.09)4 (11)1 (5.2)0.65
  Asthenia34 (61.1)25 (69.4)9 (47.3)0.11
  Myalgias16 (29.09)11 (39.5)5 (26.3)0.74
  Arthralgias16 (29.09)13 (36.1)3 (15.7)0.12
  Diarrhea8 (14.54)4 (11)4 (21)0.43
  Headache14 (25.45)11 (30.5)3 (15.7)0.33
  Sickness12 (21.81)9 (25)3 (15.7)0.51
Supplementary oxygen ** (n, %)
  Nasal cannula47 (85.45)30 (83.3)17 (89.5)0.7
  Facemask50 (90.90)33 (91.7) 17 (89.5)0.78
  Mechanic Ventilation10 (18.18)6 (16.7)4 (21.1)0.72
COVID-19 testing moment (days)
  First COVID-19 test 10.27 ± 36.469.97 ± 5.1010.84 ± 7.630.94
  Second COVID-19 test13.65 ± 6.0913.72 ± 5.3813.53 ± 7.430.6
Cq value (mean ± SD)
  ORF1 gene -29.24 ± 4.44--
  N gene-28.52 ± 5.08--
  S gene-29.17 ± 4.31--
* The reported symptoms are from the time of the hospital admission. ** Supplemental oxygen needed during hospitalization.
Table 2. Laboratory findings of the study groups at the time of hospital admission.
Table 2. Laboratory findings of the study groups at the time of hospital admission.
Laboratory FindingGeneral Population (n = 55)Study Groupp-Value
COVID-19 (n = 36)Controls (n = 19)
Hematic biometry (mean ± SD)
  Hemoglobin (g/dL)13.54 ± 3.1113.97 ± 2.7312.73 ± 3.690.20
  Platelets (103/dL)267.4 ± 117.7296.5 ± 105.8212.4 ± 122.10.01 *
  Leucocytes (counts/dL)10,733 ± 485310,797 ± 460810,611 ± 54180.85
  Lymphocytes (counts/dL)1069 ± 6201147 ± 711922 ± 3670.29
  Neutrophils (counts/dL)9261 ± 44629347 ± 41759098 ± 50790.85
Blood chemistry (mean ± SD)
  Prothrombin time (s)16.08 ± 2.7715.77 ± 2.416.63 ± 3.350.33
  Partial thromboplastin time (s)36.80 ± 11.1036.1 ± 11.638.06 ± 10.360.26
  Glucose (mg/dL) 152.58 ± 84.77167.58 ± 93.09124.16 ± 58.370.03 *
  Creatinine (mg/dL) 1.91 ± 4.671.99 ± 5.531.76 ± 2.440.42
  Blood urea nitrogen (mg/dL) 28.15 ± 22.7127.97 ± 25.0928.511± 7.990.37
  Direct bilirrubin (mg/dL) 0.10 ± 0.790.34 ± 0.230.82 ±1.290.15
  Albumin (mg/dL) 3.10 ± 0.483.12 ± 0.473.07 ± 0.510.75
Arterial blood gas test (mean ± SD)
  pH 7.42 ± 0.087.43 ± 0.067.41 ± 0.100.61
  pO2 (mmHg)72.96 ± 35.1368.50 ± 31.6981.42 ± 40.430.04 *
  pCO2 (mmHg)33.13 ± 8.9532.39 ±7.4734.53 ± 11.340.51
  HCO3 (mmol/L)22.22 ± 3.9622.29 ± 4.3022.08 ± 3.30.84
Inflammatory markers (mean ± SD)
  Fibrinogen (mg/dL)573.97 ± 193.28615.91± 186.34512.47 ± 192.770.84
  D-dimer (pg/mL)4.07 ± 6.215.24 ± 7.423.23 ± 5.280.003 *
  Lactate dehydrogenase (mg/dL)639.03 ± 402.47620.79 ± 405.07675.53 ± 407.020.89
  Ferritin (ng/mL)765.93 ± 744.26692.85 ± 576.70912.52 ± 1019.160.41
  Procalcitonin (mg/dL)2.41 ± 8.960.84 ± 1.174.88 ± 14.260.005 *
  C-reactive protein16.55 ± 11.6216.26 ± 9.3217.05 ± 15.060.73
  Troponin (ng/mL)1.03 ± 4.150.31± 0.441.81 ± 5.980.42
  Creatine kinase (mg/dL)413.08 ± 1472567.41 ± 1922.82200.9 ± 295.20.56
The data are presented as the mean ± standard deviation; * p < 0.05. pH, hydrogen potential; pO2, partial pressure of oxygen; pCO2, partial pressure of carbon dioxide; HCO3, bicarbonate.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Martinez-Fierro, M.L.; González-Fuentes, C.; Cid-Guerrero, D.; González Delgado, S.; Carrillo-Martínez, S.; Gutierrez-Vela, E.F.; Calzada-Luévano, J.Y.; Rocha-Pizaña, M.R.; Martínez-Rendón, J.; Castañeda-López, M.E.; et al. Radiological Findings Increased the Successful of COVID-19 Diagnosis in Hospitalized Patients Suspected of Respiratory Viral Infection but with a Negative First SARS-CoV-2 RT-PCR Result. Diagnostics 2022, 12, 687. https://doi.org/10.3390/diagnostics12030687

AMA Style

Martinez-Fierro ML, González-Fuentes C, Cid-Guerrero D, González Delgado S, Carrillo-Martínez S, Gutierrez-Vela EF, Calzada-Luévano JY, Rocha-Pizaña MR, Martínez-Rendón J, Castañeda-López ME, et al. Radiological Findings Increased the Successful of COVID-19 Diagnosis in Hospitalized Patients Suspected of Respiratory Viral Infection but with a Negative First SARS-CoV-2 RT-PCR Result. Diagnostics. 2022; 12(3):687. https://doi.org/10.3390/diagnostics12030687

Chicago/Turabian Style

Martinez-Fierro, Margarita L, Carolina González-Fuentes, Dagoberto Cid-Guerrero, Samantha González Delgado, Santiago Carrillo-Martínez, Edgar Fernando Gutierrez-Vela, Juan Yadid Calzada-Luévano, Maria R. Rocha-Pizaña, Jacqueline Martínez-Rendón, Maria E. Castañeda-López, and et al. 2022. "Radiological Findings Increased the Successful of COVID-19 Diagnosis in Hospitalized Patients Suspected of Respiratory Viral Infection but with a Negative First SARS-CoV-2 RT-PCR Result" Diagnostics 12, no. 3: 687. https://doi.org/10.3390/diagnostics12030687

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop