Next Article in Journal
Low-Loss and Broadband Silicon Photonic 3-dB Power Splitter with Enhanced Coupling of Shallow-Etched Rib Waveguides
Next Article in Special Issue
Detectability of Osseous Lesions with a Pre-Programmed Low-Dose Protocol for Cone-Beam Computed Tomography
Previous Article in Journal
Experimental and Numerical Study on Dynamics of Two Footbridges with Different Shapes of Girders
Previous Article in Special Issue
Alterations in Surface Roughness and Chemical Characteristics of Sandblasted and Acid-Etched Titanium Implants after Irradiation with Different Diode Lasers
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Applied Sciences
Volume 10
Issue 13
10.3390/app10134506
applsci-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Review

Diagnostic Performance of Serological Assays in the Detection of SARS-CoV-2: A Review

by
Francesco Carinci
1,†,
Giulia Moreo
2,†,
Luisa Limongelli
3,
Tiziano Testori
4,* and
Dorina Lauritano
2
1
Department of Morphology, Surgery and Experimental Medicine, University of Ferrara, 44121 Ferrara, Italy
2
Department of Medicine and Surgery, Centre of Neuroscience of Milan, University of Milano-Bicocca, 20126 Milan, Italy
3
Interdisciplinary Department of Medicine, University of Bari, 70121 Bari, Italy
4
Department of Periodontics and Oral medicine, University of Michigan, School of Dentistry, Anne Arbor, MI 48109-1078, USA
*
Author to whom correspondence should be addressed.
†
Co-first authorship.
Appl. Sci. 2020, 10(13), 4506; https://doi.org/10.3390/app10134506
Submission received: 31 May 2020 / Revised: 24 June 2020 / Accepted: 26 June 2020 / Published: 29 June 2020
(This article belongs to the Special Issue Applied Sciences in Dentistry)
Browse Figure
Versions Notes

Abstract

:

Featured Application

This review provided a thorough analysis of the existing serological tests for the detection of SARS-CoV-2 infection: sensitivity, specificity, positive and negative predictive values of each assay were reported, percentages of IgM/IgG positive patients among cases and controls were shown. This paper may help clinicians choosing the most appropriate serological test for the diagnosis of COVID-19.

Abstract

Introduction: The gold-standard method for diagnosis of the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2 or COVID-19) foresees the examination of respiratory tract swabs by real-time reverse-transcription polymerase chain reaction (rRT-PCR). Another group of diagnostic tests, developed to overcome the limitations of RT-PCR, includes the serological assays, which have the purpose of detecting the antibody response to SARS-CoV-2 infection (IgM and IgG titers). The aim of this review was to establish the diagnostic capability of the existing serological tests in the detection of SARS-CoV-2 infection. Materials and Methods: Electronic research was conducted in PubMed, Scopus, Science Direct and Cochrane Library, and only 10 articles, testing 10 different types of serological assays, met the inclusion criteria and were consequently submitted to quality assessment and data extraction. Quantitative data about the sensitivity, specificity, positive/negative predictive value and IgM/IgG titer provided by each antibody test were reported in our review. Results: Almost all the serological tests used in the included items were recorded to ensure high sensitivity and specificity, identifying the presence of IgM and IgG antibodies against SARS-CoV-2 in patients with certain COVID-19 diagnosis (confirmed by RT-PCR) and in participants with suspected infection (SARS-CoV-2 clinical diagnosis and/or RT-PCR negative subjects). Conclusions: Serological tests may represent reliable diagnostic tools in the detection of SARS-CoV-2 infection, and they could be implemented complementary to real-time RT-PCR.

1. Introduction

The current global pandemic caused by the novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), whose initial outbreak was detected in December 2019 in Wuhan (China), represents a real threat to international health [1,2,3]. This new pathogen, which is an enveloped, non-segmented, positive sense RNA virus, belongs to the Coronaviridae family, as well as the severe acute respiratory syndrome coronavirus 1 (SARS-CoV-1) and Middle East respiratory syndrome coronavirus (MERS-CoV) [4,5]. Fever, cough expectoration, myalgia, fatigue, dyspnea and gastrointestinal symptoms are the most frequent clinical manifestations induced by SARS-CoV-2 [6,7,8]. In some cases, the progression of the illness is relatively asymptomatic, while, in other cases, pneumonia, acute respiratory distress syndrome (ARDS), sepsis or septic shock may occur [9,10]. The main SARS-CoV-2 transmission routes are represented by respiratory droplets (within a long distance of 2 m), generated by infected subjects during coughing and sneezing and by contact with contaminated surfaces [11,12,13,14]. No specific antiviral therapy against SARS-CoV-2 has been introduced yet: treatment protocols include broad-spectrum antiviral drugs (remdesivir, lopinavir, ritonavir, favipiravir), which should be administrated in the early stage of the infection, and antimalarial/autoimmune disease drugs (chloroquine, hydroxychloroquine) [15,16,17]. In order to limit the spread of the infection among the population, isolating the infected persons, adequate diagnostic strategies should be implemented. The most commonly used method to detect SARS-CoV-2 foresees the analysis of nasopharyngeal and throat swabs by real-time reverse-transcription polymerase chain reaction (rRT-PCR) test [18,19,20]. A complementary diagnostic procedure is represented by the chest computerized tomography (TC) tool, which may allow the detection of the virus in rRT-PCR false-negative cases. Even if real-time RT-PCR plays a crucial role in COVID-19 diagnosis, it presents several limitations: it may not be able to identify the virus in the early stage of the infection, and it requires a long time to obtain the results, which may be influenced by external factors such as sampling operation method and the performance of detection kits and nucleic acid extraction from clinical material [21,22]. Another group of diagnostic tests used for SARS-CoV-2 identification is that of serological assays. Several different antibody tests have been developed, which differ depending on the targeted viral antigen (for example, nucleoprotein or spike protein); the most common SARS-CoV-2 biomarkers tested serological assays of IgM and IgG antibodies [23]. The purpose of this type of test may be the identification of PCR-negative cases and asymptomatic patients or the evaluation of vaccine response during clinical trials [24]. According to a study by Li-Xin Xie [25], in most cases IgM antibodies are present 3–5 days after the onset of symptoms, while IgG titer is higher in the recovery phase than in the acute one. Patel et al. [26] reported that seroconversion in infected patients may occur between 7 and 11 days after the exposure to the virus. Due to the delayed appearance of IgM and IgG antibodies, serological tests are unreliable in the acute phase of the infection. The Infectious Diseases Society of America stated that serological tests still remain clinically unverified and suggested that serology results alone should not be used for diagnostic decisions [27]. Although no clear evidence about the duration of immunity protection against SARS-CoV-2 has been recorded, the data from research by Bao et al. [28] demonstrate that primary SARS-CoV-2 infection in monkeys may generate an immunity response, which could be able to protect from subsequent exposures.
The aim of this research was to review literature in order to obtain an overview of existing serological tests for the detection of SARS-CoV-2 infection and to establish their reliability in the diagnosis of this new pathogen,

2. Materials and Methods

2.1. Protocol and Registration

In order to provide a transparent and complete protocol for systematic reviews, the PRISMA statement [29] was followed for methods and inclusion criteria selection.

2.2. Eligibility Criteria

2.2.1. Clinical Question (PICO)

P: A population of participants with certain diagnosis of COVID-19 (obtained thorough real-time RT-PCR).
I: Detection of IgM and IgG titers in patients with SARS-CoV-2 infection using serological assays.
C: Comparison between IgM/IgG titer in SARS-CoV-2-positive patients and negative controls.
O: Reliability of the existing serological assays for the diagnosis of COVID-19.

2.2.2. Inclusion and Exclusion Criteria

All the papers that aimed to evaluate the diagnostic capability of serological tests in the detection of SARS-CoV-2 infection and that met the following inclusion criteria were selected for our review:
-
Studies that reported serological assay quantitative data (percentages, rates) about the IgM and IgG titers of the analyzed blood samples.
-
Studies that tested serological assays recording both IgM and IgG titers.
-
Studies in which the analyzed blood samples were taken from patients with certain diagnosis of SARS-CoV-2 infection, obtained through real-time RT-PCR.
-
Articles written in the English language.
Reviews and case reports were not included in our study.

2.3. Search

In order to select items concerning the utilization of antibody tests to diagnose SARS-CoV-2-infected patients, we conducted electronic research in the PubMed, Scopus, Science Direct and Cochrane Library databases, analyzing papers published by April 2020. No restrictions were imposed with regards to demographics or clinical characteristics of the included patients (age, gender, comorbidities), and both articles with and without negative control groups were considered. Only studies written in the English language were selected. We combined the following keywords with the Boolean term “AND”: “serological test”, “COVID-19”, “antibodies” and “immune system”.

2.4. Study Selection and Data Collection Process

Following the inclusion criteria, eligible studies for this review were selected by two researchers (Dorina Lauritano, Giulia Moreo), who independently examined title, abstract and full texts of each article found during the electronic search. The same researchers performed data extraction from the selected items: the number of enrolled patients/blood samples, patient and blood sample sources, diagnostic tool used to confirm the presence of SARS-CoV-2 infection in the included patients, type of serological assay tested, and quantitative data about the sensitivity, specificity, positive/negative predictive value and IgM/IgG titer provided by each antibody test. Only articles that used real-time RT-PCR as a diagnostic tool to confirm the positivity of SARS-CoV-2 were considered, and rates and percentages were used for the principal outcome measures. The flow chart used for this review is shown in Figure 1.

2.5. Quality Assessment

Quality assessment of the selected items was investigated using the Newcastle–Ottawa scale (NOS) [30], recording an high quality level for the included research: the studies’ average quality was equal to 6.1, while the highest score was equal to 7 and the lowest one was 5 (Table 1). The majority of studies included case and control groups, in which data were recorded using the same methodology. The presence of SARS-CoV-2 infection in the cases of all the selected articles was confirmed following the guidelines for diagnosis of COVID-19 [31].

3. Results

3.1. Study Selection and Characteristics

A total of 212 articles were identified after the electronic research in four different databases (PubMed, Scopus, Science Direct, Cochrane Library). Ten studies were excluded based on duplication removal and 202 items were assessed for eligibility. Overall, 192 articles were not included with reasons and only 10 articles were included in this review: 83 studies were excluded after examining the title, 47 after analyzing the abstract (absence of quantitative data about IgM and IgG titer), 41 after reading the full text (diagnosis confirmation not specified or absence of both IgM and IgG titer analysis), and 21 because of their study design (reviews or case reports). The main characteristics of each selected article are summarized in Table 2, which reports the number of enrolled patients and blood samples and their source, the diagnostic tool used to ascertain the positivity of SARS-CoV-2 (real-time RT-PCR), and the type of serological assay whose diagnostic capability was evaluated. By adding the participants of each study, a total of 1362 subjects were included in this review, of which 945 were cases (SARS-CoV-2-positive patients, whose diagnosis was confirmed by real-time RT-PCR), 310 were negative controls, and 107 were subjects with suspected infection (SARS-CoV-2 clinical diagnosis, real-time RT-PCR negative patients). Eight of the included articles were conducted in China, with the remainder in Italy. Data about IgM and IgG titer, sensitivity, specificity, and the positive and negative predictive value provided by each serological test were recorded in this review.

3.2. Results of Individual Studies

The data recorded in this review (Table 3, Table 4, Table 5 and Table 6) referred to 10 different types of serological tests, which were performed in the included articles in order to establish whether the antibody response can be considered a reliable diagnostic tool for SARS-CoV-2 infection. The VivaDiag COVID-19 IgM/IgG Rapid Test lateral flow immunoassay (LFIA) was used in a sample of 110 patients in the study by Cassaniti et al. [32]: 19 out of 30 cases (63.3%), whose serum samples were taken at a median 7 days after the symptom onset, were positive for both IgM and IgG; five of them were weakly positive; all negative controls were recorded to be negative for both antibodies. The same study also included 50 patients at their first access to the emergency department, who were later tested for COVID-19 by RT-PCR, detecting 38 SARS-CoV-2-positive patients, among whom only seven (18.4%) showed positivity for both IgM and IgG. The sensitivity and specificity of the VivaDiag test were recorded to be 18.4% and 91.7% respectively, results that lead the authors not to recommend its use in the infection diagnostic process.
Two of the included items performed the serological test using an iFlash1800 [33] and iFlash3000 [34] fully automated chemiluminescence immunoassay (CLIA) analyzer from Shenzhen YHLO Biotechnology Co., Ltd. (China), respectively.
The iFlash1800 CLIA analyzer guaranteed an overall sensitivity of 75% and a specificity of 100% for IgG and 92.2% for IgM. The collected blood samples analyzed thorough this test had a mean duration of 12 days (range 8–17 days) from the onset of symptoms.
The iFlash 3000 CLIA analyzer reported, among the cases, 41 IgM/IgG-positive (64.1%), 3 IgM-positive (16.7%) and 5 IgG-positive patients out of 64, with a sensitivity of 48.1% (IgM) and 88.9% (IgG) and a specificity equal to 100% (IgM) and 90.9% (IgG). The duration from first symptoms to the first serological test was, on average, 18 days: both IgM and IgG levels were not high during the first 5 days following symptom onset.
Lee et al. [35] used the ALLTEST 2019-nCoV IgM/IgG Rapid Test Cassette (Hangzhou ALLTEST Biotech Co., Ltd. Hangzhou, China) to identify SARS-CoV-2 IgM and IgG in 14 COVID-19-positive patients (six with symptoms and eight without symptoms or with mild symptoms). All the symptomatic subjects showed IgG positivity (6/6), while two of them were IgM-negative (2/6). Three of the asymptomatic patients had positive IgG, but none of them had positive IgM. This research reported the earliest detection of IgM and IgG, on day 5.
The sensitivity and specificity of the SARS-CoV-2 rapid IgM/IgG combined antibody test kit developed in the study by Li et al. [36] were 88.66% and 90.63%, respectively; 256 out of 397 cases and 1 out 128 negative controls were positive for both IgM and IgG. This serological test was performed from day 8 to day 33 after symptoms’ appearance.
By using a colloidal gold-based immunochromatographic (ICG) strip, Pan et al. [37] analyzed 108 blood samples (starting from 7 days after symptom onset), of which 86 were taken from SARS-CoV-2-positive patients (cases) and 22 from subjects with suspected infection (with negative RT-PCR): 55.8% and 54.7% of SARS-CoV-2-positive blood samples showed positivity to IgM and IgG, respectively, while 36.4% and 59.1% of the blood samples taken from patients with suspected infection were positive for IgM and IgG, respectively.
In research by Xiang et al. [38] the antibodies against this new pathogen were found with an enzyme-linked immunosorbent assay (ELISA): of 66 positive participants, 51 were IgM-positive and 55 were IgG-positive; 21 patients with suspected infection out of 24 were IgM-positive and 17 IgG-positive; negative controls (60) were all negative for IgM and only three were positive for IgG. The detection of the antibodies was conducted from day 4 to day 28 after the beginning of the symptoms, recording an increase in both IgM and IgG rates: from 60% to 63.6% for IgM and from 40% to 90.9% for IgG.
The gold immunochromatography assay supplied by Innovita Co., LTd, China (CFDA approved) tested by Yongchen et al. [40] highlighted that among the coronavirus-2-positive group (21 persons) all the symptomatic patients (17 subjects) were recorded to be seropositive during the follow-up period. Zhang et al. [41] developed an in-house anti-SARS-CoV IgG/IgM ELISA kit (using a cross-reactive nucleocapsid protein from another SARS-related virus, Rp3, which is 92% identical to SARS-CoV-2 nucleocapsid protein), employing it to investigate the antibody response in 16 positive participants: this study demonstrated that IgM and IgG titers were low or undetectable on the day of first sampling (day 0), and that, on day 5, an increase in viral antibodies could be observed in nearly all patients: IgM and IgG positive rate increased from 50% (8/16) to 81% (13/16) and from 81% (13/16) to 100% (16/16).
The ELISA kit supplied by Beijing Wantai Biological Pharmacy Enterprise Co., Ltd., used by Zhao et al. [39] reported that 82.7% and 64.7% of 173 SARS-CoV-2-positive subjects were IgM- and IgG-positive, respectively, ensuring a sensitivity equal to 66.7% in the early phase of the illness.

4. Discussion

This review aimed to provide an overview of the existing serological tests, assessing their capacity to detect the presence of IgM and IgG antibodies against SARS-CoV-2 in blood samples taken from patients. During SARS-CoV-2 infection, an overactivation of natural immunity cells, such as macrophages and monocytes, has been recorded, which consequently causes the diminution of lymphocytes and the increase of neutrophils, interleukin-6 (IL-6) and reactive protein C (PCR) [42]. With regard to the adaptive immune response, the activation of B lymphocytes of the humoral immunity (which occurs approximately after one week from infection) leads to the production of specific antibodies against SARS-CoV-2: IgM and IgG [43]. According to the literature, IgM antibodies constitute the first line of defense during a viral infection and can indicate the acute phase of the disease; IgG antibodies represent long-term immunity and immunological memory, and their presence highlights that exposure to the pathogen has occurred several days before [43,44]. The study by Demey et al. [45] tested four different immunochromatographic assays, demonstrating that the antibodies’ detection time was, on average, 8–10 days after the symptoms’ onset, while the case report of Thevarajan et al. [46] recorded a progressive increment of anti-SARS-CoV-2 IgM and IgG from day 7 until day 20 from the beginning of the illness. Two of the articles selected for this review reported that the IgM-positive rate tended to increase at first but then declined, while those of IgG were higher than of IgM at all times [34], and that IgM and IgG positive rates increased from 50% to 81% and from 81% to 100%, respectively [41]. Pan et al. [37] divided the infection into three stages: early (1–7 days from the onset), intermediate (8–14 days) and late (more than 15 days), establishing that the positive rate of IgM was raised from 11.1% (early stage) to 78.6% (intermediate stage) and 74.2% (late stage), whereas that of IgG was 3.6% in the early stages and 57.1% and 96.8% in the intermediate and late stages, respectively. Infantino et al. [33] was able to detect IgM and IgG from day 10 to day 30 and from day 20 onwards after COVID-19 infection, respectively. On the basis of the results obtained by Lee et al. [35], the persistence of positive real-time RT-PCR seemed to be shorter in symptomatic patients, who developed IgM antibodies. The median seroconversion time detected by Zhao et al. [39] was day 12 and day 14 for IgM and IgG respectively. Among the five patients with severe symptoms analyzed by Yongchen et al. [40], the antibody response was individuated within week 2, and three out of five of these subjects developed IgG response prior to viral clearance, indicating that high levels of this new pathogen viral load may provide an early antibody response [47,48]. Almost all the included items agreed on the fact that serological tests could be effective and reliable diagnostic tools for SARS-CoV-2 infection identification, since they are able to provide high sensitivity and specificity, and that their utilization should be complementary to the execution of real-time RT-PCR. Because of its several limitations, rRT-PCR could also report negative results in infected individuals: very early or late collection of swabs, poor quality of the specimen containing insufficient material quantity, or wrong technical procedures. Furthermore, it takes a long time to generate RT-PCR test results, it requires first-rate certified laboratory facilities with ad-hoc educated staff, and it may provide different results depending on the sampling site (oropharyngeal or nasopharyngeal swabs) [49]. Antibody assays could provide a faster, less expensive and simpler (no laboratory training need) method to diagnose COVID-19 [38]. As Li et al. [36] stated, serological tests may be used to screen the possible asymptomatic carriers [50], knowing that the majority of them develop anti-SARS-CoV-2 antibodies and, since these assays are able to individuate IgM and IgG simultaneously, they could be used for early diagnosis (detecting IgM) and for monitoring during the therapy [37]. The narrative review by Cheng et al. [51] highlighted that serological tests’ negative results should not be a reason to exclude SARS-CoV-2 infection, considering that the patient may have been recently exposed to the pathogen. Moreover, present or past infections due to other coronaviruses could lead to cross-reactivity of antibody to non-SARS-CoV-2 coronavirus proteins. According to the same authors, this type of assay could be useful for epidemiological studies, vaccine studies and risk assessment of health care workers. In contrast, Cassaniti et al. [32] also performed the serological assay on 30 healthy volunteers, 10 of whom had been infected in the past with OC43, 229E, HKU1 and NL63 coronavirus, reporting no cross-reactivity with antibodies against these pathogens. Infantino et al. [33] confirmed the absence of cross reaction with other coronaviruses but demonstrated that cytomegalovirus (CMV) infections and some rheumatic diseases could interfere with the test.

5. Conclusions

The global pandemic caused by the novel SARS-CoV-2 is threatening international health and screening of the population on a large scale has become imperative. Besides the use of real-time RT-PCR to individuate the presence of COVID-19, the detection of specific antibodies in response to this pathogen thorough serological assays may represent a reliable diagnostic protocol. Serological assays seem to be able to overcome the nucleic acid test’s limitations, ensuring the diagnosis in asymptomatic patients and in false-negative RT-PCR case.

Author Contributions

Conceptualization, D.L. and G.M.; methodology, D.L and G.M.; data curation, L.L.; writing—original draft preparation, G.M.; writing—review and editing, G.M.; investigation, T.T.; supervision, F.C. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Hu, Y.; Sun, J.; Dai, Z.; Deng, H.; Li, X.; Huang, Q.; Wu, Y.; Sun, L.; Xu, Y. Prevalence and severity of corona virus disease 2019 (COVID-19): A systematic review and meta-analysis. J. Clin. Virol. 2020, 127, 104371. [Google Scholar] [CrossRef] [PubMed]
  2. Rothan, H.A.; Byrareddy, S.N. The epidemiology and pathogenesis of coronavirus disease (COVID-19) outbreak. J. Autoimmun. 2020, 109, 102433. [Google Scholar] [CrossRef] [PubMed]
  3. Sohrabi, C.; Alsafi, Z.; O’Neill, N.; Khan, M.; Kerwan, A.; Al-jabir, A.; Losifidis, C.; Agha, R. World Health Organization declares global emergency: A review of the 2019 novel coronavirus (COVID-19). Int. J. Surg. 2020, 76, 71–76. [Google Scholar] [CrossRef] [PubMed]
  4. Ahn, D.G.; Shin, H.J.; Kim, M.H.; Lee, S.; Kim, H.S.; Myoung, J.; Kim, B.T.; Kim, S.J. Current status of epidemiology, diagnosis, therapeutics, and vaccines for novel coronavirus disease 2019 (COVID-19). J. Microbiol. Biotechnol. 2020, 30, 313–324. [Google Scholar] [CrossRef]
  5. Astuti, I.; Ysrafi. Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2): An overview of viral structure and host response. Diabetes Metab. Syndr. 2020, 14, 407–412. [Google Scholar] [CrossRef]
  6. Li, L.Q.; Huang, T.; Wang, Y.Q.; Wang, Z.P.; Liang, Y.; Huang, T.B.; Zhang, H.Y.; Sun, W.; Wang, Y. COVID-19 patients’ clinical characteristics, discharge rate, and fatality rate of meta-analysis. J. Med. Virol. 2020. [Google Scholar] [CrossRef]
  7. Zhang, J.J.; Dong, X.; Cao, Y.Y.; Yuan, Y.D.; Yang, Y.B.; Yan, Y.Q.; Akdis, C.A.; Gao, Y.D. Clinical characteristics of 140 patients infected with SARS-CoV-2 in Wuhan, China. Allergy 2020. [Google Scholar] [CrossRef]
  8. Chakraborty, C.; Sharma, A.R.; Sharma, G.; Bhattacharya, M.; Lee, S.S. SARS-CoV-2 causing pneumonia-associated respiratory disorder (COVID-19): Diagnostic and proposed therapeutic options. Eur. Rev. Med. Pharmacol. Sci. 2020, 24, 4016–4026. [Google Scholar] [CrossRef]
  9. Tu, Y.F.; Chien, C.S.; Yarmishyn, A.A.; Lin, Y.Y.; Luo, Y.H.; Lin, Y.T.; Lai, W.Y.; Yang, D.M.; Chou, S.J.; Yang, Y.P.; et al. A review of SARS-CoV-2 and the ongoing clinical trials. Int. J. Mol. Sci. 2020, 21, 2657. [Google Scholar] [CrossRef] [Green Version]
  10. Hassan, S.A.; Sheikh, F.N.; Jamal, S.; Ezeh, J.K.; Akhtar, A. Coronavirus (COVID-19): A review of clinical features, diagnosis, and treatment. Cureus 2020, 12, e7355. [Google Scholar] [CrossRef] [Green Version]
  11. Law, S.; Leung, A.W.; Xu, C. Severe acute respiratory syndrome (SARS) and coronavirus disease-2019 (COVID-19): From causes to preventions in Hong Kong. Int. J. Inf. Dis. 2020. [Google Scholar] [CrossRef] [PubMed]
  12. Peng, X.; Xu, X.; Li, Y.; Cheng, L.; Zhou, X.; Ren, B. Transmission routes of 2019-nCoV and controls in dental practice. Int. J. Oral Sci. 2020, 12, 9. [Google Scholar] [CrossRef] [PubMed]
  13. Singhal, T. A review of coronavirus disease-2019 (COVID-19). Indian J. Pediatr. 2020, 87, 281–286. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Malik, Y.A. Properties of coronavirus and SARS-CoV-2. Malays. J. Pathol. 2020, 42, 3–11. [Google Scholar]
  15. Şimşek-Yavuz, S.; Ünal, S. Antiviral treatment of COVID-19. Turk. J. Med. Sci. 2020, 50, 611–619. [Google Scholar] [CrossRef]
  16. Lombardy Section Italian Society Infectious and Tropical Diseases. Vademecum for the treatment of people with COVID-19. Infez Med. 2020, 28, 143–152. [Google Scholar]
  17. Gao, J.; Tian, Z.; Yang, X. Breakthrough: Chloroquine phosphate has shown apparent efficacy in treatment of COVID-19 associated pneumonia in clinical studies. Biosci. Trends. 2020, 14, 72–73. [Google Scholar] [CrossRef] [Green Version]
  18. Zhai, P.; Ding, Y.; Wu, X.; Long, J.; Zhong, Y.; Li, Y. The epidemiology, diagnosis and treatment of COVID-19. Int. J. Antimicrob. Agents 2020, 105955. [Google Scholar] [CrossRef]
  19. Hong, K.H.; Lee, S.W.; Kim, T.S.; Huh, H.J.; Lee, J.; Kim, S.Y.; Park, J.S.; Kim, G.J.; Sung, H.; Roh, K.H.; et al. Guidelines for laboratory diagnosis of coronavirus disease 2019 (COVID-19) in Korea. Ann. Lab. Med. 2020, 40, 351–360. [Google Scholar] [CrossRef]
  20. Jin, Y.H.; Cai, L.; Cheng, Z.S.; Cheng, H.; Deng, T.; Fan, Y.P.; Fang, C.; Huang, D.; Huang, L.Q.; Huang, Q.; et al. A rapid advice guideline for the diagnosis and treatment of 2019 novel coronavirus (2019-nCoV) infected pneumonia (standard version). Mil. Med. Res. 2020, 7, 4. [Google Scholar] [CrossRef] [Green Version]
  21. Li, Y.; Xia, L. Coronavirus disease 2019 (COVID-19): Role of chest CT in diagnosis and management. AJR Am. J. Roentgenol. 2020, 1–7. [Google Scholar] [CrossRef] [PubMed]
  22. Ai, T.; Yang, Z.; Hou, H.; Zhan, C.; Chen, C.; Lv, W.; Tao, Q.; Sun, Z.; Xia, L. Correlation of chest CT and RT-PCR testing in coronavirus disease 2019 (COVID-19) in China: A report of 1014 cases. Radiology 2020, 200642. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Vashist, S.K. In vitro diagnostic assays for COVID-19: Recent advances and emerging trends. Diagnostics 2020, 10, 202. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Petherick, A. Developing antibody tests for SARS-CoV-2. Lancet 2020, 395, 1101–1102. [Google Scholar] [CrossRef]
  25. Xie, L.X. Interpretation of the 7th edition of the “diagnosis and treatment guidelines of coronavirus disease 2019 in China”: Progress and challenges. Chronic Dis. Transl. Med. 2020. [Google Scholar] [CrossRef]
  26. Patel, R.; Babady, E.; Theel, E.S.; Storch, G.A.; Pinsky, B.A.; St George, K.; Smith, T.C.; Bertuzzi, S. Report from the american society for microbiology COVID-19 international summit, 23 March 2020: Value of diagnostic testing for SARS-CoV-2/COVID-19. mBio 2020, 11. [Google Scholar] [CrossRef] [Green Version]
  27. Infectious Diseases Society of America (IDSA). IDSA COVID-19 Antibody Testing Primer. Available online: https://www.idsociety.org/globalassets/idsa/public-health/covid-19/idsa-covid-19-antibody-testing-primer.pdf (accessed on 29 April 2020).
  28. Bao, L.; Deng, W.; Gao, H.; Xiao, C.; Liu, J.; Xue, J.; Lv, Q.; Liu, J.; Yu, P.; Xu, Y.; et al. Reinfection could not occur in SARS-CoV-2 infected rhesus macaques. bioRxiv 2020. [Google Scholar] [CrossRef] [Green Version]
  29. Liberati, A.; Altman, D.G.; Tetzlaff, J.; Mulrow, C.; Gøtzsche, P.C.; Ioannidis, J.P.; Clarke, M.; Devereaux, P.J.; Kleijnen, J.; Moher, D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: Explanation and elaboration. J. Clin. Epidemiol. 2009, 62, e1–34. [Google Scholar] [CrossRef] [Green Version]
  30. Wells, G.A.; Shea, B.; O’Connell, D.; Peterson, J.; Welch, V.; Losos, M.; Tugwell, P. The Newcastle-Ottawa Scale (NOS) for Assessing the Quality of Nonrandomized Studies in Meta-Analyses. Available online: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (accessed on 24 April 2020).
  31. National Health Commission & State Administration of Traditional Chinese Medicine. Diagnosis and Treatment Protocol for Novel Coronavirus Pneumonia (Trial Version 7). Available online: https://www.fmprc.gov.cn/mfa_eng/topics_665678/kjgzbdfyyq/CERC/P020200318837942843337.pdf (accessed on 28 April 2020).
  32. Cassaniti, I.; Novazzi, F.; Giardina, F.; Salinaro, F.; Sachs, M.; Perlini, S.; Bruno, R.; Mojoli, F.; Baldanti, F. Performance of VivaDiag COVID-19 IgM/IgG Rapid Test is inadequate for diagnosis of COVID-19 in acute patients referring to emergency room department. J. Med. Virol. 2020. [Google Scholar] [CrossRef] [Green Version]
  33. Infantino, M.; Grossi, V.; Lari, B.; Bambi, R.; Perri, A.; Manneschi, M.; Terenzi, G.; Liotti, I.; Ciotta, G.; Taddei, C.; et al. Diagnostic accuracy of an automated chemiluminescent immunoassay for anti-SARS-CoV-2 IgM and IgG antibodies: An Italian experience. J. Med. Virol. 2020. [Google Scholar] [CrossRef]
  34. Jin, Y.; Wang, M.; Zuo, Z.; Fan, C.; Ye, F.; Cai, Z.; Wang, Y.; Cui, H.; Pan, K.; Xu, A. Diagnostic value and dynamic variance of serum antibody in coronavirus disease 2019. Int. J. Infect. Dis. 2020, 94, 49–52. [Google Scholar] [CrossRef] [PubMed]
  35. Lee, Y.L.; Liao, C.H.; Liu, P.Y.; Cheng, C.Y.; Chung, M.Y.; Liu, C.E.; Chang, S.Y.; Hsueh, P.R. Dynamics of anti-SARS-Cov-2 IgM and IgG antibodies among COVID-19 patients. J. Infect. 2020. [Google Scholar] [CrossRef] [PubMed]
  36. Li, Z.; Yi, Y.; Luo, X.; Xiong, N.; Liu, Y.; Li, S.; Sun, R.; Wang, Y.; Hu, B.; Chen, W.; et al. Development and clinical application of a rapid IgM-IgG combined antibody test for SARS-CoV-2 infection diagnosis. J. Med. Virol. 2020. [Google Scholar] [CrossRef] [PubMed]
  37. Pan, Y.; Li, X.; Yang, G.; Fan, J.; Tang, Y.; Zhao, J.; Long, X.; Guo, S.; Zhao, Z.; Liu, Y.; et al. Serological immunochromatographic approach in diagnosis with SARS-CoV-2 infected COVID-19 patients. J. Infect. 2020. [Google Scholar] [CrossRef]
  38. Xiang, F.; Wang, X.; He, X.; Peng, Z.; Yang, B.; Zhang, J.; Zhou, Q.; Ye, H.; Ma, Y.; Li, H.; et al. Antibody detection and dynamic characteristics in patients with COVID-19. Clin. Infect. Dis. 2020. [Google Scholar] [CrossRef]
  39. Zhao, J.; Yuan, Q.; Wang, H.; Liu, W.; Liao, X.; Su, Y.; Wang, X.; Yuan, J.; Li, T.; Li, J.; et al. Antibody responses to SARS-CoV-2 in patients of novel coronavirus disease 2019. Clin. Infect. Dis. 2020. [Google Scholar] [CrossRef]
  40. Yongchen, Z.; Shen, H.; Wang, X.; Shi, X.; Li, Y.; Yan, J.; Chen, Y.; Gu, B. Different longitudinal patterns of nucleic acid and serology testing results based on disease severity of COVID-19 patients. Emerg. Microbes Infect. 2020, 9, 833–836. [Google Scholar] [CrossRef] [Green Version]
  41. Zhang, W.; Du, R.H.; Li, B.; Zheng, X.S.; Yang, X.L.; Hu, B.; Wang, Y.Y.; Xiao, G.F.; Yan, B.; Shi, Z.L.; et al. Molecular and serological investigation of 2019-nCoV infected patients: Implication of multiple shedding routes. Emerg. Microbes Infect. 2020, 9, 386–389. [Google Scholar] [CrossRef] [Green Version]
  42. Qin, C.; Zhou, L.; Hu, Z.; Zhang, S.; Yang, S.; Tao, Y.; Xie, C.; Ma, K.; Shang, K.; Wang, W.; et al. Dysregulation of immune response in patients with COVID-19 in Wuhan, China. Clin. Infect. Dis. 2020, 12. [Google Scholar] [CrossRef]
  43. di Cristina, S.M.G.; Concetta, R.; Francesco, R.; Annalisa, C. SARS-Cov-2 infection: Response of human immune system and possible implications for the rapid test and treatment. Int. Immunopharmacol. 2020, 84, 106519. [Google Scholar] [CrossRef]
  44. Castro, R.; Luz, P.M.; Wakimoto, M.D.; Veloso, V.G.; Grinsztejn, B.; Perazzo, H. COVID-19: A meta-analysis of diagnostic test accuracy of commercial assays registered in Brazil. Braz J. Infect. Dis. 2020. [Google Scholar] [CrossRef] [PubMed]
  45. Demey, B.; Daher, N.; François, C.; Lanoix, J.-P.; Duverlie, G.; Castelain, S.; Brochot, E. Dynamic profile for the detection of anti-SARS-CoV-2 antibodies using four immunochromatographic assays. J. Infect. 2020. [Google Scholar] [CrossRef] [PubMed]
  46. Thevarajan, I.; Nguyen, T.H.O.; Koutsakos, M.; Druce, J.; Caly, L.; van de Sandt, C.E.; Jia, X.; Nicholson, S.; Catton, M.; Cowie, B.; et al. Breadth of concomitant immune responses prior to patient recovery: A case report of non-severe COVID-19. Nat. Med. 2020, 26, 453–455. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  47. Lam, J.H.; Baumgarth, N. The multifaceted B cell response to influenza virus. J. Immunol. 2019, 202, 351–359. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  48. Baumgarth, N. How specific is too specific? B-cell responses to viral infections reveal the importance of breadth over depth. Immunol. Rev. 2013, 255, 82–94. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  49. Infantino, M.; Damiani, A.; Gobbi, F.L.; Grossi, V.; Lari, B.; Macchia, D.; Casprini, P.; Veneziani, F.; Villalta, D.; Bizzaro, N.; et al. Serological assays for SARS-CoV-2 infectious disease: Benefits, limitations and perspectives. Isr. Med. Assoc. J. 2020, 22, 203–210. [Google Scholar]
  50. Wu, X.; Fu, B.; Chen, L.; Feng, Y. Serological tests facilitate identification of asymptomatic SARS-CoV-2 infection in Wuhan, China. J. Med. Virol. 2020. [Google Scholar] [CrossRef] [Green Version]
  51. Cheng, M.P.; Papenburg, J.; Desjardins, M.; Kanjilal, S.; Quach, C.; Libman, M.; Dittrich, S.; Yansouni, C.P. Diagnostic testing for severe acute respiratory syndrome-related Coronavirus-2: A narrative review. Ann. Intern. Med. 2020. [Google Scholar] [CrossRef] [Green Version]
Applsci 10 04506 g001 550
Figure 1. Flow chart of the item assessments.
Figure 1. Flow chart of the item assessments.
Applsci 10 04506 g001
Table
Table 1. Quality evaluation of the included articles.
Table 1. Quality evaluation of the included articles.
StudiesDefinition of CasesRepresentativeness of CasesSelection of ControlsDefinition of ControlsComparabilityExposureTotal
Cassaniti et al. 2020+++++−++−7
Infantino et al. 2020+++++−+−−6
Jin et al. 2020+++++−++−7
Lee et al. 2020+++++−++−7
Li et al. 2020+++++−++−7
Pan et al. 2020++−−+−++−5
Xiang et al. 2020+++++−++−7
Yongchen et al. 2020++−−+−++−5
Zhang et al. 2020++−−+−++−5
Zhao et al. 2020++−−+−++−5
+ = star assigned; − = star not assigned
Table
Table 2. List of the included items.
Table 2. List of the included items.
StudyNumber of Enrolled Subjects/Blood SamplesDiagnosis of SARS-CoV-2Patient/Blood Sample SourceSerological Test
Cassaniti et al. 2020110 subjects:
30 healthy volunteers
30 positive patients *
50 patients with fever and respiratory syndrome		Respiratory samples tested by real-time RT-PCR Fondazione IRCCS Policlinico San MatteoVivaDiag COVID-19 IgM/IgG Rapid Test LFIA
Infantino et al. 2020125 subjects:
61 positive patients *
64 negative controls		Oropharyngeal and nasopharyngeal swabs tested by RT-PCRSan Giovanni di Dio Hospital (Florence, Italy)iFlash1800 fully automated CLIA analyzer from Shenzhen YHLO Biotech Co., Ltd. (China)
Jin et al. 202076 subjects:
43 positive patients *
33 patients with suspected infection (control group)		Oral swab or sputum tested by real-time RT-PCR XiXi Hospital of Hangzhou (Zhejiang Province, China)iFlash3000 fully automated CLIA analyzer from Shenzhen YHLO Biotech Co., Ltd. (China)
Lee et al. 202042 subjects:
14 positive patients */33 serum samples
28 negative controls/28 serum samples		Oropharyngeal and nasopharyngeal swabs, oral gargling and sputum tested by real-time RT-PCR for SARS-CoV-2Enrolled patients were treated at six hospitals in Taiwan between January and March 2020ALLTEST 2019-nCoV IgM/IgG Rapid Test Cassette (Hangzhou ALLTEST Biotech Co., Ltd. Hangzhou, China)
Li et al. 2020525 subjects:
397 positive patients *
128 negative controls		Guideline for diagnosis and treatment of COVID-19Eight hospitals and Chinese CDC agenciesSARS-CoV-2 rapid IgM/IgG combined antibody test (LFIA) kit designed and manufactured by Jiangsu Medomics Medical Technologies (Nanjing, China)
Pan et al. 2020104 subjects:
67 positive patients */86 serum samples
subjects:
37 clinically diagnosed patients/39 serum samples		Throat swab tested by real-time RT-PCR for SARS-CoV-2Zhongnan Hospital of Wuhan University (Hubei, China)Colloidal gold-based immunochromatographic (ICG) strip targeting IgM/IgG, conducted in Zhongnan Hospital of Whuan University (Hubei, China)
Xiang et al. 2020169 subjects:
85 positive patients */216 blood samples
24 patients with suspected infection
60 negative controls		Nasopharyngeal and/or oropharyngeal swab samples tested by RT-PCR for SARS-CoV-2Union Hospital, Tongji Medical College, Huazhong University of Science and Technology (China)ELISA kits, Livzon Inc, Zhuhai, P.R.China, lot number of IgM: 20200308, IgG: 20200308
Yongchen et al. 202021 positive patients *:
11 non-severe
5 severe
5 asymptomatic		Throat swabs samples tested by real-time RT-PCR for SARS-CoV-2Second Hospital of Nanjing and Affiliated Hospital of Xuzhou Medical University (Jiangsu Province, China)Gold immunochromatography assay supplied by Innovita Co., LTd, China (CFDA approved)
Zhang et al. 202016 positive patients *Oral, anal and blood samples tested by qPCRWuhan Pulmonary Hospital (China)In-house anti-SARS-CoV IgG/IgM ELISA kits (using a cross-reactive nucleocapsid protein from another SARS-related virus Rp3, which is 92% identical to COVID-2019 nucleocapsid protein)
Zhao et al. 2020535 serial plasma samples from 173 positive patients *Respiratory tract samples tested by real-time RT-PCRShenzhen Third People’s Hospital (China)ELISA kits supplied by Beijing Wantai Biological Pharmacy Enterprise Co.,Ltd,
CLIA = chemiluminescence immunoassay; CoV = coronavirus; ELISA = enzyme-linked immunosorbent assay; HCoV = human coronavirus; LFIA = lateral flow immunoassay; MERS-CoV = Middle East respiratory syndrome; qPCR = quantitative polymerase chain reaction; RT-PCR = reverse transcription polymerase chain reaction; SARS-CoV = severe acute respiratory syndrome coronavirus. * Positive patients: patients with confirmed diagnosis of SARS-CoV-2 by testing respiratory tract swabs with real-time RT-PCR.
Table
Table 3. Anti-SARS-CoV-2 IgM and IgG rates in the COVID-19-positive population sample.
Table 3. Anti-SARS-CoV-2 IgM and IgG rates in the COVID-19-positive population sample.
StudyTotal SampleIgM/IgG-Positive PatientsIgM/IgG-Negative PatientsIgM-PositiveIgG-Positive
[32]30 19/30 (63.3%) 5/30 (16.7%) weakly positive5/30 (16.7%)1/30 (3.3%): IgM-positive and IgG-negative
[33]6441/64 (64.1%) 3/64 (4.7%)5/64 (7.8%)
[34]27 3/27 (11.1%)13/27 (48%)24/27 (88.9%)
[35]6 with symptoms 8 without symptoms/mild symptoms 4/6 (66.6%)
0/8 (0%)		6/6 (100%)
3/8 (37.5%)
[36]397256/397 (64.4%) 72/397 (18%)24/397 (6.04%)
[37]86 blood samples 48/86 (55.8%)47/86 (54.7%)
[38]66 51/66 (77.2%)55/66 (83.3%)
[39]173 143/173 (82.7%)112/173 (64.7%)
Table
Table 4. Anti-SARS-CoV-2 IgM and IgG rates in patients with suspected infection/fever and respiratory syndrome (positivity to COVID-19 not confirmed by RT-PCR).
Table 4. Anti-SARS-CoV-2 IgM and IgG rates in patients with suspected infection/fever and respiratory syndrome (positivity to COVID-19 not confirmed by RT-PCR).
StudyTotal SampleIgM/IgG-Positive PatientsIgM/IgG-Negative PatientsIgM-PositiveIgG-Positive
[32]50 patients at their first access to emergency department, later tested for COVID-19 by RT-PCR: 12 negative 38 positive1/12 negative patients (8.3%)
7/38 positive patients (18.4%)		31/38 positive patients
[37]22 blood samples 8/22 (36.4%)13/22 (59.1%)
[38]24 21/24 (87.5%)17/24 (70.8%)
Table
Table 5. Anti-SARS-CoV-2 IgM and IgG rates in negative controls.
Table 5. Anti-SARS-CoV-2 IgM and IgG rates in negative controls.
PTotal SampleIgM/IgG-Positive PatientsIgM/IgG-Negative PatientsIgM-PositiveIgG-Positive
[32]30 30 (100%)
[34]33 0 (0%)3 (9.1%)
[35]28 28 (100%)
[36]1281/128 (0.7%) 10/128 (7.8%)1/128 (0.7%)
[38]60 0/60 (0%)3/60 (5%)
Table
Table 6. Sensitivity, specificity, positive predictive values and negative predictive values of serum IgM and IgG antibodies to diagnose COVID-19.
Table 6. Sensitivity, specificity, positive predictive values and negative predictive values of serum IgM and IgG antibodies to diagnose COVID-19.
Serological TestNumber of Collected Blood SamplesSensitivitySpecificityPositive Predictive ValueNegative Predictive Value
VivaDiag COVID-19 IgM/IgG Rapid Test LFIA 18.4% **91.7% **87.5% **26.2% **
iFlash1800 fully automated CLIA analyzer from Shenzhen YHLO Biotech Co., Ltd. (China) 73.3% (IgM)
76.7% (IgG)		92.2% (IgM)
100% (IgG)		81.5% (IgM)
100% (IgG)		88.1% (IgM)
92.8% (IgG)
iFlash3000 fully automated CLIA analyzer from Shenzhen YHLO Biotech Co., Ltd. (China):43 positive patients→98 samples 48.1% (IgM) *
88.9% (IgG) *		100% (IgM) *
90.9% (IgG) *		100% (IgM) *
88.9% (IgG) *		70.2% (IgM) *
90.9% (IgG) *
ALLTEST 2019-nCoV IgM/IgG Rapid Test Cassette (Hangzhou ALLTEST Biotech Co., Ltd. Hangzhou, China)14 positive patients→33 samples
28 controls →28 samples		90.9% (IgM) *
99.9% (IgG) *		97.0% (IgM) *
98.0% (IgG) *
SARS-CoV-2 rapid IgM/IgG combined antibody test kit designed and manufactured by Jiangsu Medomics Medical Technologies (Nanjing, China)397 positive patients→397 samples
128 controls→128 samples		88.66% *90.63% ***
ELISA kits, Livzon Inc, Zhuhai, P.R.China, lot number of IgM: 20200308, IgG: 2020030885 positive patients→216 samples77.3% (IgM) * 83.3% (IgG) *

100% (IgM) ** 96.6% (IgG) **		100% (IgM) * 94.8% (IgG) *

87.5% (IgM) 70.8% (IgG)		100% (IgM) * 83.8% (IgG) *

100% (IgM) 85.0% (IgG)		80.0% (IgM) * 88.9% (IgG) *

95.2% (IgM) 89.1% (IgG)
ELISA kits supplied by Beijing Wantai Biological Pharmacy Enterprise Co., Ltd.173 positive patients→535 samples66.7% (early phase of illness)
CLIA = chemiluminescence immunoassay; ELISA = enzyme-linked immunosorbent assay; LFIA = lateral flow immunoassay; ICG = immunochromatographic; SN = sensitivity; SP = specificity. * values refer to SARS-CoV-2-positive patients; ** values refer to patients enrolled from emergency room departments or to patients with suspected COVID-19 pneumonia; *** values refer to negative controls.

Share and Cite

MDPI and ACS Style

Carinci, F.; Moreo, G.; Limongelli, L.; Testori, T.; Lauritano, D. Diagnostic Performance of Serological Assays in the Detection of SARS-CoV-2: A Review. Appl. Sci. 2020, 10, 4506. https://doi.org/10.3390/app10134506

AMA Style

Carinci F, Moreo G, Limongelli L, Testori T, Lauritano D. Diagnostic Performance of Serological Assays in the Detection of SARS-CoV-2: A Review. Applied Sciences. 2020; 10(13):4506. https://doi.org/10.3390/app10134506

Chicago/Turabian Style

Carinci, Francesco, Giulia Moreo, Luisa Limongelli, Tiziano Testori, and Dorina Lauritano. 2020. "Diagnostic Performance of Serological Assays in the Detection of SARS-CoV-2: A Review" Applied Sciences 10, no. 13: 4506. https://doi.org/10.3390/app10134506

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop