Next Article in Journal
UBE1a Suppresses Herpes Simplex Virus-1 Replication
Next Article in Special Issue
Dysregulated Interferon Response Underlying Severe COVID-19
Previous Article in Journal
Disease Pandemics and Major Epidemics Arising from New Encounters between Indigenous Viruses and Introduced Crops
Previous Article in Special Issue
Camelid Inoculation with Middle East Respiratory Syndrome Coronavirus: Experimental Models of Reservoir Host Infection
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients

by
Anwar M. Hashem
1,2,*,†,
Abdullah Algaissi
3,4,†,
Sarah A. Almahboub
1,†,
Mohamed A. Alfaleh
1,5,†,
Turki S. Abujamel
1,6,
Sawsan S. Alamri
1,7,
Khalid A. Alluhaybi
1,5,
Haya I. Hobani
1,
Rahaf H. AlHarbi
1,
Reem M. Alsulaiman
1,
M-Zaki ElAssouli
1,
Sharif Hala
8,
Naif K. Alharbi
9,
Rowa Y. Alhabbab
1,6,
Ahdab A. AlSaieedi
1,6,
Wesam H. Abdulaal
7,
Abdullah Bukhari
10,
Afrah A. AL-Somali
11,
Fadwa S. Alofi
12,
Asim A. Khogeer
13,
Arnab Pain
14,15,16,
Almohanad A. Alkayyal
17,18,
Naif A. M. Almontashiri
19,
Bakur Mahmoud Ahmad
18,20 and
Xuguang Li
21,22
add Show full author list remove Hide full author list
1
Vaccines and Immunotherapy Unit, King Fahd Medical Research Center, King Abdulaziz University, Jeddah 21589, Saudi Arabia
2
Department of Medical Microbiology and Parasitology, Faculty of Medicine, King Abdulaziz University, Jeddah 21589, Saudi Arabia
3
Department of Medical Laboratories Technology, College of Applied Medical Sciences, Jazan University, Jazan 45142, Saudi Arabia
4
Medical Research Center, Jazan University, Jazan 45142, Saudi Arabia
5
Department of Pharmaceutics, Faculty of Pharmacy, King Abdulaziz University, Jeddah 21589, Saudi Arabia
6
Department of Medical Laboratory Technology, Faculty of Applied Medical Sciences, King Abdulaziz University, Jeddah 21589, Saudi Arabia
7
Department of Biochemistry, Faculty of Science, King Abdulaziz University, Jeddah 21589, Saudi Arabia
8
King Abdullah International Medical Research Center, King Saud bin Abdulaziz University for Health Sciences, Ministry of National Guard Health Affairs, Jeddah 11426, Saudi Arabia
9
King Abdullah International Medical Research Center, King Saud bin Abdulaziz University for Health Sciences, Ministry of National Guard Health Affairs, Riyadh 14611, Saudi Arabia
10
Department of Medicine, Faculty of Medicine, Imam Mohammed Ibn Saud Islamic University, Riyadh 11432, Saudi Arabia
11
Infectious Diseases Department, King Abdullah Medical Complex, Jeddah 24246, Saudi Arabia
12
Infectious Diseases Department, King Fahad Hospital, Almadinah Almunwarah 11525, Saudi Arabia
13
Plan and Research Department, General Directorate of Health Affairs Makkah Region, Ministry of Health, Makkah 11176, Saudi Arabia
14
Pathogen Genomics Laboratory, Division of Biological and Environmental Sciences and Engineering (BESE), Thuwal 23955, Saudi Arabia
15
Research Center for Zoonosis Control, Hokkaido University, N20 W10, Kita-ku, Sapporo 001-0030, Japan
16
Nuffield Division of Clinical Laboratory Sciences (NDCLS), University of Oxford, Oxford OX3 9DU, UK
17
Department of Medical Laboratory Technology, University of Tabuk, Tabuk 71491, Saudi Arabi
18
Immunology Research Program, King Abdullah International Medical Research Center, Riyadh 14611, Saudi Arabia
19
Center for Genetics and Inherited Diseases, Taibah University, Almadinah Almunwarah 71491, Saudi Arabia
20
College of Applied Medical Sciences, Taibah University, Almadinah Almunwarah 71491, Saudi Arabia
21
Centre for Biologics Evaluation, Biologics and Genetic Therapies Directorate, Health Products and Food Branch (HPFB), Health Canada and WHO Collaborating Center for Standardization and Evaluation of Biologicals, Ottawa, ON K1A 0K9, Canada
22
Department of Biochemistry, Microbiology and Immunology, Faculty of Medicine, University of Ottawa, Ottawa, ON K1H 8M5, Canada
*
Author to whom correspondence should be addressed.
These authors contributed equally to this work.
Viruses 2020, 12(12), 1390; https://doi.org/10.3390/v12121390
Submission received: 31 October 2020 / Revised: 29 November 2020 / Accepted: 30 November 2020 / Published: 4 December 2020
(This article belongs to the Collection Coronaviruses)

Abstract

:
The Coronavirus Disease 2019 (COVID-19), caused by SARS-CoV-2, continues to spread globally with significantly high morbidity and mortality rates. Antigen-specific responses are of unquestionable value for clinical management of COVID-19 patients. Here, we investigated the kinetics of IgM, IgG against the spike (S) and nucleoproteins (N) proteins and their neutralizing capabilities in hospitalized COVID-19 patients with different disease presentations (i.e., mild, moderate or severe), need for intensive care units (ICU) admission or outcomes (i.e., survival vs death). We show that SARS-CoV-2 specific IgG, IgM and neutralizing antibodies (nAbs) were readily detectable in almost all COVID-19 patients with various clinical presentations. Interestingly, significantly higher levels of nAbs as well as anti-S1 and -N IgG and IgM antibodies were found in patients with more severe symptoms, patients requiring admission to ICU or those with fatal outcomes. More importantly, early after symptoms onset, we found that the levels of anti-N antibodies correlated strongly with disease severity. Collectively, these findings provide new insights into the kinetics of antibody responses in COVID-19 patients with different disease severity.

1. Introduction

The Coronavirus Disease 2019 (COVID-19) has first emerged in Wuhan, China as atypical pneumonia in a cluster of patients before it spread globally, causing a major pandemic that has affected more than 62 million people with around 1,460,000 deaths as of November 29, 2020 (WHO). The causative agent of COVID-19 was identified to be a novel betacoronavirus (beta-CoV) now known as severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [1,2]. Similar to other human CoVs, such as the Middle East respiratory syndrome coronavirus MERS-CoV and SARS-CoV, a zoonotic origin of SARS-CoV-2 was suggested but yet to be confirmed [2,3,4]. SARS-CoV-2 can infect individuals from different age groups and causes a wide spectrum of disease manifestations ranging from asymptomatic, mild, moderate to severe symptoms with possible fatal outcomes [5,6,7]. While the reasons of this wide range disease presentations could be attributed to many factors such as age, sex, pre-existing comorbidities, host genetics as well as other factors, host immune response is a key aspect in determining infection outcomes [7,8,9,10]. Specifically, specific humoral immune responses to the virus could dictate the disease outcomes; however, the early dynamics of antibody responses, including neutralizing antibodies (nAbs), in COVID-19 patients with different clinical presentations, is still not well-characterized. Such information can help our understanding of the nature of COVID-19 infection, guide patient management, as well as aid in the development and evaluation of therapeutics and vaccines. Indeed, the dynamics of early viral-specific antibody responses could influence the progression of several other viral infections including HIV, Influenza, and Ebola [11,12,13].
One of the SARS-CoV-2 proteins capable of inducing robust immune response is the viral spike (S) protein. S protein is comprised of S1 and S2 subunits, with the former known to mediate binding to angiotensin-converting enzyme 2 (ACE2) receptor on host cells and the latter being involved in viral-host membranes fusion [14,15]. As ACE2 is the main receptor, nAbs mainly target the S1 subunit. As such, quantitative determination of antibodies against the S protein is widely used to characterize the antibody responses in COVID-19 patients. A recent study analyzed antibody responses in a small cohort of COVID19 patients found a strong association between the magnitude of anti-S antibody response and patient survival [16].
Another important viral protein capable of inducing strong immune response is the abundantly expressed nucleocapsid (N) protein [17,18]. However, antibodies elicited towards the N protein are not neutralizing and might not provide protection against infection as shown in SARS-CoV infection model [19]. Interestingly, there is a report linking higher anti-N antibody response to severe and fatal outcomes [16]. These data suggest that early humoral immune responses should be further investigated.
In the past few months, several studies have focused on studying the humoral response against SARS-CoV-2 in COVID-19 patients, with antibodies against S1 and N antigens found to emerge as early as one week following disease onset and persist for at least three month after infection [18,20,21,22,23,24]. Here, we studied the kinetics of SARS-CoV-2 specific antibodies to S1 and N viral proteins in blood samples collected between 4 to 70 days post-symptoms onset from a cohort of 87 COVID-19 patients with different disease presentations (i.e., mild, moderate or severe) or outcomes (i.e., survival vs death). As a control, we included serum samples collected before the emergence of SARS-CoV-2. Our data show that most of the COVID-19 patients with different disease categories were able to elicit specific anti-S1 and N IgG and IgM antibodies as well as nAbs that were well-maintained through the period of observation (70 days). We found that the levels of nAbs were strongly correlated with anti S1-IgG antibody response. Additionally, our data show a trend of high levels of anti-N IgG and IgM antibodies in COVID-19 patients based on disease severity, particularly in the first two weeks after the onset of symptoms.

2. Materials and Methods

2.1. Human Subjects

Signed informed consent forms were obtained from all patients as per institutional ethical approvals obtained from the Unit of Biomedical Ethics in King Abdulaziz University Hospital (Reference No 245-20), the Institutional Review Board at the Ministry of Health, Saudi Arabia (IRB Numbers: H-02-K-076-0320-279 and H-02-K-076-0420-285), and the Global Center for Mass Gatherings Medicine, Saudi Arabia (GCMGM) (No. 20/03A).

2.2. Samples

Between March 31st to June 28th, 2020, 240 blood samples from 87 patients with confirmed SARS-CoV-2 infection by real-time RT-PCR were collected between days 4 and 70 post-symptoms onset. Patients were enrolled from two hospitals in Saudi Arabia; Ohoud hospital in Madinah (n = 43 patients), and King Abdullah Medical Complex (KAMC) in Jeddah (n = 44 patients). Longitudinal blood samples were collected from mild cases (n = 46 patients, 128 samples), moderate cases (n = 13 patients, 39 samples), and severe cases (n = 28 patients, 73 samples) for serological analysis. The disease severity was categorized based on Saudi MOH guidelines. Patients with mild symptoms and no oxygen requirements or pneumonia on chest X-ray were classified as mild cases. Patients with respiratory symptoms and lung infiltrates in less than 50% of the lung field were considered as moderate cases. Patients who had one or more of the following symptoms (respiratory rate more than 30 breaths/min, blood oxygen saturation <93%, PaO2/Fio2 < 300, lung infiltrates in more than 50% of the lung field within 24–48 h) were considered as severe cases. Demographics, clinical data, and disease outcomes were retrieved from the medical records. Sera (n = 50) from healthy donors collected before COVID-19 pandemic were included as negative controls.

2.3. Cells

Baby Hamster kidney BHK-21/WI-2 cell line (Kerafast, EH1011) and African Green monkey kidney-derived Vero E6 cell line (ATCC, 1586) were cultured in Dulbecco’s modified essential medium (DMEM) contained 100 U/mL of penicillin, and 100 μg/mL of streptomycin and supplemented with either 5% for BHK-21/WI-2 or 10% fetal bovine serum (FBS) for Vero E6 cells, and maintained in a 5% CO2 environment at 37 °C.

2.4. Indirect ELISA

Recombinant SARS-CoV-2 S1 subunit (amino acids 1–685) was purchased commercially (Sino Biological, Beijing, China). Recombinant SARS-CoV-2 N protein was expressed and purified in-house as previously described [18]. Indirect S1-based or N-based ELISA were performed for the detection of specific IgG and IgM as previously described [18]. Briefly, Recombinant S1 and N proteins were used to coat 96-well high binding ELISA plates (Greiner Bio One, Monroe, NC, USA) at 1  and 4 μg/mL in phosphate-buffered saline (PBS) with 50 μL per well, respectively. After overnight incubation at 4 °C, plates were washed with PBS containing 0.05% tween-20 (PBS-T) and blocked with 5% skim milk in PBS-T buffer at 37 °C for 1 h. Plates were then washed and incubated with serum samples diluted at 1:100 in PBS-T with 5% milk for 1 h at 37 °C. After another washing, plates were incubated with HRP-conjugated goat anti-human IgG (H  +  L) or IgM antibodies (Jackson ImmunoResearch, West Grove, PA, USA) for 1 h, washed again, and incubated with TMB (3,3′,5,5′-tetramethylbenzidine) substrate (KPL, Gaithersburg, MD, USA) at room temperature for 30 min. The reaction was stopped by 100 μL per well of 0.16 M sulfuric acid, and absorbance was measured at 450  nm using the ELx808™ Absorbance Microplate Reader (BioTek, Winooski, VT, USA). Cutoff values for these ELISA were previously determined as 0.17 for IgG S1-ELISA, 0.3 for IgM S1-ELISA, 0.4 for IgG N-ELISA, and 0.55 for IgM N-ELISA [18].

2.5. Generation of rVSV-ΔG/SARS-2-S*-Luciferase Pseudovirus

The pseudovirus was produced and titrated as previously described [25]. Briefly, a T-175 tissue culture flask of BHK21/WI-2 cells were transfected with 46 μg of pcDNA expressing codon-optimized full-length SARS-CoV-2 S protein (GenBank accession number: MN908947) using LipofectamineTM 2000 transfection reagent (Invitrogen). After 24 h post-transfection, cells were infected with rVSV-ΔG/G*-luciferase (Kerafast, EH1020-PM) at a multiplicity of infection (m.o.i.) of 4. The virus inoculation was removed after 1 h and the cells were washed twice with 12 mL PBS. To completely remove any excess amount of the rVSV-ΔG/G*-luciferase, 15 mL of DMEM containing rabbit polyclonal anti VSV-G antibody were added to the cells monolayer and incubated for 24 h at 37 °C in 5% CO2 humidified incubator. Supernatant containing the generated pseudovirus was harvested 24 h post-infection and the virus titer was determined by measuring luciferase activity from 2-fold serially diluted pseudovirus on Vero E6 cells as previously described [25]. The titer of pseudovirus was expressed as a relative luciferase unit (RLU).

2.6. Pseudovirus Neutralization Assay

Neutralization assay was performed as previously reported [25]. In brief, Vero E6 cells were seeded in 96-well plates (white plate with clear bottom, COSTAR) at 2 × 104 cells/well and incubated overnight at 37 °C in 5% CO2 humidified incubator. Human sera were heat-inactivated at 56 °C for 30 min, and diluted as half log serial dilutions with an initial dilution of 1:20 in DMEM containing 5% FBS. Each serum dilution was mixed with diluted pseudovirus that yields 2 × 104 RLU and incubated at 37 °C, 5 % CO2 for 1 h in duplicates. Then, a 100 μL of the pseudovirus–serum mixtures were transferred onto Vero E6 cell monolayers and incubated at 37 °C in a 5% CO2 humidified incubator for 24 h. Cell only control (CC) and virus control (VC) were included with each assay run. After 24 h, cells were lysed, and luciferase activity was measured using the Luciferase Assay System (Promega) according to the manufacturer’s instructions. The inhibition of luciferase activity by each dilution of the serum sample was determined as follows: 100 – [(mean RLU from each sample—mean RLU from CC)/(mean RLU from VC—mean RLU from CC) × 100]. Median Inhibitory Concentration (IC50) neutralization titers were determined using four-parameter logistic (4PL) curve in GraphPad Prism V8 software (GraphPad Co., San Diego, CA, USA).

2.7. Statistical Analysis

Statistical analyses and graphical presentations were conducted with GraphPad Prism version 8.0 software (Graph-Pad Software, Inc., San Diego, CA, USA) and R (version 4.0.0). Statistical analysis of the different severity groups (mild, moderate, and severe) was conducted using one-way analysis of variance with Tukey post-hoc test to adjust for multiple comparisons between groups. Differences of antibody responses between disease outcomes (death vs survival) and between intensive care units (ICU) and non-ICU patient groups were determined by Student’s t test. Pearson’s correlation coefficient was used to assess the relationship between anti-S1 and anti-N IgG and IgM from ELISA and nAbs titers as well as between anti-S1 IgG and anti-N IgG, and anti-S1 IgM and anti-N IgM. The scatterplot smoothing lines of IgG, IgM, or nAbs titers overtime were generated using locally estimated scatterplot smoothing (LOESS) regressions with ribbons depicting the 95% confidence intervals using the ggplot package in R. All values are depicted as mean ± SD and statistical significance is reported as *, p ≤ 0.05; **, p ≤ 0.01; ***; p ≤ 0.001; and ****, p ≤ 0.0001.

3. Results

3.1. SARS-CoV-2 Induces Sustained Antibody Response in Patients despite Disease Severity

To study the kinetic of SARS-CoV-2 specific antibody responses over time in COVID-19 patients with different disease severity, we collected a total of 240 serial blood samples from 87 patients with real time RT-PCR confirmation of SARS-CoV-2 infection between days 4 and 70 post-symptom onset. The COVID-19 cohort included mild (n = 46 patients, 128 samples), moderate (n = 13 patients, 39 samples), and severe cases (n = 28 patients, 73 samples).
We initially analyzed the overall nAbs and anti-S1 and N IgG and IgM antibody responses in our COVID-19 cohort. Almost all COVID-19 patients in our study produced and maintained high titers of nAbs (Figure 1a) and mounted a high level of IgG and IgM antibodies to SARS-CoV-2 S1 (Figure 1b) and N proteins (Figure 1c) despite their disease severity (Figure S1). Furthermore, the observed high levels of nAbs and anti-S1 and N IgG peaked between days 20 and 40 and were maintained up to 70 days post-disease onset, whereas IgM levels waned over time. Nonetheless, there were some variations in the magnitude of response between patients within each category. While almost all cases especially moderate ones seroconverted during the course of samples collection and eventually produced robust nAbs, IgG, and IgM antibodies, few exceptions were observed in mild and severe categories. Specifically, no detectable anti-S1 and weak anti-N IgG antibodies were found in one mild case in addition to loss of nAbs, coinciding with reduced level of anti-S1 IgG by day 60, was observed in another mild case although high levels of anti-N IgG were detected (Figure 1d–f). Furthermore, in two severe cases no nAbs or anti-S1 IgG at days 20 and 30 were found, although low levels of anti-N IgG and IgM and anti-S1 IgM were induced (Figure 1d–f). Taken together, these data show that SARS-CoV2 S1 and N specific antibodies were readily detectable in high levels that were sustained for up to 70 days in most COVID-19 cases, with nAbs being induced in all clinical categories of COVID-19 patients.

3.2. Both S1 and N-Specific IgG Antibodies Correlate with Each Other and with nAbs in COVID-19 Patients

Next, we examined the correlation between S1-IgG, N-IgG, S1-IgM, and N-IgM antibodies and nAbs titers based on disease severity. As shown in Figure 2a, in all disease categories, levels of anti-S1 and anti-N IgG independently had a significant and strong correlation with nAbs levels (r = 0.9005, p < 0.0001 and r = 0.8899, p < 0.0001, respectively), while moderate but significant correlation was also observed between nAbs and anti-S1 IgM (r = 0.5785, p < 0.0001) and anti-N IgM (r = 0.5633, p < 0.0001).
Interestingly, anti-N IgG antibodies seem to be higher in COVID-19 patients with severe and moderate infection than mild infections, while anti-S1-IgG antibodies appeared to be similar across all disease categories (Figure 2a,b). Indeed, anti-N IgG were clearly higher in severe and moderate combined as one group than levels observed in mild cases (Figure S2). Furthermore, while both anti-S1 and anti-N IgG significantly and strongly correlate with each other (r = 0.8465, p < 0.0001), IgM against S1 and N proteins seem to have lower correlation (r = 0.4165, p < 0.0001) (Figure 2b). We also found that high levels of nAb titers correlates with higher levels of IgG response against both S1 and N proteins (Figure 2c and Figure S2c). These data indicate that anti-S1 and anti-N IgG levels strongly correlate with each other and with nAbs titers, and higher levels of anti-N responses could be associated with more severe disease.

3.3. Disease Severity Impacts Antibody Responses against SARS-CoV-2

To study the correlation between the disease severity and specific antibodies response in more detail, we compared antibody levels in the different COVID-19 disease categories. As shown in Figure 3a, COVID-19 patients who developed severe or moderate infections mounted significantly higher overall nAbs levels compared to mildly infected individuals. Additionally, S1-IgG antibodies were significantly higher in severe cases than the mild ones. Importantly, levels of anti-N IgG and IgM were significantly associated with disease severity and higher levels were induced in moderate and severe cases compared to mild infections. When stratifying patients based on their need for ICU admission or based on infection outcome, we found that nAbs as well as anti-S1 and N antibodies (IgG and IgM) were significantly higher in those who required ICU or those with fatal outcomes (Figure 3b,c). These data clearly show a significant correlation between COVID-19 disease severity and the overall humoral immune responses.
To better understand whether disease severity could impact the kinetic of antibody response in patients, we analyzed antibody responses in our COVID-19 cohort over time during the observation period. While almost all patients produced nAbs and anti-S1 antibodies with no significant difference over time except for anti-S1 IgG at late time points (Figure 4a), individuals with severe symptoms appeared to mount a significantly higher anti-N IgG and IgM during the first 15 days (acute phase) post-disease onset compared with individuals with milder disease (Figure 4a). Similarly, higher levels of anti-N IgG and IgM during the acute phase were more significantly associated with the need for ICU admission (Figure 4b). Furthermore, anti-S1 antibodies seem to be sustained at a significantly higher levels after day 30 post-symptoms onset in ICU patients compared to non-ICU patients. Comparing the kinetics of antibody response in COVID-19 patients who had fatal outcomes to those who survived the infection also showed that early induction of anti-N IgG and IgM during the first 15 days post-disease onset is indicative of fatal outcomes (Figure 4c). Additionally, we observed a significantly higher nAbs and anti-S1 IgG induction during the first 15 days in those who succumbed to infection compared to those who survived (Figure 4c). Taken together, these observations revealed that type and dynamics of specific antibodies responses correlate with disease outcome.

4. Discussion

As the COVID-19 global pandemic continues to spread, there is an urgent need to conduct in-depth analyses of immune responses in COVID-19 patients. Specifically, characterization of the type and kinetics of specific antibody responses would be crucial in advancing our understanding of the mechanisms underlying disease severity or outcome [5].
Thus far, there have been a number of studies on SARS-CoV-2 specific antibody responses in COVID-19 patients, but the interpretation of antibody responses data and association with disease severity remains not well understood. In this study we studied the characteristics and kinetics of SARS-CoV-2 specific antibody response (nAbs, IgG, and IgM) in a series of serum samples collected from a total of 87 confirmed COVID-19 hospitalized patients over a period of 70 days post-symptoms onset. A total of 240 blood samples were collected from the cohort, which included mild cases (n = 46 patients, 128 samples), moderate cases (n = 13 patients, 39 samples), and severe cases (n = 28 patients, 73 samples), in addition to pre-pandemic serum samples (50 samples). These serum samples were analyzed for specific IgG and IgM responses against SARS-CoV-2 S1 and N antigens, as well as for the nAbs levels using SARS-CoV-2 S pseudovirus-based assay.
Our data showed that anti-S1 and N IgG and IgM antibodies as well as nAbs were readily detectable across all COVID-19 disease categories and that these antibody responses can persist for at least up to 70 days post disease onset, which is the observation period in this study. We also observed that, on average, IgG antibodies for S1 and N were higher in magnitude than IgM antibodies with a peak time for both antibodies at around days 20 to 40 post infection. Moreover, IgG antibodies continued at a plateau level thereafter, while the IgM antibodies declined significantly after about 40 days post infection. Clearly, the finding that most COVID-19 patients in this study mounted high levels of sustained antibody responses is encouraging, consistent with some of the previous observations [16,21,24,26,27]. It is noteworthy that the observed decline in antibody response after about day 40 which was followed by a sustained high response until the end of the observation period should not raise concerns as infection usually leads to generation of SARS-CoV-2-specific memory B and T cells that can persist for long times and could provide protection against reinfection as recently demonstrated [28].
In our study we also show that the induction of high nAbs titers was strongly correlated with IgG response and in particular anti-S1 IgG. S1 subunit of the Spike protein mediates the binding with ACE2 receptor through the receptor binding domain (RBD). It has been shown that the RBD is the main target for most of the nAbs which may explain our finding [15,29]. This finding may also suggest the utility of using serological testing on IgG antibodies as an alternative in the absence of neutralization assays. Moreover, nAbs as well as anti-S1 and N IgG and IgM antibodies were significantly higher in those who required ICU or those with fatal outcomes than non-ICU or those who survived the infection, suggesting that high antibody responses may be considered as a risk factor of severe and fatal outcomes. Intriguingly, some patients from the different categories developed low or no humoral response. While we could not find any clinical explanation for these observations based on clinical data and history, many of these samples were from early time points (Figure 1 and Figure 4) or from patients had mild symptoms except for two severe cases. Nonetheless, such patients should be investigated further to better understand the underlying genetic and/or immunological factors that could be associated with such responses.
Importantly, when we compared the kinetics of antibody responses across the COVID-19 disease severity spectrum, we found that anti-N IgG antibodies were more likely to correlate with disease severity than anti-S1 IgG, especially when mild and severe cases were compared. During the first 15 days, we noticed significantly higher levels of anti-N IgG and IgM in severe and moderate cases compared to mild infection and in ICU compared to non-ICU patients. Analysis of SARS-CoV-2 specific antibodies response in a cohort of COVID-19 patients in China showed a potential overall correlation between total antibodies (IgG and IgM) against the RBD and disease severity [30]. Interestingly, a recent study showed an opposite trend of increased IgG antibodies in milder cases compared to more severe COVID-19 patients [31]. However, this study focused on the correlation between antibodies levels and viral RNA shedding in patients and not necessarily disease severity. In our analysis, however, it was clear that early after disease onset, patients who had severe symptoms including those who were admitted to ICU and who had fatal outcomes showed significantly higher anti-N IgG and IgM antibodies. Considering that N protein is one of the highly expressed proteins, these high levels of anti-N antibodies could indicate a high virus replication in severely infected individuals compared to non-severe patients. In addition, since anti-N antibodies are usually non-neutralizing, the high levels of these antibodies in severe patients may have contributed to the disease severity and immunopathology. It is worth noting that presence of these kind of antibodies may raise concern of antibody-dependent disease enhancement which was reported in SARS patients and in feline coronavirus infections [32,33,34]. Nonetheless, our data clearly show a trend of high levels of anti-N antibodies in the first two weeks after symptoms onset in COVID-19 patients with moderate and severe symptoms. While these data may indicate the levels of anti-N antibodies could be an indicator of disease severity and could be used as a prognostic marker of disease outcomes, no cause-and-effect relationship can be deduced from the data which warrants further studies.
In summary, our data indicates that COVID-19 patients can generate high levels of antibody responses that might last for a long period despite their disease severity. Our results also show that severe infections are associated with overall stronger anti-S1 and -N IgG and IgM antibodies as well as nAbs compared to non-severe cases. Importantly, significantly higher levels of antibodies, particularly anti-N IgG and IgM, were observed during the acute phase of the disease in severely ill cases which required ICU admission or succumbed to infection, suggesting they could be helpful prognostic markers of COVID-19. Nonetheless, additional studies are clearly needed to further understand the underlying molecular mechanisms by which antibody levels could impact disease course or prognosis.

Supplementary Materials

The following are available online at https://www.mdpi.com/1999-4915/12/12/1390/s1, Figure S1: Kinetics of SARS-CoV-2 specific antibodies over time based on disease severity. Figure S2: Relationship between S1 and N-specific IgG and IgM and neutralizing titer based on disease severity.

Author Contributions

A.M.H., A.A., S.A.M., and M.A.A. contributed equally to this work. A.M.H., A.A., M.A.A., S.H., T.S.A., S.S.A., S.A.A., K.A.A., H.I.H., R.M.A., R.H.A., M.-Z.E., A.M.H., performed and optimized experiments and analyzed data. A.A., M.A.A., X.L. and A.M.H. drafted the manuscript. R.Y.A., A.A.A., W.H.A., A.A., F.S.A., A.A.K., A.A.A.-S., A.B.M., N.A.M.A., A.P., N.K.A., A.M.H. and X.L. conceptualized, and contributed to the experimental design and analyses, and edited manuscript drafts. All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by grant from the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia, project number 436.

Acknowledgments

The authors extend their appreciation to the Deputyship for Research and Innovation, Ministry of Education in Saudi Arabia for funding this research.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Zhu, N.; Zhang, D.; Wang, W.; Li, X.; Yang, B.; Song, J.; Zhao, X.; Huang, B.; Shi, W.; Lu, R.; et al. A Novel Coronavirus from Patients with Pneumonia in China, 2019. N. Engl. J. Med. 2020, 382, 727–733. [Google Scholar] [CrossRef]
  2. Lu, R.; Zhao, X.; Li, J.; Niu, P.; Yang, B.; Wu, H.; Wang, W.; Song, H.; Huang, B.; Zhu, N.; et al. Genomic characterisation and epidemiology of 2019 novel coronavirus: Implications for virus origins and receptor binding. Lancet 2020, 395, 565–574. [Google Scholar] [CrossRef] [Green Version]
  3. Wu, F.; Zhao, S.; Yu, B.; Chen, Y.M.; Wang, W.; Song, Z.G.; Hu, Y.; Tao, Z.W.; Tian, J.H.; Pei, Y.Y.; et al. A new coronavirus associated with human respiratory disease in China. Nature 2020, 579, 265–269. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Cui, J.; Li, F.; Shi, Z.L. Origin and evolution of pathogenic coronaviruses. Nat. Rev. Genet. 2019, 17, 181–192. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Rodriguez-Morales, A.J.; Cardona-Ospina, J.A.; Gutiérrez-Ocampo, E.; Villamizar-Peña, R.; Holguin-Rivera, Y.; Escalera-Antezana, J.P.; Alvarado-Arnez, L.E.; Bonilla-Aldana, D.K.; Franco-Paredes, C.; Henao-Martinez, A.F.; et al. Clinical, laboratory and imaging features of COVID-19: A systematic review and meta-analysis. Travel Med. Infect. Dis. 2020, 34, 101623. [Google Scholar] [CrossRef] [PubMed]
  6. Wang, Y.; Liu, Y.; Liu, L.; Wang, X.; Luo, N.; Li, L. Clinical Outcomes in 55 Patients With Severe Acute Respiratory Syndrome Coronavirus 2 Who Were Asymptomatic at Hospital Admission in Shenzhen, China. J. Infect. Dis. 2020, 221, 1770–1774. [Google Scholar] [CrossRef]
  7. Jin, Y.; Yang, H.; Ji, W.; Wu, W.D.; Chen, S.; Zhang, W.; Duan, G.C. Virology, Epidemiology, Pathogenesis, and Control of COVID-19. Viruses 2020, 12, 372. [Google Scholar] [CrossRef] [Green Version]
  8. Englmeier, L. A theory on SARS-COV-2 susceptibility: Reduced TLR7-activity as a mechanistic link between men, obese and elderly. J. Biol. Regul. Homeost. Agents 2020, 34. [Google Scholar] [CrossRef]
  9. Castagnoli, R.; Votto, M.; Licari, A.; Brambilla, I.; Bruno, R.; Perlini, S.; Rovida, F.; Baldanti, F.; Marseglia, G.L. Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Infection in Children and Adolescents: A Systematic Review. JAMA Pediatr. 2020, 174, 882–889. [Google Scholar] [CrossRef] [Green Version]
  10. Cai, H. Sex difference and smoking predisposition in patients with COVID-19. Lancet Respir. Med. 2020, 8, 20. [Google Scholar] [CrossRef]
  11. Tomaras, G.D.; Haynes, B.F. HIV-1-specific antibody responses during acute and chronic HIV-1 infection. Curr. Opin. HIV AIDS 2009, 4, 373–379. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  12. Cobey, S.; Hensley, S.E. Immune history and influenza virus susceptibility. Curr. Opin. Virol. 2017, 22, 105–111. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Saphire, E.O.; Schendel, S.L.; Gunn, B.M.; Milligan, J.C.; Alter, G. Antibody-mediated protection against Ebola virus. Nat. Immunol. 2018, 19, 1169–1178. [Google Scholar] [CrossRef] [PubMed]
  14. Yan, R.; Zhang, Y.; Li, Y.; Xia, L.; Guo, Y.; Zhou, Q. Structural basis for the recognition of SARS-CoV-2 by full-length human ACE2. Science 2020, 367, 1444–1448. [Google Scholar] [CrossRef] [Green Version]
  15. Du, L.; He, Y.; Zhou, Y.; Liu, S.; Zheng, B.J.; Jiang, S. The spike protein of SARS-CoV—A target for vaccine and therapeutic development. Nat. Rev. Genet. 2009, 7, 226–236. [Google Scholar] [CrossRef]
  16. Sun, B.; Feng, Y.; Mo, X.; Zheng, P.; Wang, Q.; Li, P.; Peng, P.; Liu, X.; Chen, Z.; Huang, H.; et al. Kinetics of SARS-CoV-2 specific IgM and IgG responses in COVID-19 patients. Emerg. Microbes Infect. 2020, 9, 940–948. [Google Scholar] [CrossRef]
  17. Trivedi, S.; Miao, C.; Al-Abdallat, M.M.; Haddadin, A.; Alqasrawi, S.; Iblan, I.; Nsour, M.A.; Alsanouri, T.; Ali, S.S.; Rha, B.; et al. Inclusion of MERS-spike protein ELISA in algorithm to determine serologic evidence of MERS-CoV infection. J. Med. Virol. 2018, 90, 367–371. [Google Scholar] [CrossRef] [Green Version]
  18. Algaissi, A.; Alfaleh, A.A.; Hala, S.; Abujamel, T.S.; Alamri, S.S.; Almahboub, S.A.; Alluhaybi, K.A.; Hobani, H.I.; Asulaiman, R.M.; AlHarbi, R.H.; et al. SARS-CoV-2 S1 and N-Based Serological Assays Reveal Rapid Seroconversion and Induction of Specific Antibody Response in COVID-19 Patients. Sci. Rep 2020, in press. [Google Scholar] [CrossRef]
  19. Deming, D.; Sheahan, T.; Heise, M.; Yount, B.; Davis, N.; Sims, A.; Suthar, M.; Harkema, J.; Whitmore, A.; Pickles, R.; et al. Vaccine Efficacy in Senescent Mice Challenged with Recombinant SARS-CoV Bearing Epidemic and Zoonotic Spike Variants. PLoS Med. 2006, 3, 525. [Google Scholar] [CrossRef] [Green Version]
  20. Lou, B.; Li, T.D.; Zheng, S.F.; Su, Y.Y.; Li, Z.Y.; Liu, W.; Yu, F.; Ge, S.; Zou, Q.D.; Yuan, Q.; et al. Serology characteristics of SARS-CoV-2 infection after exposure and post-symptom onset. Eur. Respir. J. 2020, 56, 2000763. [Google Scholar] [CrossRef]
  21. Yongchen, Z.; Shen, H.; Wang, X.; Shi, X.; Li, Y.; Yan, J.; Chen, Y.; Gu, B. Different longitudinal patterns of nucleic acid and serology testing results based on disease severity of COVID-19 patients. Emerg. Microbes Infect. 2020, 9, 833–836. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  22. Yong, S.E.F.; Anderson, D.E.; E Wei, W.; Pang, J.; Ni Chia, W.; Tan, C.W.; Teoh, Y.L.; Rajendram, P.; Toh, M.P.H.S.; Poh, C.; et al. Connecting clusters of COVID-19: An epidemiological and serological investigation. Lancet Infect. Dis. 2020, 20, 809–815. [Google Scholar] [CrossRef]
  23. Xu, Y.; Xiao, M.; Liu, X.; Xu, S.; Du, T.; Xu, J.; Yang, Q.; Xu, Y.; Han, Y.; Li, T.; et al. Significance of serology testing to assist timely diagnosis of SARS-CoV-2 infections: Implication from a family cluster. Emerg. Microbes Infect. 2020, 9, 924–927. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  24. Iyer, A.S.; Jones, F.K.; Nodoushani, A.; Kelly, M.; Becker, M.; Slater, D.; Mills, R.; Teng, E.; Kamruzzaman, M.; Garcia-Beltran, W.F.; et al. Dynamics and significance of the antibody response to SARS-CoV-2 infection. medRxiv 2020. [Google Scholar] [CrossRef]
  25. Almahboub, S.A.; Algaissi, A.; AlFaleh, M.A.; ElAssouli, M.Z.; Hashem, A.M. Evaluation of Neutralizing Antibodies Against Highly Pathogenic Coronaviruses: A Detailed Protocol for a Rapid Evaluation of Neutralizing Antibodies Using Vesicular Stomatitis Virus Pseudovirus-Based Assay. Front. Microbiol. 2020, 11. [Google Scholar] [CrossRef]
  26. Long, Q.; Liu, B.Z.; Deng, H.J.; Wu, G.C.; Deng, K.; Chen, Y.K.; Liao, P.; Qiu, J.F.; Lin, Y.; Cai, X.F.; et al. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat. Med. 2020, 26, 845–848. [Google Scholar] [CrossRef]
  27. Wajnberg, A.; Amanat, F.; Firpo, A.; Altman, D.; Bailey, M.; Mansour, M.; McMahon, M.; Meade, P.; Mendu, D.R.; Muellers, K.; et al. SARS-CoV-2 infection induces robust, neutralizing antibody responses that are stable for at least three months. medRxiv 2020. [Google Scholar] [CrossRef]
  28. Rodda, L.B.; Netland, J.; Shehata, L.; Pruner, K.B.; Morawski, P.A.; Thouvenel, C.D.; Takehara, K.K.; Eggenberger, J.; Hemman, E.A.; Waterman, H.R.; et al. Functional SARS-CoV-2-specific immune memory persists after mild COVID-19. medRxiv 2020. [Google Scholar] [CrossRef]
  29. Tian, X.; Li, C.; Huang, A.; Xia, S.; Lu, S.; Shi, Z.L.; Lu, L.; Jiang, S.; Yang, Z.; Wu, Y.; et al. Potent binding of 2019 novel coronavirus spike protein by a SARS coronavirus-specific human monoclonal antibody. Emerg. Microbes Infect 2020, 9, 382–385. [Google Scholar] [CrossRef] [Green Version]
  30. Zhao, J.; Yuan, Q.; Wang, H.; Liu, W.; Liao, X.; Su, Y.; Wang, X.; Yuan, J.; Li, T.; Li, J.; et al. Antibody Responses to SARS-CoV-2 in Patients With Novel Coronavirus Disease 2019. Clin. Infect. Dis. 2020, 344. [Google Scholar] [CrossRef]
  31. Wu, J.; Liang, B.; Chen, C.; Wang, H.; Fang, Y.; Shen, S.; Yang, X.; Wang, B.; Chen, L.; Chen, Q.; et al. SARS-CoV-2 infection induces sustained humoral immune responses in convalescent patients following symptomatic COVID-19. medRxiv 2020. [Google Scholar] [CrossRef]
  32. Hohdatsu, T.; Yamada, M.; Tominaga, R.; Makino, K.; Kida, K.; Koyama, H. Antibody-Dependent Enhancement of Feline Infectious Peritonitis Virus Infection in Feline Alveolar Macrophages and Human Monocyte Cell Line U937 by Serum of Cats Experimentally or Naturally Infected with Feline Coronavirus. J. Vet. Med. Sci. 1998, 60, 49–55. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  33. Liu, L.; Wei, Q.; Lin, Q.; Fang, J.; Wang, H.; Kwok, H.; Tang, H.; Nishiura, K.; Peng, J.; Tan, Z.; et al. Anti–spike IgG causes severe acute lung injury by skewing macrophage responses during acute SARS-CoV infection. JCI Insight 2019, 4, 123158. [Google Scholar] [CrossRef] [PubMed]
  34. Yip, M.S.; Leung, H.L.; Li, P.H.; Cheung, C.Y.; Dutry, I.; Li, D.; Daëron, M.; Bruzzone, R.; Peiris, J.S.; Jaume, M. Antibody-dependent enhancement of SARS coronavirus infection and its role in the pathogenesis of SARS. Hong Kong Med. J. 2016, 22, 25–31. [Google Scholar] [CrossRef] [Green Version]
Figure 1. Kinetics of SARS-CoV-2 specific antibodies over time based on disease severity. (a) Neutralizing antibodies (nAbs) titers (IC50) and OD values of (b) S1-specific IgG and IgM and (c) N-specific IgG and IgM are plotted against the days post disease onset. The line and the ribbon show the mean expected from a locally estimated scatterplot smoothing (LOESS) regression model with 95% confidence interval. The colored circles indicate disease severity with green circles represent mild cases, blue circles represent moderate cases, and red circles represent severe cases. OD = optical density. Changes in (d) nAbs titers (IC50) and OD values of (e) S1-specific IgG and (f) N-specific IgG for each patient are shown over time. Data from first sample and last collected sample are shown for patients with samples that were at least 7 days apart.
Figure 1. Kinetics of SARS-CoV-2 specific antibodies over time based on disease severity. (a) Neutralizing antibodies (nAbs) titers (IC50) and OD values of (b) S1-specific IgG and IgM and (c) N-specific IgG and IgM are plotted against the days post disease onset. The line and the ribbon show the mean expected from a locally estimated scatterplot smoothing (LOESS) regression model with 95% confidence interval. The colored circles indicate disease severity with green circles represent mild cases, blue circles represent moderate cases, and red circles represent severe cases. OD = optical density. Changes in (d) nAbs titers (IC50) and OD values of (e) S1-specific IgG and (f) N-specific IgG for each patient are shown over time. Data from first sample and last collected sample are shown for patients with samples that were at least 7 days apart.
Viruses 12 01390 g001
Figure 2. Relationship between S1 and N-specific IgG and IgM and neutralizing titer based on disease severity. (a) Pearson’s correlation between S1-IgG, N-IgG, S1-IgM, and N-IgM and neutralizing titer based on disease severity. Scatter plots were generated using OD values (y-axis) versus IC50 (x-axis) from each individual serum sample with colored circles indicate disease severity. (b) Pearson’s correlation between IgG response against S1 and N proteins and IgM response against S1 and N proteins based on disease severity. Scatter plots were generated using anti-N OD values (y-axis) versus anti-S1 OD values (x-axis) from each individual serum sample with colored circles indicate disease severity. Pearson’s correlation between (c) IgG response against S1 and N proteins and (d) IgM response against S1 and N proteins based on neutralizing titer. Colored circles indicate neutralizing titers (IC50). The dotted lines represent the cut-off of each assay. OD = optical density.
Figure 2. Relationship between S1 and N-specific IgG and IgM and neutralizing titer based on disease severity. (a) Pearson’s correlation between S1-IgG, N-IgG, S1-IgM, and N-IgM and neutralizing titer based on disease severity. Scatter plots were generated using OD values (y-axis) versus IC50 (x-axis) from each individual serum sample with colored circles indicate disease severity. (b) Pearson’s correlation between IgG response against S1 and N proteins and IgM response against S1 and N proteins based on disease severity. Scatter plots were generated using anti-N OD values (y-axis) versus anti-S1 OD values (x-axis) from each individual serum sample with colored circles indicate disease severity. Pearson’s correlation between (c) IgG response against S1 and N proteins and (d) IgM response against S1 and N proteins based on neutralizing titer. Colored circles indicate neutralizing titers (IC50). The dotted lines represent the cut-off of each assay. OD = optical density.
Viruses 12 01390 g002
Figure 3. Overall impact of disease severity on antibody responses against SARS-CoV-2. Comparison of neutralizing titer (IC50) and OD values of S1-IgG, N-IgG, S1-IgM, and N-IgM for (a) individuals from the different disease severity categories; (b) individuals who survived or died; and (c) individuals who required intensive care units (ICU) admission or not. Statistical significance was determined using one-way analysis of variance with Tukey post-hoc test in (a) and using Student’s t test in (b) and (c). Mean with SD are shown, and statistical significance is represented as *, p ≤ 0.05; **, p ≤ 0.01; ***; p ≤ 0.001; and ****, p ≤ 0.0001.
Figure 3. Overall impact of disease severity on antibody responses against SARS-CoV-2. Comparison of neutralizing titer (IC50) and OD values of S1-IgG, N-IgG, S1-IgM, and N-IgM for (a) individuals from the different disease severity categories; (b) individuals who survived or died; and (c) individuals who required intensive care units (ICU) admission or not. Statistical significance was determined using one-way analysis of variance with Tukey post-hoc test in (a) and using Student’s t test in (b) and (c). Mean with SD are shown, and statistical significance is represented as *, p ≤ 0.05; **, p ≤ 0.01; ***; p ≤ 0.001; and ****, p ≤ 0.0001.
Viruses 12 01390 g003
Figure 4. Impact of disease severity on kinetics of antibody responses against SARS-CoV-2. Comparison of neutralizing titer (IC50) and OD values of S1-IgG, N-IgG, S1-IgM, and N-IgM for (a) individuals from the different disease severity categories; (b) individuals who survived or died; and (c) individuals who required ICU admission or not. Data are shown based on time post disease onset. Statistical significance was determined using one-way analysis of variance with Tukey post-hoc test in (a) and using Student’s t test in (b) and (c). Mean with SD are shown, and statistical significance is represented as *, p ≤ 0.05; **, p ≤ 0.01; and ****, p ≤ 0.0001.
Figure 4. Impact of disease severity on kinetics of antibody responses against SARS-CoV-2. Comparison of neutralizing titer (IC50) and OD values of S1-IgG, N-IgG, S1-IgM, and N-IgM for (a) individuals from the different disease severity categories; (b) individuals who survived or died; and (c) individuals who required ICU admission or not. Data are shown based on time post disease onset. Statistical significance was determined using one-way analysis of variance with Tukey post-hoc test in (a) and using Student’s t test in (b) and (c). Mean with SD are shown, and statistical significance is represented as *, p ≤ 0.05; **, p ≤ 0.01; and ****, p ≤ 0.0001.
Viruses 12 01390 g004
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Hashem, A.M.; Algaissi, A.; Almahboub, S.A.; Alfaleh, M.A.; Abujamel, T.S.; Alamri, S.S.; Alluhaybi, K.A.; Hobani, H.I.; AlHarbi, R.H.; Alsulaiman, R.M.; et al. Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients. Viruses 2020, 12, 1390. https://doi.org/10.3390/v12121390

AMA Style

Hashem AM, Algaissi A, Almahboub SA, Alfaleh MA, Abujamel TS, Alamri SS, Alluhaybi KA, Hobani HI, AlHarbi RH, Alsulaiman RM, et al. Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients. Viruses. 2020; 12(12):1390. https://doi.org/10.3390/v12121390

Chicago/Turabian Style

Hashem, Anwar M., Abdullah Algaissi, Sarah A. Almahboub, Mohamed A. Alfaleh, Turki S. Abujamel, Sawsan S. Alamri, Khalid A. Alluhaybi, Haya I. Hobani, Rahaf H. AlHarbi, Reem M. Alsulaiman, and et al. 2020. "Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients" Viruses 12, no. 12: 1390. https://doi.org/10.3390/v12121390

APA Style

Hashem, A. M., Algaissi, A., Almahboub, S. A., Alfaleh, M. A., Abujamel, T. S., Alamri, S. S., Alluhaybi, K. A., Hobani, H. I., AlHarbi, R. H., Alsulaiman, R. M., ElAssouli, M. -Z., Hala, S., Alharbi, N. K., Alhabbab, R. Y., AlSaieedi, A. A., Abdulaal, W. H., Bukhari, A., AL-Somali, A. A., Alofi, F. S., ... Li, X. (2020). Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients. Viruses, 12(12), 1390. https://doi.org/10.3390/v12121390

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop