Next Article in Journal
Utility of Lean Body Mass Equations and Body Mass Index for Predicting Outcomes in Critically Ill Adults with Sepsis: A Retrospective Study
Previous Article in Journal
Effectiveness of Omega-3 Fatty Acid Supplementation in Improving the Metabolic and Inflammatory Profiles of Mexican Adults Hospitalized with COVID-19
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Diseases
Volume 12
Issue 1
10.3390/diseases12010029
diseases-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Plaque Reduction Neutralization Test (PRNT) Accuracy in Evaluating Humoral Immune Response to SARS-CoV-2

by
Ingrid Siciliano Horbach
1,2,
Adriana de Souza Azevedo
1,
Waleska Dias Schwarcz
1,
Nathalia dos Santos Alves
1,
Brenda de Moura Dias
1,
Bruno Pimenta Setatino
1,
Luma da Cruz Moura
1,
Ariane Faria de Souza
1,
Caio Bidueira Denani
1,2,
Stephanie Almeida da Silva
3,
Thiago Goes Pimentel
4,
Victor de Oliveira Silva Ferreira
5,
Tamiris Azamor
6,
Ana Paula Dinis Ano Bom
6,
Maria da Penha Gomes Gouvea
7,
José Geraldo Mill
7,
Valéria Valim
7,
Jessica Polese
8,
Ana Carolina Campi-Azevedo
9,
Vanessa Peruhype-Magalhães
9,
Andréa Teixeira-Carvalho
9,
Olindo Assis Martins-Filho
9,
Sheila Maria Barbosa de Lima
10,*,† and
Ivanildo Pedro de Sousa Junior
11,*,†add Show full author list remove Hide full author list
1
Laboratório de Análise Imunomolecular, Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
2
Programa de Pós-Graduação em Medicina Tropical, Instituto Oswaldo Cruz, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
3
Laboratório de Tecnologia Virológica, Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
4
Núcleo de Apoio Administrativo VDINV, Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
5
Seção de Validação Analítica, Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
6
Laboratório de Tecnologia Imunológica, Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
7
Hospital Universitário Cassiano Antônio Moraes, Universidade Federal do Espírito Santo (HUCAM-UFES/EBSERH), Vitória 29041-295, Brazil
8
Programa de Pós-Graduação em Ciências Fisiológicas da Universidade Federal do Espírito Santo, Vitória 29500-000, Brazil
9
Grupo Integrado de Pesquisa em Biomarcadores, Instituto René Rachou, FIOCRUZ-Minas, Belo Horizonte 30190-002, Brazil
10
Departamento de Desenvolvimento Experimental e Pré-clínico (DEDEP), Instituto de Tecnologia em Imunobiológicos Bio-Manguinhos, Fundação Oswaldo Cruz, Rio de Janeiro 21040-900, Brazil
11
Laboratório de Virologia e Parasitologia Molecular, Instituto Oswaldo Cruz/Fiocruz, Rio de Janeiro 21040-900, Brazil
 
add Show full affiliation list
 
remove Hide full affiliation list
*
Authors to whom correspondence should be addressed.
†
These authors contributed equally to this work.
Diseases 2024, 12(1), 29; https://doi.org/10.3390/diseases12010029
Submission received: 12 December 2023 / Revised: 13 January 2024 / Accepted: 15 January 2024 / Published: 18 January 2024
Browse Figures
Review Reports Versions Notes

Abstract

:
Massive vaccination positively impacted the SARS-CoV-2 pandemic, being a strategy to increase the titers of neutralizing antibodies (NAbs) in the population. Assessing NAb levels and understanding the kinetics of NAb responses is critical for evaluating immune protection. In this study, we optimized and validated a PRNT50 assay to assess 50% virus neutralization and evaluated its accuracy to measure NAbs to the original strain or variant of SARS-CoV-2. The optimal settings were selected, such as the cell (2 × 105 cells/well) and CMC (1.5%) concentrations and the viral input (~60 PFU/well) for PRNT-SARS-CoV-2 with cut-off point = 1.64 log5 based on the ROC curve (AUC = 0.999). The validated PRNT-SARS-CoV-2 assay presented high accuracy with an intraassay precision of 100% for testing samples with different NAb levels (low, medium, and high titers). The method displays high selectivity without cross-reactivity with dengue (DENV), measles (MV), zika (ZIKV), and yellow fever (YFV) viruses. In addition, the standardized PRNT-SARS-CoV-2 assay presented robustness when submitted to controlled variations. The validated PRNT assay was employed to test over 1000 specimens from subjects with positive or negative diagnoses for SARS-CoV-2 infection. Patients with severe COVID-19 exhibited higher levels of NAbs than those presenting mild symptoms for both the Wuhan strain and Omicron. In conclusion, this study provides a detailed description of an optimized and validated PRNT50 assay to monitor immune protection and to subsidize surveillance policies applied to epidemiologic studies of COVID-19.

1. Introduction

Globally, the health systems have been significantly impacted since the emergence of Coronavirus Disease 19 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), in late 2019 [1]. Most existing SARS-CoV-2 cases are considered mild or asymptomatic, mainly in a low-risk population [2]. However, patients from the risk group (elderly, obese, and patients with pre-existing co-morbidities) can show more severe symptoms [3]. In addition, the newly emerging SARS-CoV-2 variants of concern or interest (VOCs or VOIs) have exhibited mutations, primarily in the spike protein, which mediates the virus attachment to the host cell surface receptors by allowing the fusion of the membranes [4]. These mutations may impact various aspects of the virus’s biology, such as the pathogenicity and antigenicity, leading to its potential escape from current neutralizing antibodies [5,6].
Several COVID-19 vaccines have been developed, and they have been shown to be safe and effective [7,8], reducing the risk of severe COVID-19. The natural infection and vaccination could elicit specific neutralizing antibodies (NAbs) that can diminish or completely block viral infection, being an essential aspect of adaptive immunity [9]. Neutralizing antibodies for SARS-CoV-2 appear a few days after those episodes [10,11]. They can persist for an extended period [12,13] or decline subsequently [14,15], leading to decreased protection and an increased risk of reinfection by SARS-CoV-2.
Since the first demonstration of the presence of viral plaques on a chicken embryo fibroblast monolayer for Newcastle Disease virus and Western Equine Encephalomyelitis [16], significant progress has been achieved in this area with successful applications in virus isolation [17]. As a result, plaque reduction neutralization tests (PRNTs) for a variety of viruses such as dengue (DENV) [18], mumps (MuV) [19], yellow fever (YFV) [20], measles (MV) [21], poliovirus (PV) [22], Lassa (LASV) [23], and Japanese encephalitis (JEV) [24] were developed. The PRNT assay accesses the ability of diluted serum samples to block viral infection on cell monolayers by a known quantity of the virus, determining the level of virus-specific NAbs [25]. The PRNT is the gold standard among serological methods [26,27] once it is a more sensitive quantitative assay.
The detection of NAb responses to SARS-CoV-2 is critical for tailoring current surveillance strategies of infection control and immunity scanning, and so is the guidance of public authorities to conduct the subsequent vaccination campaigns. Although the serologic correlate of protection for COVID-19 disease is still unclear, Nab detection remains an essential tool for evaluating vaccine effectiveness [28,29,30]. The development and implementation of immunoassays is a route of significant interest for increasing the accuracy and reliability of quantitative tests that measure neutralizing antibodies [31]. Additionally, validated serological assays are also critical for clinical trials of vaccines in development, screenings of therapeutic antibodies, and seroprevalence studies [32].
In the present report, we describe the development, optimization, and validation of the 50% Plaque Reduction Neutralizing Test (PRNT50) to determine specific neutralizing antibodies against SARS-CoV-2 from volunteer serum samples with/without natural infection and/or actively immunized to COVID-19. Moreover, the validated PRNT assay was employed to compare the levels of NAbs in patients with mild and severe disease from the original Wuhan strain and the Omicron variant.

2. Materials and Methods

2.1. Ethics Statements and Sample Collection

The Ethics Committee approved all procedures regarding the acquisition of the samples (CAAE# 34728920.4.0000.5262, #30846920.7.0000.0008, and #15120613.4.0000.5262). All participants gave the authorization to use their samples, and the methods applied were carried out respecting the applicable regulations and guidelines. A total of 982 (an open cohort with 369 samples and 503 samples collected at different time points before and after vaccination [33] and 110 participants who had medical attention due to SARS-CoV-2 infection) blood samples were collected from Rio de Janeiro and Espírito Santo, Brazil, between June 2020 and February 2021. Additionally, 40 serums from healthy individuals were collected before the COVID-19 pandemic (2014) and used as negative controls, comprising 1022 samples. The samples were centrifuged, and the supernatant was collected and inactivated (56 °C for 30 min) then stored at −30 °C. As a reference control, one positive and one negative sample were selected from individuals with positive and negative RT-qPCR for SARS-CoV-2 (Real-Time PCR Charité Protocol -E/RP- SARS-CoV-2), respectively.

2.2. Virus and Cell Lines

SARS-CoV-2 viruses (Wuhan and Omicron) were kindly provided by the Laboratory of Respiratory Viruses, Exanthematics, Enteroviruses and Viral Emergencies (LVRE) at IOC/Fiocruz (SISGEN A994A37). Virus propagation (Wuhan strain, 10.10 log5 PFU/mL; Omicron variant, 9.32 log5 PFU/mL) was carried out in Vero E6 cells (CRL-1586, ATCC, Manassas, VA, USA) maintained in 199 medium with Earle salts (E199, Sigma, Livonia, MI, USA) buffered with sodium bicarbonate and supplemented with 5% fetal bovine serum (FBS, Invitrogen, Waltham, MA, USA), as described previously [34]. The African green monkey Vero CCL81 (ATCC, Manassas, VA, USA) cells were cultivated in 24-well plates (120,000 or 200,000 cells/well) one day before the assay (or two days before, in the case of robustness assay for validation step).

2.3. Plaque Reduction Neutralization Test (PRNT) Optimization

The PRNT (Figure 1) was optimized to detect the presence of neutralizing antibodies against SARS-CoV-2 in the human serum. For this purpose, 15 µL of previously inactivated (56 °C, 30 min) serum samples were serially diluted in culture medium from 1:10 to 1:31,250 in reciprocal serum dilution or 1:1.43 to 1:6.43 in log5) in 60 µL of 199 media supplemented with 5% FBS, followed by the addition of 60 µL (5-fold dilution factor) of SARS-CoV-2 tested with different Plaque-Forming Unit (PFU) inputs (100, 70, and 60 PFU) and incubated at 37 °C in 5% CO2 for 1 h. The virus–serum mixtures were added onto a confluent monolayer of Vero CCL-81 (ATCC, USA) cells (120,000 or 200,000 cells/well) and incubated for 1 h at 37 °C in a 5% CO2 incubator. The inoculum was discarded, and the cells were then covered with 1 mL of 199 Medium supplemented with 5% FBS and 1.5, 2.0, or 2.5% carboxymethylcellulose (CMC). After that, the cells were incubated for 3 days at 37 °C in 5% CO2 before fixation with 1.25% formalin solution (vol/vol). Plaques were counted after the cells had been stained with a 0.04% crystal violet dye solution. The results were ex-pressed in reciprocal serum dilution or log5. The cut-off point of 1:14 (1.64 log5) was applied to set apart seropositive samples from seronegative samples, and the upper limit of quantification of positive samples was 1:31,250 (6.43 log5). PRNT assays were handled in a BSL-3 laboratory Multi-user Research Facility of Biosafety Platform BSL3-HPP, Oswaldo Cruz Institute, Oswaldo Cruz Foundation, Rio de Janeiro, Brazil, following the approved international laboratory biosafety guidelines [35].

2.4. Validation of Analytical Procedure—PRNT-SARS-CoV-2

The method validation adhered to the fitness-for-purpose concept [36], aligning with the unique characteristics of the assay, the nature of the measurand, and the clinical relevance of the analysis. This strategic approach was adopted following a comprehensive evaluation of pertinent international and national regulations, as well as technical requirements [37,38,39,40,41,42,43]. The validation was performed by evaluating the following analytical parameters: selectivity, precision, accuracy, and robustness, which are detailed below and summarized in Table 1, as well as the acceptance criteria applied for each step. The titer measurement data arisen during the validation were log-transformed considering the five-fold dilution factor of the assay [44,45]. Thus, the base logarithm used was 5. For all parameters evaluated, the acceptance criteria were defined by the absolute difference of the log-transformed data. Considering that the 4-fold increase in antibody response is used to assess immunogenicity [46,47,48], each validation criteria parameter was defined to assure that inherent PRNT variability did not impact the measurement quality.

2.4.1. Selectivity

Selectivity is the extent to which the method can determine a particular compound in the analyzed samples without interference from matrix components [49]. The goal was evaluated to confirm that the assay was free of potential interference including endogenous matrix components, mainly other neutralization antibodies. Selectivity was evaluated in a single assay testing a set of serum samples in triplicate: SARS-CoV-2-positive and -negative serum.
Positive samples for DENV, ZIKV, YFV, and MV (but negative for SARS-CoV-2) were tested, and the acceptance for these samples was negative (<1:14).
To assess the absence of interference in the quantification of neutralizing antibodies against SARS-CoV-2, a positive sample for SARS-CoV-2 was prepared, diluted (1:2) in negative and positive samples for neutralizing antibodies to DENV, ZIKV, YFV, and MV to check whether the presence of antibodies against other viral targets alters the result of the quantification of neutralizing antibodies against SARS-CoV-2. PRNT is considered selective if the presence of NAbs to other viral targets does not change the NAb titer to SARS-CoV-2 (up to two-fold that of equivalent 0.43 log5).

2.4.2. Accuracy

PRNT-SARS-CoV-2 accuracy was evaluated by the difference between the experimental and the expected titer of an internal standard serum.
The internal standard serum was analyzed undiluted and diluted in 1:2, 1:4, 1:16, and 1:32 in negative serum, in 5 replicates in 3 runs. Accuracy was considered acceptable if the absolute difference between the experimental and the expected GMT was up 3-fold (that is equivalent to 0.68 log5) for at least 80% of the measurement.

2.4.3. Precision

The PRNT precision was evaluated in terms of intraassay variability characteristics (agreement of results provided by a single run for the same homogeneous sample) and interassay variation (agreement of findings generated by repeated assays for the same homogeneous sample) that represented the test reproducibility. For this purpose, 18 positive SARS-CoV-2 sera were used: 6 sera with low titer (>14 to 100 or 1.64 to 2.86 log5), 6 sera with medium (101 to 500 or 2.87 to 3.86 log5) titer, and 6 sera with high titer (>500 or >3.86 log5) for NAbs. Each sample was evaluated in triplicate in 3 independent runs executed on different days.
Results for intraassay and interassay precision were accepted if the absolute difference between mean and the individual results was up 4-fold (that is equivalent to 0.86 log5) for at least 80% of the measurement.

2.4.4. Robustness

The robustness of the SARS-CoV-2 PRNT was evaluated to determine its capacity to provide analytical data maintaining acceptable accuracy and precision under minor assay variations: I. Adsorption time: 30, 60, and 90 min; II. Cell monolayer preparation: Condition 1 (N-1)—24 h before experiment with 200,000 Vero CCL-81 cells per well, and Condition 2 (N-2)—48 h before experiment with 100,000 Vero CCL-81 per well. Three distinct analysts carried out the execution of experimental series, and the GMT of each condition must be within a variation limit of log5 ≤ 2.00 when compared to the GMT of the reference condition (adsorption time of 60 min and cell monolayer preparation 24 h (1 day) before experiment with 200,000 Vero CCL-81 cells per well).

2.5. Statistical Analysis

Neutralizing antibodies against SARS-CoV-2 were compared between groups by pairwise Mann–Whitney tests. Statistical analyses were performed using Prism version 5.01 (GraphPad Software, San Diego, CA, USA), Software R, version 4.2.0 (2022-04-22 ucrt), packages ROCR, mfx, and caret. The ROC (Receiver Operating Characteristic) curve was constructed using the ®2023 MedCalc Software Ltd., Oostende, Belgium.

3. Results

3.1. Optimization of the Neutralization Test

Different analytical conditions were assessed to define a PRNT-SARS-CoV-2 that provides better visualization of the lysis plaque phenotype, i.e., that presents quality in the plaque definition linked to viral dilution and cell monolayers and avoids plaques overlapping. Table 2 compares three different features of the PRNT assay that were assessed throughout development and standardization with the reference-standard PRNT [50]: (1) cell concentration, (2) virus dilution, and (3) CMC overlayer concentration.
The cell concentration of 200,000 cells/well showed the best confluence in a cell monolayer of 24-well plates, resulting in regular-sized and individualized plaques with good resolution, as shown in the virus control (VC) (Figure 2A) and in the positive and negative controls (PC and NC, respectively) (Figure 2B). It is worth mentioning that there were no plaques in the cell control (CC), a prerequisite for test acceptance (Figure 2A). Approximately 60 PFU of SARS-CoV-2 virus work stock yields individualized plaques in a suitable number for manual counting (Figure 2A). The 1.5% CMC concentration produced easily countable well-defined plaques distributed evenly throughout the wells with bounded edges (Figure 2).

3.2. SARS-CoV-2 Neutralizing Antibody Screening Using PRNT

In general, 872 samples were analyzed by PRNT50 to measure NAbs specific to SARS-CoV-2. Of these 872, 73.62% (642) of the tested samples showed seropositivity to SARS-CoV-2, while 26.38% (230/872) had negative results (Figure 3). The average positive result was 3.26 log5. The minimum and maximum titers obtained were, respectively, 1.43 and 6.43 log5, indicating that our PRNT assay can cover a wide range of neutralizing antibody titers.

3.3. Establishing a Serological Panel

Following the optimization of the PRNT assay parameters, a panel of 40 serum samples was prepared and categorized into four groups according to their titers obtained by PRNT50 assay: negative (titer 1.64 log5), low (titers between 1.64 and 2.86 log5), medium (titers between 2.87 and 3.86 log5), and high (titers > 3.86 log5) (Figure 4a). This panel was subjected to three assays, with the difference between them being less than 2 log5. Figure 4b illustrates the correlation between two assay repetitions, which was 0.9826 (p < 0.0001). These panel samples will be used for further test validation and PRNT assay follow-up.

3.4. ROC Curve

A Receiver Operating Characteristic (ROC) curve (Figure 5a) was calculated by using 46 negative samples and 378 positive samples (Figure 5b). The ROC curve revealed the best PRNT-SARS-CoV-2 cut-off point of 1.64 log5, which provided the best combination of sensitivity and specificity, improving the test’s ability to differentiate positive and negative samples. The PRNT-SARS-CoV-2 assay showed a sensitivity of 97.28% and a rough specificity of 98.92%, with the area under the ROC curve (AUC) value of 0.999 (p < 0.0001), as shown in Figure 5a.

3.5. PRNT50 for SARS-CoV-2 Validation

3.5.1. Selectivity

The PRNT-SARS-CoV-2 was selective, generating acceptable results in all COVID-19-positive samples spiked with positive sera for DENV, ZIKV, YFV, and MV (Table 3). The absolute difference between the SARS-CoV-2-positive control sample (not spiked) and spiked samples with DENV, ZIKV, YFV, and MV (0.15; 0.18; 0.09; and 0.02, respectively) was less than two-fold (0.43), as described in Table 3.

3.5.2. Accuracy

The absolute differences between the log5 obtained and log5 predicted values for the homogenous sample positive for SARS-CoV-2 NAbs were 0.37, 0.19, 0.19, and 0.15 for 1:2, 1:4, 1:16, and 1:32 dilutions, respectively (Table 4 and Figure 6a). Therefore, all the observed results were within the acceptance criteria (<log53 = 0.68) for the accuracy parameter.
Additionally, as shown in Figure 6a,b, the maximum absolute difference between the obtained and expected results was less than 0.43 (log52), indicating that the difference was not larger than twice the mean difference.

3.5.3. Precision

The PRNT-SARS-CoV-2 demonstrated a remarkable intraassay precision (repeatability) for samples with low, medium, and high NAb titers (mean Δ (log5) in Table 5). One hundred percent of the results are within three-fold (0.68 log5) of the global average results, which is acceptable according to the acceptance criteria of a four-fold maximum variation (0.86 log5) (Table 5).
Regarding interassay precision (reproducibility), one hundred percent of the results are within two-fold (0.43 log5) of the global average results, which is acceptable according to the acceptance criteria of a four-fold maximum variation (0.86 log5) (global Δ (log5) in Table 5).

3.5.4. Robustness

The PRNT50 assay’s robustness was analyzed, and no significative differences were observed for all the conditions (results falling within the acceptance criteria—log53 = 0.68) except for the positive control, which was tested in cell monolayers prepared the day before (N-1) and two days before (N-2) the experiment with 90 min of adsorption (Δ = 1.20 and 0.73, respectively), as shown in Table 6. Furthermore, Table 6 shows that under other conditions, the maximum absolute difference presented was less than 0.43 log5, indicating that it was within two-fold of the GMT difference (Table 6). These findings highlights that PRNT-SARS-CoV-2 supports variations in the analytical parameters, including the cell monolayer preparation (N-1 or N-2) and the adsorption time (30 and 60 min).

3.6. PRNT-SARS-CoV-2 as a Tool to Evaluate Immune Response in Individuals Vaccinated against COVID-19, with or without Natural SARS-CoV-2 Infection

As expected, PRNT did not detect NAbs in any of the 40 serum samples taken from healthy individuals in 2014, prior to the COVID-19 pandemic (cut-off point < 14 in reciprocal dilution or 1.64 log5), as shown in Figure 7. We analyzed 160 samples collected before the beginning of COVID-19 vaccination and 85 samples that were obtained 30 days after vaccination (DAV) with two doses of the ChAdOx1-S/nCoV-19 (AZD1222; AstraZeneca) vaccine. The mean before vaccination was 2.07, and after vaccination with two doses, it was 3.48 log5 (Figure 7).

3.7. PRNT-SARS-CoV-2 Used to Compare the Levels of NAbs Produced Specific to the Wuhan Strain and the Omicron Variant in Volunteers with COVID-19 Disease

Sera from volunteers hospitalized with COVID-19 were measured regarding the NAb titers specific to the original strain (Wuhan) and the Omicron variant using a validated PRNT50. Additionally, samples were further stratified by disease severity and gender—female or male—for these analyses (Figure 8). NAb titers for all groups tested with the Wuhan strain were higher than those observed in the Omicron variant (** p < 0.01). Moreover, NAb levels were remarkably higher in severe cases than in mild cases. The difference observed between genders was not statistically significant, except for the group of mild cases (female versus male) assessed for NAbs to the Omicron variant (* p < 0.05) (Figure 8).

4. Discussion

The classic PRNT is considered the gold-standard methodology and is widely used to evaluate the immunogenicity of viral infections (i.e., yellow fever, dengue, zika, lassa, mumps, poliovirus, herpes simplex virus type-2, Japanese encephalitis, measles, and SARS-CoV-2) [18,19,20,21,22,23,24,51,52,53,54,55,56], besides its use in vaccine development [57,58]. Since the COVID-19 pandemic began, more than one-hundred assays to detect SARS-CoV-2-specific NAbs have been developed, with different levels of accuracy [59]. In our study, PRNT-SARS-CoV-2 was optimized using 24-well plates (four samples/plate), Vero cell monolayers, and positive and negative sera for COVID-19 disease for 3 days of incubation, and the assay throughput achieved 56 sera. The optimized PRNT was validated with the fitness-for-purpose concept [36], according to international and national regulations, as well as technical requirements [37,38,39,40,41,42,43]. Our optimized assay was able to cover a wide range of NAb titers in reciprocal of the dilution from 10 (1.43 log5) to 31,250 (6.43 log5), which permitted the construction of a serological panel (negative, low, medium, and high levels of NAbs to SARS-CoV-2), which is essential to run the PRNT validation. This range proved to be wider than the ranges previously demonstrated, with titers from 1:10 to 1:600 [60], 1:20 to 1:1280 [50], 1:10 to 1:4000 [61], and 1:20 to 1:10,000 [62]. Moreover, this panel demonstrated an excellent repeatability of our PRNT-SARS-CoV-2, showing a reliable test with a positive correlation (r = 0.9826, p < 0.0001) between NAb titers in two independent assays. It is worth mentioning the critical role of a suitable cut-off point for an effective test. The COVID-19 seropositivity was determined by ROC curve analysis, which has been widely used [63,64] to demonstrate the capacity to differentiate positive and negative results. Our results established a cut-off point of 1:14 (1.63 log5) with 97.28% and 98.92% sensitivity and specificity for the assay, respectively, compared to a previous study with a limited sample size, which had 82.1% sensitivity for PRNT50 [50]. This adjusted cut-off point increases the overall confidence in the results, providing a rough specificity to discriminate seronegative samples and yielding a more accurate assessment of past SARS-CoV-2 exposure. The AUC, an index of the discriminating ability of an assay [65], varies between 0.5 and 1.0 (uninformative and perfect assay, respectively), and the value obtained in our ROC curve was 0.999, which almost reaches the desirable maximum value for a test.
During the validation analyses, PRNT-SARS-CoV-2 showed selectivity since there were no observed serum interferences (maximum demonstrated difference of 0.21 with acceptance criteria up to 0.86) when COVID-19 positive samples were spiked with positive sera for other viruses, such as yellow fever virus, dengue virus, zika virus, and measles virus. This indicates that there was no interference or cross-reaction between the optimized and verified PRNT-SARS-CoV-2 and the aforementioned viruses. Regarding the accuracy, the measurements carried out on diluted analytes demonstrated to be linearly dependent on the reference measurements (obtained/expected values: 3.37/3.74 for 1:2; 3.12/3.31 for 1:4; 2.26/2.45 for 1:16; and 2.17/2.02 for 1:32). In the present study, PRNT demonstrated excellent intraassay precision of 100% since the values were well within the acceptance criteria of below four-fold (0.86 log5) of the mean and, even better, below the limit of two-fold the global average for the interassay precision, similarly to the previous results [50]. For robustness, the positive control results are compliant for the 30 and 60 min adsorption times for both conditions of cell monolayer preparation (N-1 and N-2). The validation study successfully reached satisfactory criteria in all the analyzed parameters, providing a validated assay to quantify neutralizing antibodies in the population.
When the samples taken before the pandemic were analyzed, no cross-reactions were detected, indicating that it is a trustable assay for measuring NAbs specific to SARS-CoV-2. Previous studies have also demonstrated [66] higher titers of neutralizing antibodies in serum samples of individuals with severe manifestations of COVID-19 than in those with mild disease. This relationship may be due to the delayed viral clearance in individuals with critical symptoms, although it does not necessarily correlate with a favorable outcome. Despite previous findings, no relationship between gender and disease severity was observed [67].
In contrast to other variants, Omicron is associated with a higher reinfection rate [68]. This could be explained by the significant decline in the ability of NAbs to neutralize Omicron. As we found in this study, regardless of gender or disease severity, the levels of NAbs to the Wuhan strain are significantly higher than those to the Omicron variant, as described by Khoury and colleagues [30]. Recent studies also reported that SARS-CoV-2 variants can significantly reduce the neutralizing activity of the NAbs directed to the RBD (receptor-binding domain) located in the S protein [69,70]. It is worthwhile to highlight that although the PRNT assay may suggest a lower level of neutralizing antibodies to newly emerging variations, this does not necessarily translate into an increased risk of severe clinical illness as the immune response is broader than only that of the neutralizing antibodies.
The PRNT validation and the establishment of a serum panel for assay monitoring must be successfully finished for a PRNT assay to produce accurate and trustworthy data, which is essential in epidemiology and immunization investigations, as well as to ensure test repeatability over time.
The lack of a cohort of individuals tracked from the moment of collection before immunization through the second or third dose of vaccine to follow the unique humoral response and the limited number of participants with severe symptoms are the study’s limitations. Another point that could have been included was at least one more respiratory virus tested in specificity assays.
In conclusion, the present study provides significant improvements in the PRNT assay for NAb detection through the optimal seeded cell concentration, viral input, and overlayer concentration, providing a reliable serological test for evaluating NAbs. Furthermore, our work determined the cut-off point to be just above (>14) instead of using the first dilution to determine the negative samples, increasing the reliability of the PRNT assay, according to the ROC curve. The validated PRNT assay detected different levels of neutralizing antibodies, which is relevant for identifying a range of neutralizing antibody titers from low to medium and high levels. In most individuals, SARS-CoV-2 infection or vaccination elicits robust neutralizing antibody titers, and in vaccinated individuals, NAb levels increased more than three-fold. As SARS-CoV-2 infection remains a worldwide problem, together with mass vaccination of the population, it is important to have a gold partner test, as described here (PRNT-SARS-CoV-2), to measure the specific Nab titer, used as a reference to other methods of antibody quantification to help fill some gaps related to epidemiological perspectives of surveillance policies. Therefore, the quantity of specific and functional humoral immune responses to SARS-CoV-2 is crucial to make decisions in public policies, ensuring the monitoring and orientation of the duration of the vaccine response and evaluating future vaccine candidates for COVID-19 control.

Author Contributions

Conceptualization, I.S.H., S.M.B.d.L., I.P.d.S.J., W.D.S., A.d.S.A. and O.A.M.-F.; methodology, I.S.H., A.d.S.A., B.d.M.D., B.P.S., L.d.C.M., A.F.d.S., C.B.D. and S.A.d.S.; statistical analysis, T.G.P. and I.S.H.; validation, I.S.H., V.d.O.S.F., A.d.S.A.; B.d.M.D., C.B.D., L.d.C.M. and A.F.d.S.; formal analysis, I.S.H., I.P.d.S.J., S.M.B.d.L., W.D.S., A.d.S.A. and O.A.M.-F.; investigation, I.S.H., S.M.B.d.L., I.P.d.S.J., W.D.S., A.d.S.A., N.d.S.A., T.A., A.P.D.A.B., M.d.P.G.G., J.G.M., V.V., J.P., A.C.C.-A., V.P.-M., A.T.-C. and O.A.M.-F.; resources, S.M.B.d.L., I.P.d.S.J., W.D.S., A.d.S.A., N.d.S.A., O.A.M.-F., J.G.M. and V.V.; writing—original draft preparation, I.S.H., S.M.B.d.L., I.P.d.S.J. and A.d.S.A.; writing—review and editing, I.S.H., I.P.d.S.J., S.M.B.d.L., A.d.S.A., O.A.M.-F. and A.C.C.-A.; project administration, S.M.B.d.L., W.D.S. and A.d.S.A.; funding acquisition, S.M.B.d.L., W.D.S., A.d.S.A., V.V., O.A.M.-F., A.T.-C., A.P.D.A.B. and J.G.M. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the Bio-Manguinhos/Oswaldo Cruz Foundation.

Institutional Review Board Statement

The study was conducted in accordance with the Declaration of Helsinki and approved by Ethics Review Committee for Research of the Oswaldo Cruz Foundation under numbers 34728920.4.0000.5262, 30846920.7.0000.0008, and 15120613.4.0000.5262.

Informed Consent Statement

All participants gave the authorization to use their samples, and the methods applied were carried out respecting the applicable regulations and guidelines.

Data Availability Statement

The data analyzed during the current study are available from the corresponding author on reasonable request.

Acknowledgments

We would like to acknowledge the collaboration of the Multi-user Research Facility of Biosafety Platform BSL3-HPP, Oswaldo Cruz Institute.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Wang, Z.; Yang, B.; Li, Q.; Wen, L.; Zhang, R. Clinical Features of 69 Cases with Coronavirus Disease 2019 in Wuhan, China. Clin. Infect. Dis. 2020, 71, 769–777. [Google Scholar] [CrossRef]
  2. Zhou, F.; Yu, T.; Du, R.; Fan, G.; Liu, Y.; Liu, Z.; Xiang, J.; Wang, Y.; Song, B.; Gu, X.; et al. Clinical Course and Risk Factors for Mortality of Adult Inpatients with COVID-19 in Wuhan, China: A Retrospective Cohort Study. Lancet 2020, 395, 1054–1062. [Google Scholar] [CrossRef]
  3. Wu, Z.; McGoogan, J.M. Characteristics of and Important Lessons from the Coronavirus Disease 2019 (COVID-19) Outbreak in China: Summary of a Report of 72,314 Cases from the Chinese Center for Disease Control and Prevention. JAMA 2020, 323, 1239–1242. [Google Scholar] [CrossRef] [PubMed]
  4. Letko, M.; Marzi, A.; Munster, V. Functional Assessment of Cell Entry and Receptor Usage for SARS-CoV-2 and Other Lineage B Betacoronaviruses. Nat. Microbiol. 2020, 5, 562–569. [Google Scholar] [CrossRef] [PubMed]
  5. Harvey, W.T.; Carabelli, A.M.; Jackson, B.; Gupta, R.K.; Thomson, E.C.; Harrison, E.M.; Ludden, C.; Reeve, R.; Rambaut, A.; COVID-19 Genomics UK (COG-UK) Consortium; et al. SARS-CoV-2 Variants, Spike Mutations and Immune Escape. Nat. Rev. Microbiol. 2021, 19, 409–424. [Google Scholar] [CrossRef]
  6. Shen, X.; Tang, H.; Pajon, R.; Smith, G.; Glenn, G.M.; Shi, W.; Korber, B.; Montefiori, D.C. Neutralization of SARS-CoV-2 Variants B.1.429 and B.1.351. N. Engl. J. Med. 2021, 384, 2352–2354. [Google Scholar] [CrossRef] [PubMed]
  7. Fiolet, T.; Kherabi, Y.; MacDonald, C.-J.; Ghosn, J.; Peiffer-Smadja, N. Comparing COVID-19 Vaccines for Their Characteristics, Efficacy and Effectiveness against SARS-CoV-2 and Variants of Concern: A Narrative Review. Clin. Microbiol. Infect. 2022, 28, 202–221. [Google Scholar] [CrossRef]
  8. Carvalho, T.; Krammer, F.; Iwasaki, A. The First 12 Months of COVID-19: A Timeline of Immunological Insights. Nat. Rev. Immunol. 2021, 21, 245–256. [Google Scholar] [CrossRef]
  9. Qi, H.; Liu, B.; Wang, X.; Zhang, L. The Humoral Response and Antibodies against SARS-CoV-2 Infection. Nat. Immunol. 2022, 23, 1008–1020. [Google Scholar] [CrossRef]
  10. Suthar, M.S.; Zimmerman, M.; Kauffman, R.; Mantus, G.; Linderman, S.; Vanderheiden, A.; Nyhoff, L.; Davis, C.; Adekunle, S.; Affer, M.; et al. Rapid Generation of Neutralizing Antibody Responses in COVID-19 Patients. medRxiv 2020, 9, 1–15. [Google Scholar] [CrossRef]
  11. Jackson, L.A.; Anderson, E.J.; Rouphael, N.G.; Roberts, P.C.; Makhene, M.; Coler, R.N.; McCullough, M.P.; Chappell, J.D.; Denison, M.R.; Stevens, L.J.; et al. An mRNA Vaccine against SARS-CoV-2-Preliminary Report. N. Engl. J. Med. 2020, 383, 1920–1931. [Google Scholar] [CrossRef] [PubMed]
  12. Wajnberg, A.; Amanat, F.; Firpo, A.; Altman, D.R.; Bailey, M.J.; Mansour, M.; McMahon, M.; Meade, P.; Mendu, D.R.; Muellers, K.; et al. Robust Neutralizing Antibodies to SARS-CoV-2 Infection Persist for Months. Science 2020, 370, 1227–1230. [Google Scholar] [CrossRef]
  13. Dan, J.M.; Mateus, J.; Kato, Y.; Hastie, K.M.; Yu, E.D.; Faliti, C.E.; Grifoni, A.; Ramirez, S.I.; Haupt, S.; Frazier, A.; et al. Immunological Memory to SARS-CoV-2 Assessed for up to 8 Months after Infection. Science 2021, 371, eabf4063. [Google Scholar] [CrossRef]
  14. Seow, J.; Graham, C.; Merrick, B.; Acors, S.; Pickering, S.; Steel, K.J.A.; Hemmings, O.; O’Byrne, A.; Kouphou, N.; Galao, R.P.; et al. Longitudinal Observation and Decline of Neutralizing Antibody Responses in the Three Months following SARS-CoV-2 Infection in Humans. Nat. Microbiol. 2020, 5, 1598–1607. [Google Scholar] [CrossRef] [PubMed]
  15. Wang, K.; Long, Q.-X.; Deng, H.-J.; Hu, J.; Gao, Q.-Z.; Zhang, G.-J.; He, C.-L.; Huang, L.-Y.; Hu, J.-L.; Chen, J.; et al. Longitudinal Dynamics of the Neutralizing Antibody Response to Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) Infection. Clin. Infect. Dis. 2021, 73, e531–e539. [Google Scholar] [CrossRef]
  16. Dulbecco, R. Production of Plaques in Monolayer Tissue Cultures by Single Particles of an Animal Virus. Proc. Natl. Acad. Sci. USA 1952, 38, 747–752. [Google Scholar] [CrossRef] [PubMed]
  17. Dulbecco, R.; Vogt, M. Plaque Formation and Isolation of Pure Lines with Poliomyelitis Viruses. J. Exp. Med. 1954, 99, 167–182. [Google Scholar] [CrossRef]
  18. Russell, P.K.; Nisalak, A.; Sukhavachana, P.; Vivona, S. A Plaque Reduction Test for Dengue Virus Neutralizing Antibodies. J. Immunol. 1967, 99, 285–290. [Google Scholar] [CrossRef]
  19. Ennis, F.A. Immunity to Mumps in an Institutional Epidemic. Correlation of Insusceptibility to Mumps with Serum Plaque Neutralizing and Hemagglutination-Inhibiting Antibodies. J. Infect. Dis. 1969, 119, 654–657. [Google Scholar] [CrossRef]
  20. Porterfield, J.S. A Plaque Technique for the Titration of Yellow Fever Virus and Antisera. Trans. R. Soc. Trop. Med. Hyg. 1959, 53, 458–466. [Google Scholar] [CrossRef]
  21. Cohen, B.J.; Audet, S.; Andrews, N.; Beeler, J.; WHO working group on measles plaque reduction neutralization test. Plaque Reduction Neutralization Test for Measles Antibodies: Description of a Standardised Laboratory Method for Use in Immunogenicity Studies of Aerosol Vaccination. Vaccine 2007, 26, 59–66. [Google Scholar] [CrossRef]
  22. Weldon, W.C.; Oberste, M.S.; Pallansch, M.A. Standardized Methods for Detection of Poliovirus Antibodies. Methods Mol. Biol. 2016, 1387, 145–176. [Google Scholar] [CrossRef]
  23. Tomori, O.; Johnson, K.M.; Kiley, M.P.; Elliott, L.H. Standardization of a Plaque Assay for Lassa Virus. J. Med. Virol. 1987, 22, 77–89. [Google Scholar] [CrossRef] [PubMed]
  24. Okuno, Y.; Igarashi, A.; Fukai, K. Neutralization Tests for Dengue and Japanese Encephalitis Viruses by the Focus Reduction Method Using Peroxidase-Anti-Peroxidase Staining. Biken J. 1978, 21, 137–147. [Google Scholar] [PubMed]
  25. Albrecht, P.; Herrmann, K.; Burns, G.R. Role of Virus Strain in Conventional and Enhanced Measles Plaque Neutralization Test. J. Virol. Methods 1981, 3, 251–260. [Google Scholar] [CrossRef]
  26. Jacofsky, D.; Jacofsky, E.M.; Jacofsky, M. Understanding Antibody Testing for COVID-19. J. Arthroplast. 2020, 35, S74–S81. [Google Scholar] [CrossRef] [PubMed]
  27. Mauldin, J.; Carbone, K.; Hsu, H.; Yolken, R.; Rubin, S. Mumps Virus-Specific Antibody Titers from Pre-Vaccine Era Sera: Comparison of the Plaque Reduction Neutralization Assay and Enzyme Immunoassays. J. Clin. Microbiol. 2005, 43, 4847–4851. [Google Scholar] [CrossRef] [PubMed]
  28. Castro Dopico, X.; Ols, S.; Loré, K.; Karlsson Hedestam, G.B. Immunity to SARS-CoV-2 Induced by Infection or Vaccination. J. Intern. Med. 2022, 291, 32–50. [Google Scholar] [CrossRef]
  29. Machado, B.A.S.; Hodel, K.V.S.; Fonseca, L.M.D.S.; Pires, V.C.; Mascarenhas, L.A.B.; da Silva Andrade, L.P.C.; Moret, M.A.; Badaró, R. The Importance of Vaccination in the Context of the COVID-19 Pandemic: A Brief Update Regarding the Use of Vaccines. Vaccines 2022, 10, 591. [Google Scholar] [CrossRef]
  30. Khoury, D.S.; Cromer, D.; Reynaldi, A.; Schlub, T.E.; Wheatley, A.K.; Juno, J.A.; Subbarao, K.; Kent, S.J.; Triccas, J.A.; Davenport, M.P. Neutralizing Antibody Levels Are Highly Predictive of Immune Protection from Symptomatic SARS-CoV-2 Infection. Nat. Med. 2021, 27, 1205–1211. [Google Scholar] [CrossRef]
  31. Khoury, D.S.; Wheatley, A.K.; Ramuta, M.D.; Reynaldi, A.; Cromer, D.; Subbarao, K.; O’Connor, D.H.; Kent, S.J.; Davenport, M.P. Measuring Immunity to SARS-CoV-2 Infection: Comparing Assays and Animal Models. Nat. Rev. Immunol. 2020, 20, 727–738. [Google Scholar] [CrossRef] [PubMed]
  32. Lee, C.Y.-P.; Lin, R.T.P.; Renia, L.; Ng, L.F.P. Serological Approaches for COVID-19: Epidemiologic Perspective on Surveillance and Control. Front. Immunol. 2020, 11, 879. [Google Scholar] [CrossRef]
  33. Azamor, T.; Horbach, I.S.; Brito, E.; Cunha, D.; Melgaço, J.G.; Silva, A.M.V.D.; Tubarão, L.N.; Azevedo, A.S.; Santos, R.T.; Alves, N.D.S.; et al. Protective Immunity of COVID-19 Vaccination with ChAdOx1 nCoV-19 Following Previous SARS-CoV-2 Infection: A Humoral and Cellular Investigation. Viruses 2022, 14, 1916. [Google Scholar] [CrossRef] [PubMed]
  34. Gomes, M.P.B.; Linhares, J.H.R.; Dos Santos, T.P.; Pereira, R.C.; Santos, R.T.; da Silva, S.A.; Souza, M.C.O.; da Silva, J.F.A.; Trindade, G.F.; Gomes, V.S.; et al. Inactivated and Immunogenic SARS-CoV-2 for Safe Use in Immunoassays and as an Immunization Control for Non-Clinical Trials. Viruses 2023, 15, 1486. [Google Scholar] [CrossRef] [PubMed]
  35. CDC. Interim Laboratory Biosafety Guidelines for Handling and Processing Specimens Associated with Coronavirus Disease 2019 (COVID-19). 2021. Available online: https://www.cdc.gov/coronavirus/2019-nCoV/lab/lab-biosafety-guidelines.html (accessed on 1 December 2023).
  36. Magnusson, B.; Örnemark, U. (Eds.) Eurachem Guide: The Fitness for Purpose of Analytical Methods—A Laboratory Guide to Method Validation and Related Topics; A Laboratory Guide to Method Validation and Related Topics; LGC: London, UK, 2014; ISBN 978-91-87461-59-0. Available online: http://www.eurachem.org (accessed on 9 December 2023).
  37. WHO. WHO TRS 1019 Good Manufacturing Pratices Annex 3: Validation and Qualification. In WHO Expert Committee on Specifications for Pharmaceutical Preparations-Fifty-Third Report; World Health Organization: Geneva, Switzerland, 2019; Available online: https://www.who.int/publications/m/item/trs1019-annex3 (accessed on 9 December 2023).
  38. Who Guidelines on Validation—Appendix 4-Analytical Method Validation. Working Document QAS/16.671. 2016. Available online: https://dcvmn.org/wp-content/uploads/2017/01/who_guideline_validation_analyticalmethodvalidationqas16-671.pdf (accessed on 9 December 2023).
  39. European Medicines Agency. CPMP ICH Topic Q 2 (R1) Validation of Analytical Procedures: Text and Methodology. In Note For Guidance on Validation of Analytical Procedures: Text and Methodology (Cpmp/Ich/381/95); European Medicines Agency: Amsterdam, The Netherlands, 1995; Available online: https://www.ema.europa.eu/en/documents/scientific-guideline/ich-guideline-q2r1-validation-analytical-procedures-text-and-methodology-step-5-first-version_en.pdf (accessed on 9 December 2023).
  40. Bioanalytical Method Validation Guidance for Industry. 2018. Available online: https://www.fda.gov/regulatory-information/search-fda-guidance-documents/bioanalytical-method-validation-guidance-industry (accessed on 5 December 2023).
  41. USP 1237 VIROLOGY TEST METHODS. 2020. Available online: https://doi.usp.org/USPNF/USPNF_M915_01_01.html (accessed on 5 December 2023).
  42. USP 1225 Validation of Compendial Procedures. 2020. Available online: http://ftp.uspbpep.com/v29240/usp29nf24s0_c1225.html (accessed on 5 December 2023).
  43. Pereira, P.R.T. ANVISA RDCo27/2012. Available online: https://bvsms.saude.gov.br/bvs/saudelegis/anvisa/2012/rdc0027_17_05_2012.html (accessed on 5 December 2023).
  44. Rosner, B. Fundamentals of Biostatistical; Cengage Learning: Carolyn Crockets, MA, USA, 2000. [Google Scholar]
  45. Armitage, P.; Berry, G.; Matthews, J.N.S. Statistical Methods in Medical Research; John Wiley & Sons: Hoboken, NJ, USA, 2002; ISBN 978-0-470-77366-6. [Google Scholar]
  46. Committee for Medicinal Products for Human Use (CHMP). COMIRNATY; European Medicines Agencyl: Amsterdam, The Netherlands, 2021. [Google Scholar]
  47. Thuluva, S.; Paradkar, V.; Gunneri, S.R.; Yerroju, V.; Mogulla, R.; Turaga, K.; Kyasani, M.; Manoharan, S.K.; Medigeshi, G.; Singh, J.; et al. Evaluation of Safety and Immunogenicity of Receptor-Binding Domain-Based COVID-19 Vaccine (Corbevax) to Select the Optimum Formulation in Open-Label, Multicentre, and Randomised Phase-1/2 and Phase-2 Clinical Trials. eBioMedicine 2022, 83, 104217. [Google Scholar] [CrossRef]
  48. Song, J.Y.; Choi, W.S.; Heo, J.Y.; Kim, E.J.; Lee, J.S.; Jung, D.S.; Kim, S.-W.; Park, K.-H.; Eom, J.S.; Jeong, S.J.; et al. Immunogenicity and Safety of SARS-CoV-2 Recombinant Protein Nanoparticle Vaccine GBP510 Adjuvanted with AS03: Interim Results of a Randomised, Active-Controlled, Observer-Blinded, Phase 3 Trial. eClinicalMedicine 2023, 64, 102140. [Google Scholar] [CrossRef]
  49. Moo-Young, M. Comprehensive Biotechnology; Elsevier: Amsterdam, The Netherlands, 2011; ISBN 978-0-08-088504-9. [Google Scholar]
  50. Valcourt, E.J.; Manguiat, K.; Robinson, A.; Lin, Y.-C.; Abe, K.T.; Mubareka, S.; Shigayeva, A.; Zhong, Z.; Girardin, R.C.; DuPuis, A.; et al. Evaluating Humoral Immunity against SARS-CoV-2: Validation of a Plaque-Reduction Neutralization Test and a Multilaboratory Comparison of Conventional and Surrogate Neutralization Assays. Microbiol. Spectr. 2021, 9, e0088621. [Google Scholar] [CrossRef]
  51. Espejo, A.P.; Akgun, Y.; Al Mana, A.F.; Tjendra, Y.; Millan, N.C.; Gomez-Fernandez, C.; Cray, C. Review of Current Advances in Serologic Testing for COVID-19. Am. J. Clin. Pathol. 2020, 154, 293–304. [Google Scholar] [CrossRef]
  52. Habli, Z.; Saleh, S.; Zaraket, H.; Khraiche, M.L. COVID-19 in-Vitro Diagnostics: State-of-the-Art and Challenges for Rapid, Scalable, and High-Accuracy Screening. Front. Bioeng. Biotechnol. 2020, 8, 605702. [Google Scholar] [CrossRef]
  53. Montesinos, I.; Dahma, H.; Wolff, F.; Dauby, N.; Delaunoy, S.; Wuyts, M.; Detemmerman, C.; Duterme, C.; Vandenberg, O.; Martin, C.; et al. Neutralizing Antibody Responses following Natural SARS-CoV-2 Infection: Dynamics and Correlation with Commercial Serologic Tests. J. Clin. Virol. 2021, 144, 104988. [Google Scholar] [CrossRef]
  54. Cunha, L.E.R.; Stolet, A.A.; Strauch, M.A.; Pereira, V.A.R.; Dumard, C.H.; Gomes, A.M.O.; Monteiro, F.L.; Higa, L.M.; Souza, P.N.C.; Fonseca, J.G.; et al. Polyclonal F(Ab’)2 Fragments of Equine Antibodies Raised against the Spike Protein Neutralize SARS-CoV-2 Variants with High Potency. iScience 2021, 24, 103315. [Google Scholar] [CrossRef] [PubMed]
  55. Chu, J.J.H.; Ahidjo, B.A.; Mok, C.K. SARS-CoV-2 Methods and Protocols; Methods in Molecular Biology; Humana Press Inc.: Totowa, NJ, USA, 2022; p. 2452. ISBN 978-1-07-162111-0. [Google Scholar]
  56. Cyster, J.G.; Allen, C.D.C. B Cell Responses–Cell Interaction Dynamics and Decisions. Cell 2019, 177, 524–540. [Google Scholar] [CrossRef]
  57. Goldblatt, D.; Alter, G.; Crotty, S.; Plotkin, S.A. Correlates of Protection against SARS-CoV-2 Infection and COVID-19 Disease. Immunol. Rev. 2022, 310, 6–26. [Google Scholar] [CrossRef]
  58. Plotkin, S.A. Correlates of Protection Induced by Vaccination. Clin. Vaccine Immunol. 2010, 17, 1055–1065. [Google Scholar] [CrossRef]
  59. Zheng, X.; Duan, R.; Gong, F.; Wei, X.; Dong, Y.; Chen, R.; Liang, M.; Tang, C.; Lu, L. Accuracy of Serological Tests for COVID-19: A Systematic Review and Meta-Analysis. Front. Public. Health 2022, 10, 923525. [Google Scholar] [CrossRef] [PubMed]
  60. Cotugno, N.; Ruggiero, A.; Bonfante, F.; Petrara, M.R.; Zicari, S.; Pascucci, G.R.; Zangari, P.; De Ioris, M.A.; Santilli, V.; Manno, E.C.; et al. Virological and Immunological Features of SARS-CoV-2-Infected Children Who Develop Neutralizing Antibodies. Cell Rep. 2021, 34, 108852. [Google Scholar] [CrossRef] [PubMed]
  61. Deshpande, G.R.; Sapkal, G.N.; Tilekar, B.N.; Yadav, P.D.; Gurav, Y.; Gaikwad, S.; Kaushal, H.; Deshpande, K.S.; Kaduskar, O.; Sarkale, P.; et al. Neutralizing Antibody Responses to SARS-CoV-2 in COVID-19 Patients. Indian. J. Med. Res. 2020, 152, 82–87. [Google Scholar] [CrossRef]
  62. Merluza, J.; Ung, J.; Makowski, K.; Robinson, A.; Manguiat, K.; Mueller, N.; Audet, J.; Chen, J.C.-Y.; Strong, J.E.; Wood, H.; et al. Validation and Establishment of the SARS-CoV-2 Lentivirus Surrogate Neutralization Assay as a Prescreening Tool for the Plaque Reduction Neutralization Test. Microbiol. Spectr. 2023, 11, e03789-22. [Google Scholar] [CrossRef]
  63. Metz, C.E. Basic Principles of ROC Analysis. Semin. Nucl. Med. 1978, 8, 283–298. [Google Scholar] [CrossRef]
  64. Suhail, A.R. Methods of Clinical Epidemiology; Springer Series on Epidemiology and Public Health; Springer: Berlin/Heidelberg, Germany, 2013; ISBN 978-3-642-37131-8. [Google Scholar]
  65. DeLong, E.R.; DeLong, D.M.; Clarke-Pearson, D.L. Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 1988, 44, 837–845. [Google Scholar] [CrossRef]
  66. Hashem, A.M.; Algaissi, A.; Almahboub, S.A.; Alfaleh, M.A.; Abujamel, T.S.; Alamri, S.S.; Alluhaybi, K.A.; Hobani, H.I.; AlHarbi, R.H.; Alsulaiman, R.M.; et al. Early Humoral Response Correlates with Disease Severity and Outcomes in COVID-19 Patients. Viruses 2020, 12, 1390. [Google Scholar] [CrossRef] [PubMed]
  67. Gao, Y.; Ding, M.; Dong, X.; Zhang, J.; Kursat Azkur, A.; Azkur, D.; Gan, H.; Sun, Y.; Fu, W.; Li, W.; et al. Risk Factors for Severe and Critically Ill COVID-19 Patients: A Review. Allergy 2021, 76, 428–455. [Google Scholar] [CrossRef] [PubMed]
  68. Pulliam, J.R.C.; van Schalkwyk, C.; Govender, N.; von Gottberg, A.; Cohen, C.; Groome, M.J.; Dushoff, J.; Mlisana, K.; Moultrie, H. Increased Risk of SARS-CoV-2 Reinfection Associated with Emergence of Omicron in South Africa. Science 2022, 376, eabn4947. [Google Scholar] [CrossRef] [PubMed]
  69. Cao, Y.; Wang, J.; Jian, F.; Xiao, T.; Song, W.; Yisimayi, A.; Huang, W.; Li, Q.; Wang, P.; An, R.; et al. Omicron Escapes the Majority of Existing SARS-CoV-2 Neutralizing Antibodies. Nature 2022, 602, 657–663. [Google Scholar] [CrossRef]
  70. Cele, S.; Jackson, L.; Khoury, D.S.; Khan, K.; Moyo-Gwete, T.; Tegally, H.; San, J.E.; Cromer, D.; Scheepers, C.; Amoako, D.G.; et al. Omicron Extensively but Incompletely Escapes Pfizer BNT162b2 Neutralization. Nature 2022, 602, 654–656. [Google Scholar] [CrossRef]
Diseases 12 00029 g001
Figure 1. Schematic workflow representation of PRNT assay optimization steps and the tested parameters (viral input, cell, and CMC concentrations). The test begins with adding samples (15 µL) and their subsequent dilution (five-fold). Then, the SARS-CoV-2 virus is added (60, 70, or 100 PFU), followed by the first one-hour incubation at 37 °C, 5% CO2. Subsequently, the supernatant is transferred to 24-well plates containing the cell monolayer previously prepared (120,000 or 200,000 cells/well), followed by the second one-hour incubation at 37 °C, 5% CO2. The supernatant is removed, and the cell monolayer is overlayed by CMC (1.5, 2.0, or 2.5%). Afterward (3 days, 37 °C, 5% CO2), the formalin fixation (1.25%) is performed, and the plates are stained with 0.04% crystal violet solution. Figure designed at canva.com.
Figure 1. Schematic workflow representation of PRNT assay optimization steps and the tested parameters (viral input, cell, and CMC concentrations). The test begins with adding samples (15 µL) and their subsequent dilution (five-fold). Then, the SARS-CoV-2 virus is added (60, 70, or 100 PFU), followed by the first one-hour incubation at 37 °C, 5% CO2. Subsequently, the supernatant is transferred to 24-well plates containing the cell monolayer previously prepared (120,000 or 200,000 cells/well), followed by the second one-hour incubation at 37 °C, 5% CO2. The supernatant is removed, and the cell monolayer is overlayed by CMC (1.5, 2.0, or 2.5%). Afterward (3 days, 37 °C, 5% CO2), the formalin fixation (1.25%) is performed, and the plates are stained with 0.04% crystal violet solution. Figure designed at canva.com.
Diseases 12 00029 g001
Diseases 12 00029 g002
Figure 2. Representative images of infected or mock Vero cells, with the PRNT-SARS-CoV-2 procedure optimized. (A) Cell monolayer with 2 × 105 Vero cell/well (CC: cell control), resulting in regular-sized plaques (VC: virus control) and a good resolution. (B) Representative image of positive (PC) and negative (NC) controls of Vero cells infected with approximately 60 PFU of SARS-CoV-2.
Figure 2. Representative images of infected or mock Vero cells, with the PRNT-SARS-CoV-2 procedure optimized. (A) Cell monolayer with 2 × 105 Vero cell/well (CC: cell control), resulting in regular-sized plaques (VC: virus control) and a good resolution. (B) Representative image of positive (PC) and negative (NC) controls of Vero cells infected with approximately 60 PFU of SARS-CoV-2.
Diseases 12 00029 g002
Diseases 12 00029 g003
Figure 3. SARS-CoV-2 neutralizing antibodies screening of 872 serum samples. The first and second columns show the samples considered negative (230) or positive (642). The dashed line indicates the first sera dilution (1:10 or 1.43 log5).
Figure 3. SARS-CoV-2 neutralizing antibodies screening of 872 serum samples. The first and second columns show the samples considered negative (230) or positive (642). The dashed line indicates the first sera dilution (1:10 or 1.43 log5).
Diseases 12 00029 g003
Diseases 12 00029 g004
Figure 4. Serological panel titers (a) SARS-CoV-2 NAb levels were obtained using a sera panel composed of 40 samples (10 samples for each: negative, low, medium, and high titers). (b) Correlation of SARS-CoV-2 NAb levels obtained from a panel of 40 samples titrated by PRNT-SARS-CoV-2 in two independent assays performed on different days. Linear regression shows a positive correlation (r = 0.9655) that is statistically significant (p < 0.0001) between assay repetitions.
Figure 4. Serological panel titers (a) SARS-CoV-2 NAb levels were obtained using a sera panel composed of 40 samples (10 samples for each: negative, low, medium, and high titers). (b) Correlation of SARS-CoV-2 NAb levels obtained from a panel of 40 samples titrated by PRNT-SARS-CoV-2 in two independent assays performed on different days. Linear regression shows a positive correlation (r = 0.9655) that is statistically significant (p < 0.0001) between assay repetitions.
Diseases 12 00029 g004
Diseases 12 00029 g005
Figure 5. (a) ROC curve was calculated using sensitivity and 100-specificity to select the optimal cut-off point (14 in reciprocal dilution or 1.64 log5) for PRNT-SARS-CoV-2. The PRNT-SARS-CoV-2 assay showed 97.28% and 98.92% sensitivity and specificity, respectively. Area under the curve (AUC = 0.999) was statistically significant (p < 0.0001). (b) Distribution of neutralizing antibody titers against SARS-CoV-2 in the 424 samples (46 negative samples, first column; 378 positive, second column). The dashed line shows the PRNT cut-off point calculated with the ROC curve (1:14 or 1.64 log5).
Figure 5. (a) ROC curve was calculated using sensitivity and 100-specificity to select the optimal cut-off point (14 in reciprocal dilution or 1.64 log5) for PRNT-SARS-CoV-2. The PRNT-SARS-CoV-2 assay showed 97.28% and 98.92% sensitivity and specificity, respectively. Area under the curve (AUC = 0.999) was statistically significant (p < 0.0001). (b) Distribution of neutralizing antibody titers against SARS-CoV-2 in the 424 samples (46 negative samples, first column; 378 positive, second column). The dashed line shows the PRNT cut-off point calculated with the ROC curve (1:14 or 1.64 log5).
Diseases 12 00029 g005
Diseases 12 00029 g006
Figure 6. Accuracy profile for the validation of PRNT-SARS-CoV-2: (a) obtained and expected values (log5) for each sera dilution. ND: not diluted. (b) Absolute differences (solid line with bullets) for all dilutions tested. All values obtained are below the acceptability limits (dashed lines) proposed, indicating a minimal variation between theoretical and experimental tests.
Figure 6. Accuracy profile for the validation of PRNT-SARS-CoV-2: (a) obtained and expected values (log5) for each sera dilution. ND: not diluted. (b) Absolute differences (solid line with bullets) for all dilutions tested. All values obtained are below the acceptability limits (dashed lines) proposed, indicating a minimal variation between theoretical and experimental tests.
Diseases 12 00029 g006
Diseases 12 00029 g007
Figure 7. PRNT50 optimized and validated as a tool to evaluate immune response in individuals vaccinated to COVID-19, with or without natural SARS-CoV-2 infection. Distribution of NAb titers to SARS-CoV-2 Wuhan strain. The “before pandemic” group shows negative results for samples collected before the COVID-19 pandemic. The second group represents status before vaccination (baseline), and the third one shows the group immunized with two doses of ChAdOx1-S/nCoV-19 (AZD1222; AstraZeneca) vaccine; sera were analyzed 30 days after last immunization. Asterisks indicate a statistically significant difference (*** p < 0.001). The PRNT titer levels were expressed in log5. The dashed line shows the PRNT cut-off (1:14 or 1.64 log5) established in the validation process.
Figure 7. PRNT50 optimized and validated as a tool to evaluate immune response in individuals vaccinated to COVID-19, with or without natural SARS-CoV-2 infection. Distribution of NAb titers to SARS-CoV-2 Wuhan strain. The “before pandemic” group shows negative results for samples collected before the COVID-19 pandemic. The second group represents status before vaccination (baseline), and the third one shows the group immunized with two doses of ChAdOx1-S/nCoV-19 (AZD1222; AstraZeneca) vaccine; sera were analyzed 30 days after last immunization. Asterisks indicate a statistically significant difference (*** p < 0.001). The PRNT titer levels were expressed in log5. The dashed line shows the PRNT cut-off (1:14 or 1.64 log5) established in the validation process.
Diseases 12 00029 g007
Diseases 12 00029 g008
Figure 8. NAb titers to the original (Wuhan) and Omicron strains of the SARS-CoV-2 virus, separated by gender and COVID-19 severity. Asterisks indicate statistically significant differences (* p < 0.05, ** p < 0.01, *** p < 0.001) as determined by paired t-test. The PRNT titer levels were expressed in log5. The dashed line shows the PRNT cut-off point (1:14 or 1.64 log5).
Figure 8. NAb titers to the original (Wuhan) and Omicron strains of the SARS-CoV-2 virus, separated by gender and COVID-19 severity. Asterisks indicate statistically significant differences (* p < 0.05, ** p < 0.01, *** p < 0.001) as determined by paired t-test. The PRNT titer levels were expressed in log5. The dashed line shows the PRNT cut-off point (1:14 or 1.64 log5).
Diseases 12 00029 g008
Table 1. Summary of the PRNT-SARS-CoV-2 assay validation steps, their respective designs, number of runs, replicates, and acceptance criteria. DENV: dengue virus, ZIKV: zika virus, YFV: yellow fever virus, and MV: measles virus.
Table 1. Summary of the PRNT-SARS-CoV-2 assay validation steps, their respective designs, number of runs, replicates, and acceptance criteria. DENV: dengue virus, ZIKV: zika virus, YFV: yellow fever virus, and MV: measles virus.
Validation StepDesignRunsReplicatesAcceptance Criteria
SelectivityDENV + SARS-CoV-2, ZIKV + SARS-CoV-2, YFV + SARS-CoV-2, and MV + SARS-CoV-213Compliant if 80% of SARS-CoV-2 positive samples, when mixed with positive samples for DENV, ZIKV, YFV, and MV, present titers below four-fold variation, compared with positive serum only for SARS-CoV-2
Precision18 positive SARS-CoV-2 sera (6 low, 6 medium, and 6 high titers)33Compliant if more than 90% of the results are within 4-fold the mean titer difference in more than 80% of the positive samples tested
Accuracystandard serum diluted or not in 1:2, 1:4, 1:16, and 1:32 in negative serum 35Compliant if the absolute difference between the expected and observed values is below log53 = 0.68 in more than 80% for each dilution
Robustnessadsorption time: 30, 60, and 90 min; monolayer preparation: N-1 and N-2 31Compliant if the average experimental value of each series is within the variation limit (log53 = 0.68) when compared to the results obtained in the reference test
Table 2. Comparison between the reference-standard PRNT [50] and the improved PRNT for SARS-CoV-2: a summary of the assessed features, their testing settings, and the chosen PRNT improvement variables. NA: not available; PFU: Plaque-Forming Units; CMC: carboxymethylcellulose; MEM: Minimum Essential Medium; FBS: fetal bovine serum.
Table 2. Comparison between the reference-standard PRNT [50] and the improved PRNT for SARS-CoV-2: a summary of the assessed features, their testing settings, and the chosen PRNT improvement variables. NA: not available; PFU: Plaque-Forming Units; CMC: carboxymethylcellulose; MEM: Minimum Essential Medium; FBS: fetal bovine serum.
Assay
Characteristic		Reference-Standard PRNTPRNT SARS-CoV-2 Tested ConditionsPRNT SARS-CoV-2
Selected Condition
Cell concentrationNA120,000 cells/well200,000 cells/well
200,000 cells/well
Virus dilution100 PFU/100 µL60 PFU/60 µL60 PFU/60 µL
70 PFU/60 µL
100 PFU/60 µL
Overlay1.5% CMC in MEM,
4% FBS, 37 °C, 72 h		1.5% CMC in E199,
5% FBS, 37 °C, 72 h		1.5% CMC in E199,
5% FBS, 37 °C, 72 h
2.0% CMC in E199,
5% FBS, 37 °C, 72 h
2.5% CMC in E199,
5% FBS, 37 °C, 72 h
Table 3. Results for selectivity of PRNT50 for SARS-CoV-2.
Table 3. Results for selectivity of PRNT50 for SARS-CoV-2.
SampleGMTGMT log5Average log5Absolute DifferenceCriteriaConclusion
SARS-CoV-24193.753.75NANANA
SARS-CoV-2 + DENV4473.793.610.150.43 (up to 2-fold)Compliant
2033.3
4013.72
SARS-CoV-2 + ZIKV3783.693.570.180.43 (up to 2-fold)Compliant
3223.59
2503.43
SARS-CoV-2 + YFV2913.523.660.090.43 (up to 2-fold)Compliant
3993.72
4083.73
SARS-CoV-2 + MV5593.933.730.020.43 (up to 2-fold)Compliant
4393.78
2753.49
Table 4. Results for PRNT50 accuracy for SARS-CoV-2.
Table 4. Results for PRNT50 accuracy for SARS-CoV-2.
Titer (log5)Obtained Values (log5)Expected
Values (log5)		Absolute Difference (log5) (%)CriteriaConclusion
Assay #1Assay #2Assay #3
Not
diluted		4.094.124.364.17----
4.094.204.26
3.934.034.42
4.134.174.49
4.054.094.16
1:23.333.223.793.373.740.37 <0.68Compliant
3.313.273.35
3.313.653.42
3.363.243.28
3.273.223.52
1:43.123.113.103.123.310.19<0.68Compliant
3.113.113.07
3.172.973.58
3.132.903.37
3.022.863.13
1:162.352.172.412.262.450.19<0.68Compliant
2.282.132.38
2.122.182.36
2.112.212.33
2.292.262.30
1:321.992.182.102.172.020.15<0.68Compliant
2.182.132.23
2.192.162.20
2.262.242.16
2.131.962.40
Table 5. Results for PRNT50 precision for SARS-CoV-2.
Table 5. Results for PRNT50 precision for SARS-CoV-2.
TiterSampleRepetitionMean Δ (log5)Global Δ (log5)CriteriaConclusion
Low#110.1020.100<0.86Compliant
20.148
30.051
#210.1450.103Compliant
20.120
30.043
#310.0700.096Compliant
20.100
30.166
#410.1170.093Compliant
20.089
30.074
#510.2790.279Compliant
20.279
3NA
#610.3460.286Compliant
20.320
30.191
Medium#110.3690.372<0.86Compliant
20.397
30.350
#210.3500.288Compliant
20.363
30.150
#310.3580.269Compliant
20.378
30.071
#410.5180.369Compliant
20.371
30.217
#510.1230.191Compliant
20.286
30.163
#610.4960.424Compliant
20.421
30.354
High#110.3190.234<0.86Compliant
20.270
30.113
#210.2980.316Compliant
20.508
30.143
#310.0810.089Compliant
20.065
30.122
#410.2200.146Compliant
20.098
30.122
#510.1940.234Compliant
20.300
30.209
#610.5110.341Compliant
20.256
30.255
Table 6. Results of positive and negative controls for robustness of PRNT50 for SARS-CoV-2.
Table 6. Results of positive and negative controls for robustness of PRNT50 for SARS-CoV-2.
Positive ControlConditionsAssay #1Assay #2Assay #3Conclusion
Positive Control Mean (log5)ΔPositive Control Mean (log5)ΔPositive Control Mean (log5)Δ
Positive control Standard condition3.38N-130 m3.070.162.910.333.160.08Compliant
3.1460 m3.260.033.130.112.810.08Compliant
3.1890 m4.431.203.520.283.300.06Non-compliant
3.07N-230 m3.640.403.010.233.080.16Compliant
3.3860 m3.290.063.120.123.150.09Compliant
3.2490 m3.540.303.270.043.960.73Non-compliant
Mean3.23
Negative control Standard condition1.98N-130 m1.430.141.430.141.430.14Compliant
1.5360 m1.360.211.620.041.620.14Compliant
1.4390 m1.430.141.430.141.430.14Compliant
1.62N-230 m1.430.141.600.031.430.14Compliant
1.4360 m1.430.141.430.141.430.14Compliant
1.4390 m1.540.031.430.141.430.14Compliant
Mean1.57
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Horbach, I.S.; de Souza Azevedo, A.; Schwarcz, W.D.; Alves, N.d.S.; de Moura Dias, B.; Setatino, B.P.; da Cruz Moura, L.; de Souza, A.F.; Denani, C.B.; da Silva, S.A.; et al. Plaque Reduction Neutralization Test (PRNT) Accuracy in Evaluating Humoral Immune Response to SARS-CoV-2. Diseases 2024, 12, 29. https://doi.org/10.3390/diseases12010029

AMA Style

Horbach IS, de Souza Azevedo A, Schwarcz WD, Alves NdS, de Moura Dias B, Setatino BP, da Cruz Moura L, de Souza AF, Denani CB, da Silva SA, et al. Plaque Reduction Neutralization Test (PRNT) Accuracy in Evaluating Humoral Immune Response to SARS-CoV-2. Diseases. 2024; 12(1):29. https://doi.org/10.3390/diseases12010029

Chicago/Turabian Style

Horbach, Ingrid Siciliano, Adriana de Souza Azevedo, Waleska Dias Schwarcz, Nathalia dos Santos Alves, Brenda de Moura Dias, Bruno Pimenta Setatino, Luma da Cruz Moura, Ariane Faria de Souza, Caio Bidueira Denani, Stephanie Almeida da Silva, and et al. 2024. "Plaque Reduction Neutralization Test (PRNT) Accuracy in Evaluating Humoral Immune Response to SARS-CoV-2" Diseases 12, no. 1: 29. https://doi.org/10.3390/diseases12010029

APA Style

Horbach, I. S., de Souza Azevedo, A., Schwarcz, W. D., Alves, N. d. S., de Moura Dias, B., Setatino, B. P., da Cruz Moura, L., de Souza, A. F., Denani, C. B., da Silva, S. A., Pimentel, T. G., de Oliveira Silva Ferreira, V., Azamor, T., Ano Bom, A. P. D., da Penha Gomes Gouvea, M., Mill, J. G., Valim, V., Polese, J., Campi-Azevedo, A. C., ... de Sousa Junior, I. P. (2024). Plaque Reduction Neutralization Test (PRNT) Accuracy in Evaluating Humoral Immune Response to SARS-CoV-2. Diseases, 12(1), 29. https://doi.org/10.3390/diseases12010029

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop