Clinical Evaluation of Three Commercial RT-PCR Kits for Routine COVID-19 Diagnosis

Amongst the multiple ways to diagnose coronavirus disease-2019 (COVID-19), reverse transcription polymerase chain reaction (RT-PCR) remains the reference gold standard, providing fast and accurate results. This study evaluated and compared the performance of three commercially available COVID-19 RT-PCR kits-Aridia® COVID-19 Real-Time PCR Test (CTK Biotech, Inc., Poway, CA, USA), Novel Coronavirus (2019-nCoV) Nucleic Acid Detection Kit (Sansure Biotech Inc., Changsha, China) and AllplexTM 2019-nCoV assay (Seegene Inc., Seoul, Republic of Korea) for the detection of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). A total of 326 clinically suspected patients were enrolled for the study, and among them, 209 were diagnosed as positive and 117 as negative when tested with the reference method, US CDC 2019-Novel Coronavirus (2019-nCoV) Real Time RT-PCR Diagnostic Panel. The Aridia® kit showed total agreement with the reference test, with a sensitivity of 100% (95% CI: 98.25% to 100.0%) and a specificity of 100% (96.90% to 100.00%). The AllplexTM kit also showed 100% specificity (95% CI: 96.90% to 100.00%), but a lower sensitivity (98.09%, 95% CI: 95.17% to 99.48%). Among the three kits, the Novel Coronavirus (2019-nCoV) Nucleic Acid Detection Kit showed the worst performance, with a sensitivity of 98.6% (95% CI: 95.9% to 99.7%) and a specificity of 95.73, 95% (CI: 90.31% to 98.60%). While all these kits conform to the requirement for routine molecular diagnosis with high performances, the Aridia® COVID-19 Real-Time PCR Test showed the best performance among the three kits.


Introduction
COVID-19 has already caused more than 6.5 million deaths and over 625 million cases all over the world (accessed on 21 October 2022) [1]. For fast diagnosis of the disease, there are numerous rapid tests commercially available in the market [2]. Among other diagnostic methods reverse transcription polymerase chain reaction (RT-PCR), Enzyme-linked Immuno Sorbent Assay (ELISA), radiographic analysis (X-Ray, CT-scan), next-generation sequencing (NGS), and whole genome sequencing (WGS) are currently available [3][4][5][6]. Of these, the most common and accurate method to diagnose COVID-19 is quantitative (real-time) RT-PCR analysis of viral RNA [7]. The most used samples for this analysis are nasal, nasopharyngeal, oropharyngeal or naso-oropharyngeal derived, collected with cotton swabs and then transferred into viral transport medium (VTM) [8]. Viral RNA is then extracted from the VTM and subjected to RT-PCR analysis that provides the SARS-CoV-2 status of the sample.
There are more than 350 commercially available RT-PCR kits that offer highly sensitive and more specific platforms for the diagnosis of COVID-19 [9]. Several commercially available RT-PCR kits for COVID- 19 have obtained authorization for emergency use by various authorities such as World Health Organization (WHO) and the United States food and drug administration (US-FDA). Up to recently, twenty-nine kits have obtained US-FDA Emergency Use Authorization (EUA) [9]. Multiple other kits are also available commercially, but a thorough investigation of their performance is warranted so that the most accurate result possible can be obtained at the patient level.
Coronavirus RNA contains several gene targets for molecular detection and, by the amplification of any one or more target genes by RT-PCR, the presence or absence of the virus that causes COVID-19 can be determined. Some of the most commonly amplified genes include the nucleocapsid (N) gene, envelope (E) gene, RNA-dependent RNA polymerase (RdRP), open reading frame 1ab (ORF1ab), N1 and N2 regions of N gene, etc. [10]. Different manufacturers choose different sets of genes for their RT-PCR kits [11]. The different RT-PCR kits generally contain primers, probes, reverse transcriptase enzymes, PCR buffers, RNase-free water, etc. These components come in different formats and containers, and the user typically has to mix them to prepare a master mix for PCR analysis [4]. Some manufacturers offer more user-friendly kits that include a single tube master mix, and the analysts just have to add the RNA template, which minimizes sample handling errors and setup time [12].
In this study, we evaluated and compared the diagnostic efficacy of these three commercially available RT-PCR kits-Aridia ® COVID-19 Real-Time PCR Test (CTK Biotech, Inc., Poway, CA, USA), Novel Coronavirus (2019-nCoV) Nucleic Acid Detection Kit (Sansure Biotech Inc., Changsha, China) and Allplex TM 2019-nCoV assay (Seegene Inc., Seoul, Republic of Korea) for the purpose of detecting SARS-CoV-2, because these kits are widely used in the diagnostic laboratories of many countries of the world, including ours.

Study Population and Ethical Consideration
The participants of this study were from Dhaka city, Bangladesh, and were symptomatic at the time of sample collection. Patients coming to the diagnostic facility of the International Center for Diarrhoeal Disease Research Bangladesh (icddr,b) in Dhaka, Bangladesh for a COVID-19 RT-PCR test between 3rd February and 8th August of 2021 were enrolled in the study when most of the circulating variants were beta variant followed by the delta variant. The study was conducted with 326 patients showing fever, cough, sore throats and any other COVID-19 symptoms selected and enrolled for testing by healthcare professionals during that time. This study was approved by the institutional Ethical Review Committee (ERC) of icddr,b. Written informed consent was obtained from the adult participants. For children below 11 years, written informed consent was obtained from the parent or legal guardian. In the case of children between 11 and 17 years of age, verbal assent was also obtained in addition to written consent from the parent or legal guardian.

Specimen Collection
During the specimen collection session, nasopharyngeal as well as oropharyngeal swabs were collected from the enrolled participants in 3 mL of VTM by expert healthcare professionals to perform the gold standard RT-PCR assay.

COVID-19 Diagnosis by RT-PCR
RT-PCR amplification of RNA extracted from naso-oropharyngeal swabs collected in VTM from prospective patients was carried out in the Molecular Diagnostic Laboratory, which is an ISO 15189 and ISO 15190 accredited facility.
The naso-oropharyngeal samples were extracted using MagMAX™ Viral/Pathogen II nucleic acid isolation kit with a King Fisher Flex high-throughput automated extraction system (Thermo Fisher Scientific, Waltham, MA, USA).

Aridia ® COVID-19 Real-Time PCR Test
CTK's Aridia ® COVID-19 Real-Time PCR Test packaging contains a COVID-19 PCR mix, COVID-19 positive and negative controls and PCR-grade water is also provided to reconstitute the lyophilized components. The COVID-19 PCR mix contains all real-time PCR components including DNA polymerase, reverse transcriptase, primers, probes, and dNTPs. For the RT-PCR, 15 µL of PCR mix and 5 µL of RNA sample template, positive control and negative control were mixed to target the ORF1ab, N, E, and a housekeeping gene RNase P. PCR was performed according to the manufacturer's instruction. A Bio-Rad CFX Opus96 Real-Time PCR System was used for the amplification and the thermocycler was programmed as followed: reverse transcription (10min, 50 • C, one cycle), (2 min, 95 • C, one cycle), denaturation (5 s, 95 • C, 45 cycles) and finally, annealing/extension (20 s, 60 • C, 45 cycles). At the extension step, fluorescence data are collected through HEX, JOE or VIC (ORF1ab), FAM (N gene), Cy5 (E gene), and ROX (RNase P) channels by Bio-Rad CFX Maestro Software Version 2.2. When N, ORF1ab and E genes showed cycle threshold (Ct) < 40, samples were considered positive and when ≥40, samples were considered negative for COVID-19.

Allplex TM 2019-nCoV Assay (Seegene Inc.)
Components of Seegene's Allplex TM 2019-nCoV assay kit (ref.: RP10252W, Lot: RP5022A01) are slightly different from Aridia ® . This kit contains 2019-nCoV MOM, realtime one-step enzyme, 5X real-time one-step buffer and RNase-free water. For preparing the master mix, 2 µL of real-time one-step enzyme and 5 µL of each of the other three components are mixed and 8 µL of RNA template is added for RT-PCR. For amplification, the same instrument was used as the Aridia ® test kit; the thermocycler protocol was as follows: reverse transcription (20 min, 50 • C, one cycle), initial denaturation (15 min, 95 • C, one cycle), denaturation (15 s, 94 • C, 45 cycles) and finally, annealing (30 s, 58 • C, 45 cycles). Four fluorophores: Quasar 670, FAM, Cal Red 610 and HEX are selected for the analytes: N gene, E gene, RdRp gene and internal control respectively; the same software recorded the fluorescence data. The Ct value of internal control must be <40, otherwise, the test will be considered invalid and that particular sample should be retested for clarification. If the Ct value is <40 for three other genes, or only for the N gene, the sample is considered COVID-19 positive [14]. As per the manufacturer's manual, the volume of the reaction mix is 50 µL (30 µL PCR-master mix and 20 µL RNA template) but in this study, maintaining the recommended proportion, 15 µL reaction mix was prepared (9 µL PCR-master mix and 6 µL RNA template) [15]. Cycle parameters were as follows: reverse transcription (50 • C, 30 min, one cycle), cDNA denaturation (95 • C, 1 min, one cycle), denaturation (95 • C, 15 s, 45 cycles), annealing, extension and fluorescence collection (60 • C, 30 s, 45 cycles) and finally device cooling (25 • C, 10 s, one cycle). FAM(ORF1ab), ROX(N) and CY5(IC) channels were selected for signal receiving. Just like the Allplex TM kit, the internal control gene's Ct value must be <40 to confirm the presence of RNA, otherwise, the samples were retested. To consider a sample COVID-19 positive, the Ct value of the N gene and/or ORF1ab gene must be <40 [15].

Lower Limit of Detection (LLOD), Linearity and Assay Reproducibility
To detect the analytical sensitivity of these three kits, the NGS sequence verified SARS-CoV-2 Plasmid Controls with 200,000 copies/µL (Research Use Only) was diluted sequentially to obtain a set of 100,000 copies/µL to 0.01 copies/µL. For each kit, the RT-PCR was performed in triplicates using similar conditions for each one to detect the LLOD. A standard curve was also generated for each kit and the assay was repeated to verify its linearity and reproducibility. The variation was determined in terms of the Coefficient of Variation, CV (the ratio of the standard deviation, SD to the mean) for intra and interassay [16].

Data Analysis
For statistical analysis and graphical representation of the data, GraphPad Prism 8.0.2 (GraphPad Software, Inc., San Diego, CA, USA) was used and the diagnostic sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive and negative likelihood ratio with respective confidence interval were calculated. MedCalc statistical software (V.20.013) (MedCalc Software, Ostend, Belgium) was also used for building Bland-Altman plots. The Cohen's kappa coefficient was estimated to determine the performance of the kits using SPSS statistics software (IBM, version 20.0.0.2, Armonk, NY, USA) [7].

RT-qPCR Results
Among the 326 prospective COVID-19 patients, 209 (64.10%) were diagnosed positive and the remaining 117 (35.89%) were negative by the reference RT-PCR method. The demographic data can be found in our previous rapid test study [7].

Performance of Aridia ® COVID-19 Real-Time PCR Test
The Aridia ® COVID-19 kit gave similar results to the reference method. Thus, the sensitivity and specificity of the Aridia ® COVID-19 kit were 100% in both cases (95% CI: 98.3% to 100.0% for sensitivity and 96.9% to 100.0% for specificity), compared to the reference method. The performance has been described in Table 1. Comparing this result to that of reference RT-PCR, the sensitivity and specificity of the Allplex TM kit were 98.1% (95% CI: 95.2% to 99.5%) and 100% (95% CI: 96.9% to 100%) respectively. These have been described in brief in Table 1. The sensitivity (98.6%; 95% CI: 95.9% to 99.7%) was similar to the Allplex TM kit, and lower than the Aridia ® COVID-19 kit (Table 1).

Sensitivity in Detection of N Gene
The N gene was a common target of all three kits but their sensitivity of detection was different. The distribution of Ct values of the N gene is summarized in Table 2 and illustrated in Figure 1.

Sensitivity in Detection of N Gene
The N gene was a common target of all three kits but their sensitivity of detection was different. The distribution of Ct values of the N gene is summarized in Table 2 and illustrated in Figure 1.   The Ct values of the N gene were slightly higher for the Allplex TM kits than the Aridia ® kits but for the Sansure kits, they were mostly lower for the corresponding samples (p < 0.001). These have been shown in Figure 2  The Ct values of the N gene were slightly higher for the Allplex TM kits than the Aridia ® kits but for the Sansure kits, they were mostly lower for the corresponding samples (p < 0.001). These have been shown in Figure 2 with a line graph indicating the Ct values of each sample by the terminal of each line.  The difference in Ct values has been also portrayed through the Bland-Altman plot in Figure 3a,b for the N gene of different kits. The analysis shows a significant mean difference of Ct values for Allplex TM -Aridia ® is 2.7 and 1.8 for Aridia ® -Sansure (p < 0.001).
The Ct values of the N gene were slightly higher for the Allplex TM kits than the Aridia ® kits but for the Sansure kits, they were mostly lower for the corresponding samples (p < 0.001). These have been shown in Figure 2 with a line graph indicating the Ct values of each sample by the terminal of each line.

Sensitivity in Detection of E Gene
E gene was a common target of the Aridia ® and Allplex TM kits but their sensitivity of detection was different. The E gene of six positive samples could not be amplified by Allplex TM kits. Ct values were significantly different from each other (p < 0.0001). The distribution of Ct values of the E gene is summarized in Table 3 and illustrated in Figure 4.

Sensitivity in Detection of E Gene
E gene was a common target of the Aridia ® and Allplex TM kits but their sensitivity of detection was different. The E gene of six positive samples could not be amplified by Allplex TM kits. Ct values were significantly different from each other (p < 0.0001). The distribution of Ct values of the E gene is summarized in Table 3 and illustrated in Figure 4.

Sensitivity in Detection of ORF1ab Gene
The ORF1ab gene was a common target of the Aridia ® and Sansure kits but their sensitivity of detection was different. The ORF1ab gene of one sample could not be amplified by the Sansure kit. The distribution of Ct values of the ORF1ab gene is summarized in Table 4 and illustrated in Figure 5.

Sensitivity in Detection of ORF1ab Gene
The ORF1ab gene was a common target of the Aridia ® and Sansure kits but their sensitivity of detection was different. The ORF1ab gene of one sample could not be amplified by the Sansure kit. The distribution of Ct values of the ORF1ab gene is summarized in Table 4 and illustrated in Figure 5. Additionally, the ORF1ab gene of two negative samples was falsely amplified by the Sansure kit with Ct > 35 which was shown in red triangles in Figure 5.

LLOD, Linearity, and Reproducibility
All three real-time RT-PCR kits evaluated in this study could detect as low as 1 copy/μL. Some of these kits could even detect even lower copy numbers but, due to lack of reproducibility, we have found the lower limit of detection as 1 copy/μL. The correlation coefficients (R 2 ) were ≥0.994 and efficiency was around 100% in all cases. The intraassay CV of Ct values for Aridia ® , Allplex TM , and Sansure ranged from 0.17% to 0.97%, 0.26% to 0.54% and 0.31% to 2.82% respectively over six different concentrations. In the case of the Sansure kit, increased variations were observed in low copy numbers. Assay Additionally, the ORF1ab gene of two negative samples was falsely amplified by the Sansure kit with Ct > 35 which was shown in red triangles in Figure 5.

LLOD, Linearity, and Reproducibility
All three real-time RT-PCR kits evaluated in this study could detect as low as 1 copy/µL. Some of these kits could even detect even lower copy numbers but, due to lack of reproducibility, we have found the lower limit of detection as 1 copy/µL. The correlation coefficients (R 2 ) were ≥0.994 and efficiency was around 100% in all cases. The intra-assay CV of Ct values for Aridia ® , Allplex TM , and Sansure ranged from 0.17% to 0.97%, 0.26% to 0.54% and 0.31% to 2.82% respectively over six different concentrations. In the case of the Sansure kit, increased variations were observed in low copy numbers. Assay reproducibility was assessed through inter-assay Ct value variation for the same concentration level in two different independent triplicate runs. The CV for inter-assay fluctuated from 0.8% to 1.28%, 0.02% to 0.63% and 0.01% to 1.17% for Aridia ® , Allplex TM , and Sansure respectively.
The CV values point out the excellent reproducibility of all three assay methods. The LLOD, linearity, repeatability and reproducibility have been summarized in Tables 5-7 and portrayed in Figures 6-8.

Discussion
A number of studies have been done so far to assess the performance of commercially available COVID-19 RT-PCR kits, but there are many kits being used for regular diagnosis that warrant further evaluation as well. Of the kits we evaluated in this study, CTK's Aridia ® Kit was the most convenient compared to the two other kits, as it was an all-in-

Discussion
A number of studies have been done so far to assess the performance of commercially available COVID-19 RT-PCR kits, but there are many kits being used for regular diagnosis that warrant further evaluation as well. Of the kits we evaluated in this study, CTK's Aridia ® Kit was the most convenient compared to the two other kits, as it was an all-in-one master mix in ambient-stable lyophilized form, easy to prepare with the provided nucleasefree water and controls for each assay. The lyophilized format, also eases shipment and storage at room temperature. On the other hand, Allplex TM kit and Sansure kits contain several vials and the contents are in liquid form, which require maintenance of the cold chain (−20 • C to 4 • C) and if the cold chain is interrupted because of logistic issues, the performance of the kits will be compromised.
Although several studies have already evaluated the Allplex TM kit and Sansure Kit [15,[17][18][19], none has been undertaken so far for the Aridia ® kit. Our study evaluated and compared the performance of the Aridia ® kit with already established and available commercial kits that are also highly used in this region, Allplex TM kit and Sansure kit. The Aridia ® kit result showed the best performance among the three kits, with a sensitivity and specificity of 100%. The Allplex TM kit's specificity was also 100%, but it showed higher Ct values for the detection of the N gene (p < 0.001) than the other two kits. The performance of the Sansure Kit in this study fell slightly behind, as it failed to detect three positive and five negative samples correctly.
Compared to our study, a recent study evaluated three PCR kits, namely, GeneFind-erTM (OSANG Healthcare Co., Seongnam, Korea), Sansure Biotech (Sansure Biotech Inc., Changsha, China), and TaqPathTM (Thermo Fisher, Waltham, MA, USA) on 354 randomly selected samples. They showed that the Sansure Biotech assay had better diagnostic performance than the GeneFinderTM and TaqPathTM PCR kits [20]. The Sansure kit was evaluated against the FDA EUA 2019-nCoV CDC kit (IDT, Coralville, IA) and showed 95.3% of sensitivity with a limit of detection as low as 1000 copies/mL in South America [15]. In another study, the Sansure kit had a low sensitivity of 83.3% although a specificity of 100% in a very limited number of samples in a study conducted in Guangxi, China [21]. For routine COVID-19 real-time RT-PCR tests, the Sansure kit had a 90% of positive percent agreement in 86 symptomatic patients in Malaysia [22]. The Sansure kit performs well in samples Ct < 30 with a sensitivity of 94.6% even for extraction-free SARS-CoV-2 assay but the overall sensitivity falls to 69.6% in a sample size of 94 patients with 69 positives. Compared to that, the Allplex TM had a sensitivity of 98.7% in a study with 115 positive samples when NGS was used as a reference in Malaysia [19]. Another study assessed the performance of the Allplex TM in more than 1000 samples including extracted RNA, nasopharyngeal swabs, and saline gargles using the whole genome sequencing process as standard. This kit could easily detect several mutations with 100% sensitivity [18]. The performance can drastically fall to 74% of sensitivity for the extraction-free assay method [23].
In our study, we found the LLOD for all kits to be the same, 1 copy/µL and hence, a comparative analysis could not be drawn among these kits for this parameter. A notable major limitation was the multiple freeze-thaw of the samples. The reference RT-PCR assay was performed within 24 h of sample collection, with the maintenance of the cold chain. Due to limitations of resources and logistics, these kits were evaluated later, with archived extracted RNA that has been thawed multiple times. Another drawback of the study is that we did not have any choice in sample collection, especially with higher Ct values or patients in critical hospital settings for better comparison of the kits as we used the diagnostic facilities only. Additionally, these are commercial kits and hence, they only mentioned their gene target but did not disclose the sequences. Their varied performance can be due to different gene targets than the reference one. The primer can also play a role in better detection but we could not analyze the performance in terms of primer sequences. Nevertheless, these kits performed very well in terms of sensitivity and specificity. These kits are only used for diagnostic purposes, and other studies are required to evaluate these