SARS-CoV-2 Detection Limits in Swab and Gargle Samples by Comparing Antigen and RT-qPCR Testing
, Natascha Borchers 1, Ezgi Cibali 2, Thea Würfel 1, Sarah Nowosadtko 1, Michael Kabesch 1,3,† and Andreas Ambrosch 2,†Round 1
Reviewer 1 Report
Kheiroddin et al. performed an extensive survey comparing nasal and gargle samples for SARS-CoV-2 detection with antigen and RT-PCR tests
Major points:
1) Cohort composition is unclear. Paper only describes “patients and medical students.” Are there pediatric samples? Results and agreement rate between tests are age-correlated? A better description of the patients is more suitable to describe the results. Which was the principal circulating strain during the collection period? This could also change the comprehension of the results.
2) Authors claim that gargle antigen tests were performed “according to the manufacturer’s instructions.” Does the manufacturer recommend gargle testing using their kit? If not, 150µl of gargle was mixed with which amount of extraction buffer? Authors testes other ratios? Authors tested gargles in buffered or non-buffered saline? How the water-dilution of extraction buffer impairs the performance of the test?
3) The authors tested the performance of pooling only with 1:21 sample dilution. However, this implies 4.4 Ct unit displacement, which impairs the detection for >35 Ct values in single samples. Do authors test other pooling proportions to evaluate >34 Ct samples?
4) How do results from LightCycler® Multiplex RNA Virus Master correlate with Xpert Xpress™ SARS-CoV-2 assay? How did single-well dual-target correlate with other quantitative results? Authors should describe the primer-probe pairs used in this type of detection. Were figures 1 and 2 Ct value from which kit?
5) In lines 124-135, the authors should compare gargle RT-PCRs to the gold-standard RT-PCR from nasopharyngeal swabs.
Minor points:
1) Ethics committee approval is missing
2) Lines 37-38: nasopharyngeal samples could also be pooled. Why do authors point to this aspect as a gargle advantage?
3) Lines 56-57: The collection period (at least year/month) is missing. Were Both swabs collected from the same nostril?
4) Line 65: Why were samples stored in 250 ml flasks?
5) Line 104: “table1” for “table 3”.
6) Both figures 1 and 2 should be smaller and not full-page ones. Paired t-test/paired Wilcoxon should be performed in both figures.
7) Line 99: Xpert Xpress™ SARS-CoV-2 assay is not a qualitative test.
Author Response
Major points:
1) Cohort composition is unclear. Paper only describes “patients and medical students.” Are there pediatric samples? Results and agreement rate between tests are age-correlated? A better description of the patients is more suitable to describe the results. Which was the principal circulating strain during the collection period? This could also change the comprehension of the results.
This technical analysis was performed on anonymized samples. Therefore, the origin of samples from the patient or medical student group is documented but no further data have been stored with the samples. All samples were gargle samples. We had extensively studied the applicability of that method to all ages and found gargling a feasible method of sample retrieval at all ages.
The collection period was between October 2019 until April 2020 and thus, the variants predominant in our region at that time was the original SARS-CoV-2 form and the alpha variant becoming dominant in February 2021. This has now been mentioned in the manuscript as follows: “In total, 309 gargle samples and 311 nasal swabs were collected for routine testing in hospitals of the Order of St. John in Regensburg and Straubing, Germany from October 2019 until April 2020 when the original and the Alpha Variant of SARS-CoV-2 was pre-dominant in Germany” (line 58 onwards)
2) Authors claim that gargle antigen tests were performed “according to the manufacturer’s instructions.” Does the manufacturer recommend gargle testing using their kit? If not, 150µl of gargle was mixed with which amount of extraction buffer? Authors testes other ratios? Authors tested gargles in buffered or non-buffered saline? How the water-dilution of extraction buffer impairs the performance of the test?
The manufacturer has validated naso-, oropharyngeal smears and UTM transport medium (Copan, Italy) as suitable materials. We have checked how the sensitivity of the antigen testing is when gargle solution is transferred into the extraction buffer (350 ul) (sensitivity to RT-PCR up to CT values of 30 about 80%; which is in line with the manufacturers information about the tests sensitivity for testing of swabs (CT 25 - 30: 76.5 %)). Furthermore, we also has tested whether gargle with physiological saline solution compared to drinking water is also suitable for diagnostics (not suitable; high rate of false positive results depending on the NaCl concentration). Therefore, we have used tab water water for gargeling.
From experiments with gargle solution compared to swabbing, it is known that the viral load (copies / mL) is reduced by about 1 - 1.5 log levels due to dilution effects (Hitzenbichler F, Bauernfeind S, Salzberger B, Schmidt B, Wenzel JJ. Comparison of Throat Washings, Nasopharyngeal Swabs and Oropharyngeal Swabs for Detection of SARS-CoV-2. Viruses. 2021 Apr 10; 13(4):653. doi: 10.3390/v13040653. PMID: 33920072; PMCID: PMC8069237.)
3) The authors tested the performance of pooling only with 1:21 sample dilution. However, this implies 4.4 Ct unit displacement, which impairs the detection for >35 Ct values in single samples. Do authors test other pooling proportions to evaluate >34 Ct samples?
The focus of this paper was not to systematically test the limits of pool PCR testing, this has already been done and published in papers from us and others (line 38, reference 3-5 of this manuscript). Our aim was to compare the power of pool PCR with antigen testing using an average pool PCR setting. We found, also based on our previous experience, that a pool of 20 is economically reasonable, reduce the time of process, and easy to handle in case of positive. We added this to our method section as follows: “To investigate detectability of individual positive gargle samples in a standardized gargle pool in our PCR setting, we added 1ml of each positive sample with different Ct values (table1) to 20 ml of a negative gargle pool of 20 participants” (line 105 onwards)
4) How do results from LightCycler® Multiplex RNA Virus Master correlate with Xpert Xpress™ SARS-CoV-2 assay? How did single-well dual-target correlate with other quantitative results? Authors should describe the primer-probe pairs used in this type of detection. Were figures 1 and 2 Ct value from which kit?
There was 100% correlation between both assays (results from LightCycler® Multiplex RNA Virus Master correlate with Xpert Xpress™ SARS-CoV-2 assay). We used Ct values from the E gene for the confirmations since it is a common viral target.
We used single-well dual-target (ORF1b and N2 gene) only to show how the gargle pooling is working. We added a sentence in the text that “The further information regarding primer and probes sequences is available in supplementary table 1.” (line 112 onwards)
To test the correlation between different PCR machines, we ran identical samples with increasing Ct values on both systems to show that both have almost the same sensitivity and specificity of detecting SARS-CoV2. We added a paragraph on the comparisons between two systems in text as follow” To show that BIORAD and Light Cycler 480 II Instrument (Roche Diagnostics) have very similar sensitivity and specificity of detecting SARS-CoV2 we ran identical samples on both systems (supplementary table 2) (line 114 onwards)
In figures 1 and 2, Ct values acquired from Light Cycler 480 II Instrument (Roche Diagnostics) using LightCycler® Multiplex RNA Virus Master (target E gene- TibMolBiol). This information now added to the figure. (line 118 onwards)
5) In lines 124-135, the authors should compare gargle RT-PCRs to the gold-standard RT-PCR from nasopharyngeal swabs.
We did not aim to compare the RT-PCR of gargle and swab samples here, this has been done before as we mentioned in our introduction (lines 33-42, reference 2-8 of this manuscript). We focused on antigen test in this paper and in our result (lines 130-138), we compared false negative and positive rate of the antigen test result from swab and gargle samples.
Minor points:
1) Ethics committee approval is missing
As all samples were anonymized for this analysis and only viral features were analyzed, an ethics approval was not required.
2) Lines 37-38: nasopharyngeal samples could also be pooled. Why do authors point to this aspect as a gargle advantage?
Indeed, swabs can also be pooled. However, liquid gargle pool samples are much easier to handle in pooling, making pooling much more efficient as we described it before in our WICOVIR paper (reference 3 of this manuscript). Using gargle sample reduces work load and exposure for lab personnel, prevent of cross contamination and mistake of mixing samples. We modified the text as follows: “ (c) In respect to pooling samples to increase test capacity and save resources, gargle samples offer an easier and safer option since every individual can through his/her sample in the pool container, which also reduce the chance of contamination and mixing samples(3)(4)(5). (line 37 onwards)
3) Lines 56-57: The collection period (at least year/month) is missing. Were Both swabs collected from the same nostril?
We added the time: “from October 2019 until April 2020 when the original and the Alpha Variant of SARS-CoV-2 was pre-dominant in Germany” (line 59 onwards)
4) Line 65: Why were samples stored in 250 ml flasks?
The 250 ml flasks are the original flasks that were used in the hospital to collect gargle fluid. The wide diameter of the flask makes it easy for patients to spit in it. We kept the sample in their original flask to avoid unnecessary waste and risk of contamination.
5) Line 104: “table1” for “table 3”.
Thank you for reading the manuscript so carefully. It corrected now. (line 107 onwards)
6) Both figures 1 and 2 should be smaller and not full-page ones. Paired t-test/paired Wilcoxon should be performed in both figures.
According to the reviewer’s suggestion, we made the figures smaller and put them together as one figure. (line 123onwards)
7) Line 99: Xpert Xpress™ SARS-CoV-2 assay is not a qualitative test.
I am not sure if I got the comment in a correct way. The Xpert Xpress SARS-CoV2 is a RT-PCR based test that have both qualitative and quantitative results; it first shows if the sample is negative or positive and the Ct value of two genes (N2 and E) in case of the positive samples. We used it as secondary test to confirm the results from Light Cycler 480 II Instrument (Roche Diagnostics). However, at the end we did not use the Ct value of the positive samples from Xpert Xpress; we only used the qualitative output from this test, which is that the sample was positive or negative.
Reviewer 2 Report
The paper is well written an adresses issues that are important for the use of SARS-CoV2-RATs (rapid antigen tests). The study does not show novel data; the fact that RATs are much less performant that qPCR when used on nasopharygeal swabs and basically useless for gargles has been shown before. However the the study is well designed and sufficent specimen (n) have been inculded to conclude a sound result, corroborating the already existing data with a high number of specimens analyzed.
Methodological inputs: Please give more manufacturer published perfomance data if available, eg. LODs of PCR(s) used and LOD(s) of RATs used. Using PCR-c(t) values vs. qualitative RAT results depends very strongly on the performance of the PCR itself.
Results: Figures 1 & 2 show to an adequate degree the interdependance of PCR positivity and RAT positivity.
The fact that a relatively high ratio of false positive RAT results was obtained should be discussed further in "Discussion", see https://doi.org/10.4414/smw.2021.20526
Table 1 (pp 137-139) does not appear to hold significant value as presented. The tested speciment number is not sufficient to draw conlcusions here. If possible, repetitive sampling and/or inclusion of more samples to assess the performance of the comparator RATs (NADAL/CLINTEST). It might be worthwile to define ct ranges (high pos/pos/low pos) and test more samples to give an adequate compariosn between the three tests.
Table 2 shows that a simple dilution effect does not contribute to the lower performance of RATs on gargles. This should also be discussed in the Discussion.
Table 3 shows the dilution of gargles in pooled testing. The detected ct values of the dilution fit the theoretically calulatated values (3.3. ct per log dilution) and therefore shows that pool testing is a viable tool to process large amounts of samples economically.
Disussion is well written. Please provide additional points as mintioned above.
Author Response
1) Methodological inputs: Please give more manufacturer published perfomance data if available, eg. LODs of PCR(s) used and LOD(s) of RATs used. Using PCR-c(t) values vs. qualitative RAT results depends very strongly on the performance of the PCR itself.
We mentioned other studies in the discussion part (line 173 and 180 onwards). One of them used the same antigen test that we used in our study (reference 12). We also added one more study please see the answer of question 2. (line 196 onwards)
2) Results: Figures 1 & 2 show to an adequate degree the interdependence of PCR positivity and RAT positivity.
The fact that a relatively high ratio of false positive RAT results was obtained should be discussed further in "Discussion", see https://doi.org/10.4414/smw.2021.20526 .
Based on the reviewer’s suggestion we added one more reference to this part (line 196 onwards)
3) Table 1 (pp 137-139) does not appear to hold significant value as presented. The tested speciment number is not sufficient to draw conlcusions here. If possible, repetitive sampling and/or inclusion of more samples to assess the performance of the comparator RATs (NADAL/CLINTEST). It might be worthwile to define ct ranges (high pos/pos/low pos) and test more samples to give an adequate compariosn between the three tests.
We did not aim to do a statistic comparison with the other two kits; we wanted to see if other kits perform obviously better than the one that we selected. It was not the systematic analyses of the other kits but to assure our selection was not off the average. We explained this now as follows” The other two antigen-tests (CLINITEST / NADAL) were tested as indicator for our selected SD BIOSENSOR FIA test, none of the other two test kits was more appropriate (table1)” (line 143 onwards)
4) Table 2 shows that a simple dilution effect does not contribute to the lower performance of RATs on gargles. This should also be discussed in the Discussion.
We explained it in more details in the discussion “Dilution alone seems to be no reasonable explanation for this dramatic change, even though nucleoprotein antigens of SARS-CoV2 are expected to be more concentrated in a nasal swab than in 10ml of gargle fluid. If dilution would be the main factor, lowering the COI determining a positive result could be a possible strategy to adjust the detection limit. However, lowering the detection limit does not lead to a reasonable increase in sensitivity without an unreasonable decrease in specificity as shown in table 2. Therefore, we conclude, that antigen tests chemistry would have to be specifically designed to work with gargle samples. This is not a specific feature of the test we used here but a common feature of antigen tests on the market designed to be used with nasal swabs, as we show in our experiments” (line 182 onwards)
5) Table 3 shows the dilution of gargles in pooled testing. The detected ct values of the dilution fit the theoretically calulatated values (3.3. ct per log dilution) and therefore shows that pool testing is a viable tool to process large amounts of samples economically.
Thank you for your feedback We agree with you and pointed that in discussion.
Disussion is well written. Please provide additional points as mintioned above.
