Next Article in Journal
Phenotypic and Genotypic Comparison of a Live-Attenuated Genotype I Japanese Encephalitis Virus SD12-F120 Strain with Its Virulent Parental SD12 Strain
Next Article in Special Issue
External Quality Assessment Program for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Logistical Considerations
Previous Article in Journal
The Interplay between HIV-1 Gag Binding to the Plasma Membrane and Env Incorporation
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Viruses
Volume 12
Issue 5
10.3390/v12050550
viruses-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Commentary

External Quality Assessment for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Unique Requirements and Challenges

by
Emma R. Lee
1,
Feng Gao
2,
Paul Sandstrom
1,3 and
Hezhao Ji
1,3,*
1
National HIV and Retrovirology Laboratories, National Microbiology Laboratory at JC Wilt Infectious Diseases Research Centre, Public Health Agency of Canada, Winnipeg, MB R3E 3R2, Canada
2
Department of Medicine, Duke University Medical Center, Durham, NC 27710, USA
3
Department of Medical Microbiology and Infectious Diseases, University of Manitoba, Winnipeg, MB R3E 0J9, Canada
*
Author to whom correspondence should be addressed.
Viruses 2020, 12(5), 550; https://doi.org/10.3390/v12050550
Submission received: 21 April 2020 / Revised: 9 May 2020 / Accepted: 14 May 2020 / Published: 16 May 2020
(This article belongs to the Special Issue Next Generation Sequencing for HIV Drug Resistance Testing)
Browse Figures
Versions Notes

Abstract

:
Over the past decade, there has been an increase in the adoption of next generation sequencing (NGS) technologies for HIV drug resistance (HIVDR) testing. NGS far outweighs conventional Sanger sequencing as it has much higher throughput, lower cost when samples are batched and, most importantly, significantly higher sensitivities for variants present at low frequencies, which may have significant clinical implications. Despite the advantages of NGS, Sanger sequencing remains the gold standard for HIVDR testing, largely due to the lack of standardization of NGS-based HIVDR testing. One important aspect of standardization includes external quality assessment (EQA) strategies and programs. Current EQA for Sanger-based HIVDR testing includes proficiency testing where samples are sent to labs and the performance of the lab conducting such assays is evaluated. The current methods for Sanger-based EQA may not apply to NGS-based tests because of the fundamental differences in their technologies and outputs. Sanger-based genotyping reports drug resistance mutations (DRMs) data as dichotomous, whereas NGS-based HIVDR genotyping also reports DRMs as numerical data (percent abundance). Here we present an overview of the need to develop EQA for NGS-based HIVDR testing and some unique challenges that may be encountered.

1. Introduction

Next-generation sequencing (NGS) or high throughput sequencing has revolutionized DNA sequencing methodology. This technology performs massive parallel sequencing of individual input templates which generates incredible amounts of data per run [1,2]. In comparison, conventional Sanger sequencing (SS) methods produce only a single consensus sequence per specimen, which represents the dominant genotypes in the test sample (Table 1). The main advantage of NGS over SS is its high sensitivity. In HIV drug resistance (HIVDR) genotyping, NGS could detect drug-resistance mutations (DRMs) present at frequencies below 20%, also known as minority resistance variants (MRVs), which are mostly undetectable by SS methods [1,3,4]. There is increasing evidence showing that the presence of MRVs may be clinically relevant since they are associated with an increase in the risk of treatment failure in patients who carry MRVs to the administered antiretroviral drugs [5,6,7,8,9,10,11]. As more labs are implementing NGS as the preferred platform for HIVDR genotyping, the need to develop appropriate external quality assessment (EQA) strategies suitable for such assays is urgent.
As with all other molecular clinical laboratory tests, NGS-based HIVDR assays must implement quality management systems to ensure high-quality test results [12,13,14,15,16]. When a lab starts a new clinical test, the test must be validated with quality assurance measures before the test can be administered for patient care purposes (Figure 1). Generally, a lab can use a commercial kit that has been approved by an appropriate regulatory agency (e.g., FDA), where performance specifications have been validated by the manufacturer. Alternatively, if no commercial options are available, the lab can independently develop its own test, specifically termed “Laboratory Developed Tests” (LDT), or sometimes referred to as “home-brew” or “in-house” tests. There are several considerations involved in LDTs, including target selection and data analysis approaches that are subjected to regulatory bodies such as Clinical and Laboratory Improvement Amendments (CLIA) and Clinical and Laboratory Standards Institute (CSLI) [17,18,19]. The assay must be validated by assessing multiple operational characteristics including accuracy, precision, analytical sensitivity, analytical specificity, robustness, reportable range and reference range [20]. Once the assay is validated and implemented, performance monitoring needs to be conducted regularly to ensure that the test continues to meet performance specifications. Performance monitoring includes internal and external quality controls, as well as participation in an EQA program [14,20].
EQA programs establish the comparison of assay performance amongst different participating labs, helping to identify analytical or interpretive errors, flag areas that need improvement, and identify training needs. EQA program uses methods such as proficiency testing (PT), re-testing, and on-site evaluation to check a lab’s assay performance conducted by a third party agency [16]. PT is the most commonly applied EQA approach for biomedical assays such as HIVDR testing. With a PT-oriented EQA program, participant labs receive blind-coded samples, process the samples as they would do for clinical specimens, and then submit the results to the EQA administrator for performance assessments. The EQA program then analyzes the performance data based on established criteria and returns a summary to each lab, which describes results of how the lab compared with the peers [16,21].

2. Ongoing EQA is Critical for SS-Based HIVDR Genotyping

Currently, PT testing for SS-based HIVDR genotyping is offered by several international agencies or programs including the Virology Quality Assurance (VQA) programs funded through the National Institute of Allergy and Infectious Diseases (NIAID) in the U.S.; the HIVDR Typing Proficiency Program through the Quality Control for Molecular Diagnostics (QCMD) in Europe; the Therapeutics, Research, Education, and AIDS Training (TREAT) Asia Quality Assessment Scheme (TAQAS) in Asia; and the Japanese external quality assessment program (JESQ) in Japan [22,23,24,25]. Apart from JESQ which uses lyophilized in-vitro transcribed RNA as panel samples, other programs use clinical samples or viral isolates that have been well-characterized for viral load and DRMs using commercially available kits.
The HIVDR genotyping assay is a complex multi-procedural process involving both sample processing in the lab and subsequent data analysis using sophisticated bioinformatics tools. All available EQA programs allow labs to perform their LDTs as well as the use of commercially available kits such as ViroSeqTM and TRUGENE TM HIV-1 genotyping systems. Sanger sequences are analyzed using specialized software such as RECall or others to create a consensus sequence for each PT specimen. The Stanford HIV Drug Resistance Database or International AIDS Society-USA (IAS-USA) reference DRM list is then used for HIV DRM identification and clinical interpretation [26,27,28]. At present, most laboratories submit the Sanger sequences (in FASTA format), the HIV DRM reports and the specimen information report via web utility. The data is then compiled and assessed against the peers and proficiency scores for each lab are then determined.
Typically, the EQA program would create a consensus sequence for each PT specimen based on the alignment of all submitted sequences derived from it using similar assays (LDTs or specific commercial assay). For instance, the VQA program requires an ≥ 80% absolute nucleotide agreement for the consensus sequence construction [22]. The QCMD from Europe creates a consensus based on the observations from ≥ 60% of the sequences [23], whereas TAQAS from Asia uses a ≥ 70% threshold [24]. Once constructed, the consensus sequences will then be taken as the reference against which the quality of the sequence submissions from individual labs will be assessed. Proficiency scores are calculated generally based on the number of disagreements from the consensus or reference sequence. Labs are also scored on their ability to detect HIV DRMs identified by the majority or reference sequence. Each participant lab then receives an assessment report that includes sequence alignments, homology tables, mutation outputs and total scores [22,23,24,25,29].

3. The Development of EQA for NGS-Based HIVDR is Essential

As described above, SS-based HIVDR assays have been widely applied for decades with well-defined EQA strategies in place and many sophisticated EQA programs currently in operation. In contrast, implementation of NGS-based HIVDR testing in clinical HIVDR monitoring is still in its infancy, although such technologies have been broadly applied in the research settings since 2007 [30,31]. Through years of research and development efforts, many NGS HIVDR assay protocols with proven performance have been established for effective sample processing, and some bioinformatics pipelines have also become readily available for general end-users [32,33,34,35]. Besides these LDTs, the Vela Sentosa® SQ HIV-1 Genotyping platform has been approved as the first commercial NGS assay for clinical HIVDR typing by regulatory agencies, including the U.S. Food & Drug Administration. Despite these, the EQA component that helps to ensure consistency and high-quality results is still missing for NGS-based HIVDR assays.
It is acknowledged that several recommendations have been made for standardizing NGS-based clinical tests [2,12,13,14,15,36,37]. However, none of these guidelines are specific for NGS assays targeting viral pathogens with significant intra-host diversity. Some effort had been made to conduct EQA on NGS HIVDR assays using the consensus sequences derived from the NGS reads, to approximate SS outputs, by employing the existing PT panels and EQA strategies originally designed for SS-based HIVDR assays [38]. This approach may serve as a transitional resolution for EQA on NGS HIVDR tests. However, it oversimplifies the complexity and richness of NGS HIVDR data when millions of NGS reads are taken into account in the quantitative assessment. Fully validated EQA strategies specific to NGS-based HIVDR testing remain to be established.

4. EQA for NGS-Based HIVDR Assays: Unique Requirements and Challenges

The rapid replication of HIV and the lack of HIV reverse transcription proofreading machinery results in a high mutation rate and highly diversified viral populations or “quasispecies” within a host. Thus, there is a high genetic variability of HIV viruses that already exist among the infected subjects. Consequently, because the HIVDR genotyping assay needs to analyze such a complex, heterogeneous viral population, it will not be as straightforward as sequencing prokaryotic and eukaryotic genomes which have significantly less genetic variation. NGS enables analysis of individual HIV genomes in a quasispecies population with high resolution. However, the high volume and complexity of NGS data further convolute the EQA effort that aims to operationalize NGS HIVDR assays for clinical applications. Some unique requirements and challenges in EQA development for NGS-based HIVDR assays are highlighted below.

4.1. PT Panel Design for NGS-Based HIVDR Testing

PT, as part of EQA for NGS-based HIVDR testing, is imperative for the validation of such assays. The panel composition for current SS-based EQA programs is plasma, serum or dried blood spots consisting of donor specimens, clinical isolates or infectious molecular clones [22,23,24,39], as well as lyophilized in-vitro transcribed RNA as panel samples [25]. Similar sample types could be adapted for NGS-based proficiency panels which are well quantitatively characterized for the relative abundance of DRMs found in each sample.
Table 2 summarizes some PT sample options for NGS-based HIVDR assays and their advantages and disadvantages when being utilized for EQA purposes. The use of various infectious molecular clones (IMC) that have defined DRMs could be mixed at different ratios to generate DRMs at different frequencies. The mixed IMC could be accurately characterized by single genome sequencing [40] or by parallel allele-specific sequencing [41]. One of the challenges in using IMC is that spontaneous mutations may be introduced during propagation [42], although such mutations are generally rare and random [43]. The use of plasmid mixtures harboring well-characterized DRMs holds the most promise, where true frequencies of HIV DRMs would be consistent. Although not a true clinical sample, plasmids and plasmid mixtures with confirmed ratios could be established and used as reference materials for detecting DRMs with varied abundance, particularly at lower frequencies, and for monitoring systematic error [32,44]. Alternatively, in vitro transcription of viral RNA from a construct that contains a T7 promoter could be used to generate bulk RNA templates for use as reference materials, similar to those used in the JEQS program [25,45,46,47]. These templates could be spiked into appropriate mediums such as plasma, serum or blood. Theoretically, the transcripts would be clonal with low error due to the T7 RNA polymerase (nucleotide substitution error rate at 10−4–10−5) [48,49]. The templates could be characterized with NGS and the consensus could be used as the reference material sequence. Deviations from the reference material sequence would be interpreted as RT-PCR/PCR error, PCR bias and sequencing error [3,50].
One of the disadvantages of using IMC, plasmids and in vitro-generated RNA is that these sample types are homogenous viral populations and do not represent authentic clinical specimens that usually contain highly diversified HIV genomes. As a result, the use of such samples oversimplifies the NGS-based HIVDR assay. Plasmids, because they are DNA-based, completely omit the RT-PCR step found in the early steps of the assay. On the other hand, the use of traditional Sanger-based PT samples which better represent clinical specimens are difficult to characterize, and the ground truth of the exact frequencies of the DRMs they harbor may not be reliably determined. Therefore a careful combination of real samples, plasmids, IMC and/or in vitro generated RNAs that contain DRMs of interest and at known abundance may be required to assess the full scope of the NGS-based HIVDR assays.

4.2. Data Collection for NGS-Based HIVDR Testing

The data collected from the participating labs are the only information source for all EQA programs. In SS-based HIVDR EQA programs, laboratories are required to submit the consensus sequences, the HIV DRM reports for the panel specimens and an assay summary or specimen information report. The assay summary includes information about wet-lab methods such as extraction, RT-PCR, PCR, sequencing protocols and software used to analyze sequencing data. While these satisfy all the EQA needs for SS-based HIVDR assays, additional information is required for NGS-based HIVDR assays. Some EQA data collection considerations are summarized in Figure 2.
The recommended data collections aim to facilitate an objective assessment of the lab’s capacity and quality in performing NGS-based HIVDR assay by (1) gauging the initial HIV template input and the subsequent sensitivity for HIV DRM detection it may enable (data from Step 1); (2) evaluating how different NGS platforms may affect the final assay outcome (data from step 2); (3) assessing NGS consensus sequences with specified minor allele identification threshold(s) (data from Step 3); (4) assessing NGS-specific EQA data using quantitative HIVDR reports and comprehensive amino acid variant files (AAVFs) or equivalent, which report all amino acid variants at all frequencies for each specimen (data from Step 3) (Figure 2). Given the ultra-deep sequencing capabilities of NGS platforms, theoretically, NGS-based HIVDR assays could detect all HIV amino acid variations at all frequencies that are above the assay error rate. However, the assay sensitivity limit is ultimately determined by the initial HIV viral input copies, therefore, collecting assay details from Step 1 is essential. The use of unique molecular identifiers (UMIs) is highly recommended when possible, as UMIs can detect the number of actual templates used in the initial RT-PCR reaction and can correct for PCR and sequencing errors [52,53,54].

4.3. EQA: Data Assessment and Scoring Strategies for NGS-Based HIVDR Testing

In 2018, a joint project involving ten labs in Canada, USA, Mexico and Europe was conducted to assess whether panels from the NIAID’s HIV VQA program could be used for NGS-based proficiency testing [38]. Two panels, each containing five specimens (real donors or plasma spiked with clinical viral isolates), at various viral loads ranging from 3656 to 29,139 copies/ml were distributed to the participant labs and were processed using their independent LDTs for NGS-based HIVDR typing. Consensus sequences derived from all labs using varied NGS platforms were collected at different frequency thresholds specifically, 5%, 10%, 15% and 20% and were compared. Results from the sequence homology analysis which was assessed using the current VQA scoring methods showed high nucleotide concordance at all examined thresholds [38]. To evaluate the DRM frequency readouts, raw NGS reads (FASTQ files) from six of the ten labs were processed through the HyDRA pipeline [32], and the HIV DRM reports derived from the NGS reads were compared (Table S1). Figure 3, which was previously presented at XXVII International HIV Drug Resistance and Treatment Strategies Workshop in Johannesburg, South Africa (October 2018), shows the DRM frequency readouts for all the DRMs found in the two panels at a median threshold of 5%. The frequency of the DRMs widely fluctuates, especially when their abundance was below 90% [55]. The volatility of the DRM frequency results implies that innovative statistical scoring and assessment methods will have to be developed to assess the NGS HIVDR assay outputs from different labs, especially when dealing with the following issues.

4.3.1. Inconsistencies in Detecting DRMs

The data shown in Figure 3 demonstrates that the DRM frequency outputs from each lab participant are inconsistent. In most cases, all labs were able to detect the DRMs present in the VQA samples, albeit at different frequencies. However, there were cases where one or two labs could not detect a DRM, which was otherwise identified by the majority of the labs. One suggestion, raised at the 2019 International Symposium on NGS HIVDR, was that DRM frequency readouts obtained from each lab would not be directly assessed. Instead, the threshold for the NGS consensus sequences should be lowered from 20% to 5%, and the ability of whether the lab could qualitatively detect the presence of the DRM in the group consensus would be assessed. However, even with this method of evaluating NGS data, the challenge of whether a lab receives a penalty for not detecting a mutation at the suggested 5% threshold is complex. Whether the pre-characterization of the panel samples would be considered as the standard, or the standard should base on the consensus of the participants’ results has not been resolved. If the consensus of the participants’ results from PT is set as the standard for evaluating whether a mutation is truly present, the percentage that comprises the consensus is debatable. For example, would the current VQA measure of 80% of participant results comprise the consensus, or would the value of 80% need to be lower to assess MRVs? This method of assessment could use the proposed “assessment panel” described previously by H. Ji et al. (2020) and current EQA scoring strategies could be applied.

4.3.2. Large Variations in DRM Frequencies

One of the main advantages of NGS is that the relative abundance (% frequency) of DRMs can be determined. To assess the accuracy of the detected relative abundance, one would require the proposed “verification panel” with samples for which DRMs and exact frequencies have been well-characterized [56]. However, large variations were observed in DRM frequency readouts from different NGS assays while SS EQA panel specimens were tested. Notably, the higher the median frequencies are, the lower inter-lab variations were observed (Figure 3). If frequency measurement is taken into account, would it be assessed in the frequency intervals (e.g. 5%–20%, 20%–90%, 90%–100%) with different respective deviation thresholds, or would one general deviation be used for assessment for all frequencies? We believe it is fair to apply a higher acceptable deviation cut-off for those DRM frequencies at lower frequency brackets for EQA purposes. The difficulty in determining a method to assess frequency read-outs also highlights the need for more comprehensive studies to better define practical deviation cut-off values for various DRM frequency levels. This type of research would also aide in the development of an appropriate statistical method for DRM frequency assessment and help to identify whether participants can consistently detect MRVs.

4.3.3. Variations in Wet-Lab Methods, NGS Platforms and Bioinformatics Pipelines

Current SS-based EQA strategies for HIVDR genotyping accept LDTs, however, NGS-based HIVDR assays introduce more variations as different NGS technologies and diverse bioinformatics pipelines are utilized [32,33,34,35,57,58,59,60,61,62,63]. NGS platform-specific EQA strategies may not be necessary, although sequence-specific errors for different NGS systems exist [58]. Many different bioinformatics pipelines have been developed for HIVDR testing [32,33,34,35,57,59,60,61,62,63]. Recently, we compared and assessed the performance of five pipelines to detect amino acid variants within specimens and found they were highly concordant. All pipelines demonstrated high sensitivity detecting variants even at a 1% threshold. Specificity, on the other hand, was much better at a 2% threshold versus a 1% threshold. Results from this study imply that a 2% threshold may be a more reliable cut-off for drug resistance mutation calling and reporting when NGS technologies are utilized [64]. As addressed previously, careful documentation of all wet-lab and dry-lab steps may aide in EQA and appropriate remedial recommendations.

5. Conclusions

NGS is a powerful tool with the ability to detect variants at low-frequency levels that were otherwise undetectable by Sanger sequencing. Research continues to define the clinical importance of MRVs and inform how drug regimens for HIV treatment should be modified when MRVs for DR are detected. The current EQA strategies for SS-based HIVDR testing are insufficient for NGS-based testing because of the drastically different outputs between the two technologies. Unlike SS, NGS can identify MRVs and provides numerical data on the frequencies of DRMs. Well-characterized panels, including the use of clinical specimens, plasmids and in vitro generated HIV RNA templates, are required to assess the ability of a lab to detect MRVs accurately. Although large variations occur in DRM frequencies when comparing different labs performing NGS-based HIVDR testing on the same panels, continued efforts are being made to develop appropriate assessment methods to address this issue. Future studies to include a larger group to not only assess DRMs but all amino acid variations (AAVs) will provide more information on the comparability between labs and how consistent they are in detecting MRVs. Meanwhile, accommodation of both LDTs and commercial NGS assays, such as the Sentosa® SQ HIV-1 assay, in the perspective EQA programs would be essential. The implementation of EQA for NGS-HIVDR is challenging, but with the urgent need for more accurate assays to detect MRVs and the continued efforts from the field, the thoughts can become a reality.

Supplementary Materials

The following are available online at https://www.mdpi.com/1999-4915/12/5/550/s1.

Author Contributions

E.R.L. drafted the manuscript. H.J. and F.G. edited the manuscript. All authors contributed to the manuscript planning and gave consent to this submission. All authors have read and agreed to the published version of the manuscript

Funding

We appreciate the great in-kind support from all participating institutes and the financial supports from funding agencies, including the Canadian Federal Initiative to Address HIV and AIDS (PS, HJ, ERL). This study was also supported by the National Institute of Allergy and Infectious Diseases (NIAID), National Institutes of Health (NIH) for Virology Quality Assurance (VQA; 75N93019C00015) (FG). The APC was funded by the Public Health Agency of Canada, Canada.

Acknowledgments

Part of the data from this study was presented at the 27th International Workshop on HIV Drug Resistance and Treatment Strategies (October 2018, Johannesburg, South Africa), and the International Symposium on External Quality Assurance Strategies for NGS-based HIV drug resistance testing (9–10 September 2019, Winnipeg, Canada).

Conflicts of Interest

The authors have no competing interests to declare.

References

  1. Voelkerding, K.V.; Dames, S.A.; Durtschi, J.D. Next-generation sequencing:from basic research to diagnostics. Clin. Chem. 2009, 55, 641–658. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Maljkovic Berry, I.; Melendrez, M.C.; Bishop-Lilly, K.A.; Rutvisuttinunt, W.; Pollett, S.; Talundzic, E.; Morton, L.; Jarman, R.G. Next Generation Sequencing and Bioinformatics Methodologies for Infectious Disease Research and Public Health: Approaches, Applications, and Considerations for Development of Laboratory Capacity. J. Infect. Dis. 2020, 221, S292–S307. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  3. Brumme, C.J.; Poon, A.F.Y. Promises and pitfalls of Illumina sequencing for HIV resistance genotyping. Virus Res. 2017, 239, 97–105. [Google Scholar] [CrossRef] [PubMed]
  4. Casadellà, M.; Paredes, R. Deep sequencing for HIV-1 clinical management. Virus Res. 2017, 239, 69–81. [Google Scholar] [CrossRef]
  5. Hamers, R.L.; Schuurman, R.; Sigaloff, K.C.E.; Wallis, C.L.; Kityo, C.; Siwale, M.; Mandaliya, K.; Ive, P.; Botes, M.E.; Wellington, M.; et al. Effect of pretreatment HIV-1 drug resistance on immunological, virological, and drug-resistance outcomes of first-line antiretroviral treatment in sub-Saharan Africa: A multicentre cohort study. Lancet Infect. Dis. 2012, 12, 307–317. [Google Scholar] [CrossRef]
  6. Boender, T.S.; Hoenderboom, B.M.; Sigaloff, K.C.E.; Hamers, R.L.; Wellington, M.; Shamu, T.; Siwale, M.; Labib Maksimos, E.E.F.; Nankya, I.; Kityo, C.M.; et al. Pretreatment HIV drug resistance increases regimen switches in sub-saharan Africa. Clin. Infect. Dis. 2015, 61, 1749–1758. [Google Scholar] [CrossRef]
  7. Ávila-Ríos, S.; García-Morales, C.; Matías-Florentino, M.; Romero-Mora, K.A.; Tapia-Trejo, D.; Quiroz-Morales, V.S.; Reyes-Gopar, H.; Ji, H.; Sandstrom, P.; Casillas-Rodríguez, J.; et al. Pretreatment HIV-drug resistance in Mexico and its impact on the effectiveness of first-line antiretroviral therapy: A nationally representative 2015 WHO survey. Lancet HIV 2016, 3, e579–e591. [Google Scholar] [CrossRef]
  8. Vandenhende, M.A.; Bellecave, P.; Recordon-Pinson, P.; Reigadas, S.; Bidet, Y.; Bruyand, M.; Bonnet, F.; Lazaro, E.; Neau, D.; Fleury, H.; et al. Prevalence and evolution of low frequency HIV drug resistance mutations detected by ultra deep sequencing in patients experiencing first line antiretroviral therapy failure. PLoS ONE 2014, 9, e86771. [Google Scholar] [CrossRef]
  9. Kantor, R.; Smeaton, L.; Vardhanabhuti, S.; Hudelson, S.E.; Wallis, C.L.; Tripathy, S.; Morgado, M.G.; Saravanan, S.; Balakrishnan, P.; Reitsma, M.; et al. Pretreatment HIV Drug Resistance and HIV-1 Subtype C Are Independently Associated with Virologic Failure: Results from the Multinational PEARLS (ACTG A5175) Clinical Trial. Clin. Infect. Dis. 2015, 60, 1541–1549. [Google Scholar] [CrossRef] [Green Version]
  10. Cozzi-Lepri, A.; Noguera-Julian, M.; Di Giallonardo, F.; Schuurman, R.; Däumer, M.; Aitken, S.; Ceccherini-Silberstein, F.; Monforte, A.D.A.; Geretti, A.M.; Booth, C.L.; et al. Low-frequency drug-resistant HIV-1 and risk of virological failure to first-line NNRTI-based ART: A multicohort European case-control study using centralized ultrasensitive 454 pyrosequencing. J. Antimicrob. Chemother. 2015, 70, 930–940. [Google Scholar] [CrossRef] [Green Version]
  11. Inzaule, S.C.; Hamers, R.L.; Noguera-Julian, M.; Casadellà, M.; Parera, M.; Kityo, C.; Steegen, K.; Naniche, D.; Clotet, B.; Rinke de Wit, T.F.; et al. Clinically relevant thresholds for ultrasensitive HIV drug resistance testing: A multi-country nested case-control study. Lancet HIV 2018, 5, e638–e646. [Google Scholar] [CrossRef]
  12. Endrullat, C.; Glökler, J.; Franke, P.; Frohme, M. Standardization and quality management in next-generation sequencing. Appl. Transl. Genom. 2016, 10, 2–9. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Gargis, A.S.; Kalman, L.; Lubin, I.M. Assuring the quality of next-generation sequencing in clinical microbiology and public health laboratories. J. Clin. Microbiol. 2016, 54, 2857–2865. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Gargis, A.S.; Kalman, L.; Berry, M.W.; Bick, D.P.; Dimmock, D.P.; Hambuch, T.; Lu, F.; Lyon, E.; Voelkerding, K.V.; Zehnbauer, B.A.; et al. Assuring the quality of next-generation sequencing in clinical laboratory practice. Nat. Biotechnol. 2012, 30, 1033–1036. [Google Scholar] [CrossRef]
  15. Hutchins, R.J.; Phan, K.L.; Saboor, A.; Miller, J.D.; Muehlenbachs, A. Practical Guidance to Implementing Quality Management Systems in Public Health Laboratories Performing Next-Generation Sequencing: Personnel, Equipment, and Process Management (Phase 1). J. Clin. Microbiol. 2019, 5, e00261-19. [Google Scholar] [CrossRef] [Green Version]
  16. WHO: Content Sheet 10-1: Overview of External Quality Assessment (EQA). Available online: https://www.who.int/ihr/training/laboratory_quality/10_b_eqa_contents.pdf (accessed on 10 February 2020).
  17. Wayne, P.; CSLI. Quality System Regulation for Laboratory-Developed Tests: A Practical Guide for the Laboratory; Clinical Laboratory Standards Institute: Stanford, CA, USA, 2015. [Google Scholar]
  18. Genzen, J.R. Regulation of Laboratory-Developed Tests. Am. J. Clin. Pathol. 2019, 152, 122–131. [Google Scholar] [CrossRef]
  19. U.S. Department of Health and Human Services; Rockville, M.; CMS. Clinical Laboratory Improvement Ammendments of 1988 (part 493); Centers for Medicare and Medicaid Services: Baltimore, MD, USA, 1988. [Google Scholar]
  20. (CLSI 2014) Nucleic Acid Sequencing Methods in Diagnostic Laboratory Medicine; Approved Guideline; MM09-A2 2014; CLSI: Stanford, CA, USA, 2014; p. 34.
  21. Kalman, L.V.; Lubin, I.M.; Barker, S.; Du Sart, D.; Elles, R.; Grody, W.W.; Zehnbauer, B. Current Landscape and New Paradigms of Proficiency Testing and External Quality Assessment for Molecular Genetics. Arch. Pathol. Lab. Med. 2013, 137, 983–988. [Google Scholar] [CrossRef] [Green Version]
  22. Parkin, N.; Bremer, J.; Bertagnolio, S. Genotyping external quality assurance in the world health organization HIV drug resistance laboratory network during 2007–2010. Clin. Infect. Dis. 2012, 54, S266–S272. [Google Scholar] [CrossRef] [Green Version]
  23. Pandit, A.; Mackay, W.G.; Steel, C.; van Loon, A.M.; Schuurman, R. HIV-1 drug resistance genotyping quality assessment: Results of the ENVA7 Genotyping Proficiency Programme. J. Clin. Virol. 2008, 43, 401–406. [Google Scholar] [CrossRef]
  24. Land, S.; Cunningham, P.; Zhou, J.; Frost, K.; Katzenstein, D.; Kantor, R.; Chen, Y.-M.A.; Oka, S.; DeLong, A.; Sayer, D.; et al. TREAT Asia Quality Assessment Scheme (TAQAS) to standardize the outcome of HIV genotypic resistance testing in a group of Asian laboratories. J. Virol. Methods 2009, 159, 185–193. [Google Scholar] [CrossRef] [Green Version]
  25. Yoshida, S.; Hattori, J.; Matsuda, M.; Okada, K.; Kazuyama, Y.; Hashimoto, O.; Ibe, S.; Fujisawa, S.I.; Chiba, H.; Tatsumi, M.; et al. Japanese external quality assessment program to standardize HIV-1 drug-resistance testing (JEQS2010 program) using in vitro transcribed RNA as reference material. AIDS Res. Hum. Retrovir. 2015, 31, 318–325. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  26. Harrigan, P.R.; Trevino, C.; Hall, T.A.; Chui, C.K.S.; Wynhoven, B.; Woods, C.K.; Liu, T.F.; Chu, A.L.; Shafer, R.W.; Brumme, C.J. Automating HIV Drug Resistance Genotyping with RECall, a Freely Accessible Sequence Analysis Tool. J. Clin. Microbiol. 2012, 50, 1936–1942. [Google Scholar]
  27. Liu, T.F.; Shafer, R.W. Web Resources for HIV Type 1 Genotypic-Resistance Test Interpretation. Clin. Infect. Dis. 2006, 42, 1608–1618. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  28. Wensing, A.M.; Calvez, V.; Ceccherini-Silberstein, F.; Charpentier, C.; Günthard, H.F.; Paredes, R.; Shafer, R.W.; Richman, D.D. 2019 update of the drug resistance mutations in HIV-1. Top. Antivir. Med. 2019, 27, 111–121. [Google Scholar] [PubMed]
  29. WHO. Who/Hivresnet HIV Drug Resistance; WHO: Geneva, Switzerland, 2017; ISBN 9789241512879. [Google Scholar]
  30. Wang, C.; Mitsuya, Y.; Gharizadeh, B.; Ronaghi, M.; Shafer, R.W. Characterization of mutation spectra with ultra-deep pyrosequencing: Application to HIV-1 drug resistance. Genome Res. 2007, 17, 1195–1201. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  31. Hoffmann, C.; Minkah, N.; Leipzig, J.; Wang, G.; Arens, M.Q.; Tebas, P.; Bushman, F.D. DNA bar coding and pyrosequencing to identify rare HIV drug resistance mutations. Nucleic Acids Res. 2007, 35, e91. [Google Scholar] [CrossRef] [Green Version]
  32. Taylor, T.; Lee, E.R.; Nykoluk, M.; Enns, E.; Liang, B.; Capina, R.; Gauthier, M.K.; Van Domselaar, G.; Sandstrom, P.; Brooks, J.; et al. A MiSeq-HyDRA platform for enhanced HIV drug resistance genotyping and surveillance. Sci. Rep. 2019, 9, 1–11. [Google Scholar] [CrossRef] [Green Version]
  33. Noguera-Julian, M.; Edgil, D.; Harrigan, P.R.; Sandstrom, P.; Godfrey, C.; Paredes, R. Next-Generation Human Immunodeficiency Virus Sequencing for Patient Management and Drug Resistance Surveillance. J. Infect. Dis. 2017, 216, S829–S833. [Google Scholar] [CrossRef]
  34. Chui, C.K.S.; Dong, W.W.Y.; Joy, J.B.; Poon, A.F.Y.; Dong, W.Y.; Mo, T.; Woods, C.K.; Beatty, C.; Hew, H.; Harrigan, P.R.; et al. Development and validation of two screening assays for the hepatitis C virus NS3 Q80K polymorphism associated with reduced response to combination treatment regimens containing simeprevir. J. Clin. Microbiol. 2015, 53, 2942–2950. [Google Scholar] [CrossRef] [Green Version]
  35. Howison, M.; Coetzer, M.; Kantor, R. Measurement error and variant-calling in deep Illumina sequencing of HIV. Bioinformatics 2019, 35, 2029–2035. [Google Scholar] [CrossRef]
  36. Aziz, N.; Zhao, Q.; Bry, L.; Driscoll, D.K.; Funke, B.; Gibson, J.S.; Grody, W.W.; Hegde, M.R.; Hoeltge, G.A.; Leonard, D.G.B.; et al. College of American pathologists’ laboratory standards for next-generation sequencing clinical tests. Arch. Pathol. Lab. Med. 2015, 139, 481–493. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  37. Lefterova, M.I.; Suarez, C.J.; Banaei, N.; Pinsky, B.A. Next-Generation Sequencing for Infectious Disease Diagnosis and Management: A Report of the Association for Molecular Pathology. J. Mol. Diagn. 2015, 17, 623–634. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  38. Parkin, N.; Zaccaro, D.; Avila-Rios, S.; Brumme, C.; Hunt, G.; Ji, H.; Kantor, R.; Mbisa, J.; Paredes, R.; Rivera-Amill, V.; et al. Multi-Laboratory Comparison of Next-Generation to Sanger-Based Sequencing for HIV-1 Drug Resistance Genotyping. In Proceedings of the XXVII International HIV Drug Resistance and Treatment Strategies Workshop, Johannesburg, South Africa, 22–23 October 2018. [Google Scholar]
  39. WHO. World Health Organization Global Strategy For The Surveillance And Monitoring Of HIV Drug Resistance. HIV/AIDS Programme. 2012. Available online: http:/www.who.int/hiv/pub/drugresistance/drug_resistance_strategy/en/ (accessed on 10 February 2020).
  40. Palmer, S.; Kearney, M.; Maldarelli, F.; Halvas, E.K.; Bixby, C.J.; Bazmi, H.; Rock, D.; Falloon, J.; Davey, R.T.; Dewar, R.L.; et al. Multiple, linked human immunodeficiency virus type 1 drug resistance mutations in treatment-experienced patients are missed by standard genotype analysis. J. Clin. Microbiol. 2005, 43, 406–413. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  41. Cai, F.; Chen, H.; Hicks, C.B.; Bartlett, J.A.; Zhu, J.; Gao, F. Detection of minor drug-resistant populations by parallel allele-specific sequencing. Nat. Methods 2007, 4, 123–125. [Google Scholar] [CrossRef]
  42. Sanjuán, R.; Domingo-Calap, P. Mechanisms of viral mutation. Cell. Mol. Life Sci. 2016, 73, 4433–4448. [Google Scholar] [CrossRef] [Green Version]
  43. Song, H.; Pavlicek, J.W.; Cai, F.; Bhattacharya, T.; Li, H.; Iyer, S.S.; Bar, K.J.; Decker, J.M.; Goonetilleke, N.; Liu, M.K.P.; et al. Impact of immune escape mutations on HIV-1 fitness in the context of the cognate transmitted/founder genome. Retrovirology 2012, 9, 1. [Google Scholar] [CrossRef] [Green Version]
  44. Kireev, D.E.; Lopatukhin, A.E.; Murzakova, A.V.; Pimkina, E.V.; Speranskaya, A.S.; Neverov, A.D.; Fedonin, G.G.; Fantin, Y.S.; Shipulin, G.A. Evaluating the accuracy and sensitivity of detecting minority HIV-1 populations by Illumina next-generation sequencing. J. Virol. Methods 2018, 261, 40–45. [Google Scholar] [CrossRef]
  45. Orton, R.J.; Wright, C.F.; Morelli, M.J.; King, D.J.; Paton, D.J.; King, D.P.; Haydon, D.T. Distinguishing low frequency mutations from RT-PCR and sequence errors in viral deep sequencing data. BMC Genom. 2015, 16, 229. [Google Scholar] [CrossRef] [Green Version]
  46. Kugelman, J.R.; Wiley, M.R.; Nagle, E.R.; Reyes, D.; Pfeffer, B.P.; Kuhn, J.H.; Sanchez-Lockhart, M.; Palacios, G.F. Error baseline rates of five sample preparation methods used to characterize RNA virus populations. PLoS ONE 2017, 12, e0171333. [Google Scholar] [CrossRef]
  47. Santiago, G.A.; Vergne, E.; Quiles, Y.; Cosme, J.; Vazquez, J.; Medina, J.F.; Medina, F.; Colón, C.; Margolis, H.; Muñoz-Jordán, J.L. Analytical and Clinical Performance of the CDC Real Time RT-PCR Assay for Detection and Typing of Dengue Virus. PLoS Negl. Trop. Dis. 2013, 7, e2311. [Google Scholar] [CrossRef]
  48. Long, C.; Chao, E.; Da, L.T.; Yu, J. A Viral T7 RNA Polymerase Ratcheting Along DNA With Fidelity Control. Comput. Struct. Biotechnol. J. 2019, 17, 638–644. [Google Scholar] [CrossRef] [PubMed]
  49. Huang, J.; Brieba, L.G.; Sousa, R. Misincorporation by wild-type and mutant T7 RNA polymerases: Identification of interactions that reduce misincorporation rates by stabilizing the catalytically incompetent open conformation. Biochemistry 2000, 39, 11571–11580. [Google Scholar] [CrossRef] [PubMed]
  50. Mcelroy, K.; Thomas, T.; Luciani, F. Deep sequencing of evolving pathogen populations : Applications, errors, and bioinformatic solutions. Microb. Inform. Exp. 2014, 4, 1. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  51. Andino, R.; Domingo, E. Viral Quasi Species. Virology 2015, 344, 1173–1178. [Google Scholar]
  52. Boltz, V.F.; Rausch, J.; Shao, W.; Hattori, J.; Luke, B.; Maldarelli, F.; Mellors, J.W.; Kearney, M.F.; Coffin, J.M. Ultrasensitive single-genome sequencing: Accurate, targeted, next generation sequencing of hiv-1 rna. Retrovirology 2016, 13, 87. [Google Scholar] [CrossRef] [Green Version]
  53. Zhou, S.; Jones, C.; Mieczkowski, P.; Swanstrom, R. Primer ID Validates Template Sampling Depth and Greatly Reduces the Error Rate of Next-Generation Sequencing of HIV-1 Genomic RNA Populations. J. Virol. 2015, 89, 8540–8555. [Google Scholar] [CrossRef] [Green Version]
  54. Kou, R.; Lam, H.; Duan, H.; Ye, L.; Jongkam, N.; Chen, W.; Zhang, S.; Li, S. Benefits and Challenges with Applying Unique Molecular Identifiers in Next Generation Sequencing to Detect Low Frequency Mutations. PLoS ONE 2016, 11, e0146638. [Google Scholar] [CrossRef] [Green Version]
  55. Lee, E.R.; Enns, E.; Parkin, N.; Brumme, C.J.; Casadella, M.; Howison, M.; Avila Rios, S.; Jennings, R.; Capina, R.; Marinier, E.; et al. Characterization and data assessment of next generation sequencing-based genotyping using existing HIV-1 drug resistance proficiency panels. In Proceedings of the XXVII International HIV Drug Resistance and Treatment Strategies Workshop, Johannesburg, South Africa, 22–23 October 2018. [Google Scholar]
  56. Ji, H.; Parkin, N.; Gao, F.; Denny, T.; Jennings, C.; Bremer, J.; Kantor, R. External quality assessment program for next generation sequencing-based HIV drug resistance testing: Logistical considerations. Viruses 2020. (accepted on 14 May 2020).. [Google Scholar]
  57. Gibson, R.M.; Meyer, A.M.; Winner, D.; Archer, J.; Feyertag, F.; Ruiz-Mateos, E.; Leal, M.; Robertson, D.L.; Schmotzer, C.L.; Quiñones-Mateu, M.E. Sensitive Deep-Sequencing-Based HIV-1 Genotyping Assay To Simultaneously Determine Susceptibility to Protease, Reverse Transcriptase, Integrase, and Maturation Inhibitors, as Well as HIV-1 Coreceptor Tropism. Antimicrob. Agents Chemother. 2014, 58, 2167–2185. [Google Scholar] [CrossRef] [Green Version]
  58. Goodwin, S.; McPherson, J.D.; McCombie, W.R. Coming of age: Ten years of next-generation sequencing technologies. Nat. Rev. Genet. 2016, 17, 333–351. [Google Scholar] [CrossRef]
  59. Yang, X.; Charlebois, P.; Macalalad, A.; Henn, M.R.; Zody, M.C. V-Phaser 2: Variant inference for viral populations. BMC Genom. 2013, 14, 674. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  60. Zagordi, O.; Bhattacharya, A.; Eriksson, N.; Beerenwinkel, N. ShoRAH: Estimating the genetic diversity of a mixed sample from next-generation sequencing data. BMC Bioinform. 2011, 12, 119. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  61. Verbist, B.M.P.; Thys, K.; Reumers, J.; Wetzels, Y.; Van Der Borght, K.; Talloen, W.; Aerssens, J.; Clement, L.; Thas, O. VirVarSeq: A low-frequency virus variant detection pipeline for Illumina sequencing using adaptive base-calling accuracy filtering. Bioinformatics 2015, 31, 94–101. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  62. Cassarino, T.G.; Frampton, D.; Sugar, R.; Charles, E.; Zisis Kozlakidid, P.K. High-throughput pipeline for de-novo assembly and drug resistance mutations identifcation from next-generation sequencing viral data of residual diagnostic samples. bioRxiv 2016. [Google Scholar] [CrossRef]
  63. Huber, M.; Metzner, K.J.; Geissberger, F.D.; Shah, C.; Leemann, C.; Klimkait, T.; Böni, J.; Trkola, A.; Zagordi, O. MinVar: A rapid and versatile tool for HIV-1 drug resistance genotyping by deep sequencing. J. Virol. Methods 2017, 240, 7–13. [Google Scholar] [CrossRef] [Green Version]
  64. Lee, E.R.; Parkin, N.; Jennings, C.; Brumme, C.J.; Enns, E.; Casadellà, M.; Howison, M.; Coetzer, M.; Avila-Rios, S.; Capina, R.; et al. Performance comparison of next generation sequencing analysis pipelines for HIV-1 drug resistance testing. Sci. Rep. 2020, 10, 1634. [Google Scholar] [CrossRef] [Green Version]
Viruses 12 00550 g001 550
Figure 1. External quality assessment workflow for clinical tests (Adapted from https://euformatics.com/validation-for-ngs-based-clinical-tests/).
Figure 1. External quality assessment workflow for clinical tests (Adapted from https://euformatics.com/validation-for-ngs-based-clinical-tests/).
Viruses 12 00550 g001
Viruses 12 00550 g002 550
Figure 2. External quality assessment (EQA)/PT for NGS-based HIVDR testing: data collection considerations. (UMI: unique molecular identifier; AAVF: amino acid variant file; AAV: amino acid variant).
Figure 2. External quality assessment (EQA)/PT for NGS-based HIVDR testing: data collection considerations. (UMI: unique molecular identifier; AAVF: amino acid variant file; AAV: amino acid variant).
Viruses 12 00550 g002
Viruses 12 00550 g003 550
Figure 3. Variation of HIV drug resistance mutations (DRM) frequencies derived from NGS-based HIVDR assays. Six labs received two Virology Quality Assurance (VQA) panels (10 specimens in total) and processed the samples using their own laboratory developed tests (LDT) for NGS-based HIVDR typing. The NGS data (FASTQ files) derived from each of the six labs were analyzed using the HyDRA pipeline [32]. This scatter plot shows the median and interquartile range for HIV DRM frequencies between 1%–100% from each of the six labs. For certain DRMs, some labs did not detect the presence of the DRM and were excluded from the analysis.
Figure 3. Variation of HIV drug resistance mutations (DRM) frequencies derived from NGS-based HIVDR assays. Six labs received two Virology Quality Assurance (VQA) panels (10 specimens in total) and processed the samples using their own laboratory developed tests (LDT) for NGS-based HIVDR typing. The NGS data (FASTQ files) derived from each of the six labs were analyzed using the HyDRA pipeline [32]. This scatter plot shows the median and interquartile range for HIV DRM frequencies between 1%–100% from each of the six labs. For certain DRMs, some labs did not detect the presence of the DRM and were excluded from the analysis.
Viruses 12 00550 g003
Table
Table 1. Comparison of Sanger and next-generation sequencing (NGS)-based HIV drug resistance (HIVDR) assays.
Table 1. Comparison of Sanger and next-generation sequencing (NGS)-based HIV drug resistance (HIVDR) assays.
ItemSanger SequencingNGS
ExtractionRequiredRequired
RT-PCRRequiredRequired
PCRRequiredRequired
Specific sequencing primersMultiple specific primers Not required
Library preparationNot requiredRequired
Sequencing reactionSingle reaction Massive parallel clonal sequencing
Data outputOne sequence per sampleThousands of sequences per sample
DRM frequency detection threshold~20%~1%
Qualitative DRM detectionEnabledEnabled
Quantitative DRM detectionNot applicableEnabled
Table
Table 2. Sample suggestions for proficiency testing (PT) panels for NGS-based HIVDR testing.
Table 2. Sample suggestions for proficiency testing (PT) panels for NGS-based HIVDR testing.
Sample TypeAdvantagesDisadvantages
Donor Specimens (plasma, serum, DBS)Real specimensUnpredictable DRMs
Quasispecies population*Unknown DRM frequency
Limited supply
Complicated and expensive to acquire
Clinical Viral IsolatesQuasispecies populationUnpredictable DRMs
Known DRMsUnknown DRM frequency
Unlimited amountViral culture required
ReusableExpensive and complicated to prepare
Minor DRMs may arise during viral culture
Infectious Molecular ClonesCulture of clone-derived isolatesHomogenous population with defined DRMs
Clone mixtures can be producedViral culture required
Abundant SupplyMinor DRMs may arise during viral culture
Known DRMs
Any DRMs in any genes
Any DRM frequency
Plasmids, Plasmid Mixtures, Synthetic RNAKnown sequencesHomogenous population
Known DRMsPlasmids are DNA-based and are not suitable
Any DRMs in any genesfor RNA related protocol validations
Any DRM frequencyPlasmids underestimate PCR bias
Ideal for low-frequency DRMs
Ideal for NGS standard
Ideal for monitoring systematic error
Economical
Unlimited amount
Stable for storage and transportation
* Quasispecies refers to a swarm of highly related but genetically different viral variants that arise in a host during replication [51].

Share and Cite

MDPI and ACS Style

Lee, E.R.; Gao, F.; Sandstrom, P.; Ji, H. External Quality Assessment for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Unique Requirements and Challenges. Viruses 2020, 12, 550. https://doi.org/10.3390/v12050550

AMA Style

Lee ER, Gao F, Sandstrom P, Ji H. External Quality Assessment for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Unique Requirements and Challenges. Viruses. 2020; 12(5):550. https://doi.org/10.3390/v12050550

Chicago/Turabian Style

Lee, Emma R., Feng Gao, Paul Sandstrom, and Hezhao Ji. 2020. "External Quality Assessment for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Unique Requirements and Challenges" Viruses 12, no. 5: 550. https://doi.org/10.3390/v12050550

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop