External Quality Assessment Program for Next-Generation Sequencing-Based HIV Drug Resistance Testing: Logistical Considerations

Next-generation sequencing (NGS) is likely to become the new standard method for HIV drug resistance (HIVDR) genotyping. Despite the significant advances in the development of wet-lab protocols and bioinformatic data processing pipelines, one often-missing critical component of an NGS HIVDR assay for clinical use is external quality assessment (EQA). EQA is essential for ensuring assay consistency and laboratory competency in performing routine biomedical assays, and the rollout of NGS HIVDR tests in clinical practice will require an EQA. In September 2019, the 2nd International Symposium on NGS HIVDR was held in Winnipeg, Canada. It convened a multidisciplinary panel of experts, including research scientists, clinicians, bioinformaticians, laboratory biologists, biostatisticians, and EQA experts. A themed discussion was conducted on EQA strategies towards such assays during the symposium. This article describes the logistical challenges identified and summarizes the opinions and recommendations derived from these discussions, which may inform the development of an inaugural EQA program for NGS HIVDR in the near future.


Introduction
External quality assessment (EQA), also referred to as an external quality assessment scheme (EQAS), plays a vital role in assuring that a laboratory performs biomedical assays competently [1]. EQA is defined as a system for objectively verifying performance using an external agency or facility [2], relying on interlaboratory or inter-site comparisons. It facilitates the identification of areas that need improvement, the determination of potential training needs, and the evaluation and monitoring of training impact. An EQA is often administered by a third-party agency for ensuring the consistency of a laboratory in performing specific assays of interest or by regulatory agencies for accreditation purposes. For instance, the ISO 15189-accredited medical laboratories and the Clinical Laboratory Improvement HIVDR test enables quantitative, sensitive detection of low abundance nucleotide and amino acid variations. Moreover, it also allows simultaneous analyses of multiple specimens with unprecedented high data throughput in multiplexed runs, rendering improved time-efficiency and cost-effectiveness when conducting batched sample testing [7,[18][19][20]. With the increasing affordability of equipment and consumables, NGS HIVDR assays are being adopted by more laboratories worldwide and may soon become the new standard for HIVDR genotyping.
While it is well-appreciated that the generalized adoption of NGS HIVDR assays requires fully validated, standard operating procedures (SOPs) for sample processing and a sophisticated bioinformatics pipeline for effective data analysis, one essential but still missing aspect of ensuring assay consistency and reliability is an EQA program that functions ( Figure 1). NGS HIVDR assays are multiprocedural and involve many potential "check-points", where artificial biases or significant variations may arise and subsequently compromise the accuracy and reliability of the final output. Therefore, an EQA is at least as critical for laboratories performing NGS HIVDR assays, as it is for other clinical laboratory assays. While EQA programs for SS HIVDR have been widely applied, innovative EQA strategies have yet to be established for NGS HIVDR due to the fundamental differences between SS-and NGS-based assays and the data they generate [21][22][23]. For instance, conventional SS assays generate a single sequence per specimen, and DRMs are qualitatively detected and reported as being present (sometimes in mixtures) or absent. The EQA strategies for such tests are based on similarity analysis of the sequences and the concordance of DRM detection from individual laboratories against the consensus from the combined group [13,24] (Table 1). In contrast, NGS HIVDR assays differ from SS in many ways; EQA strategies developed for SS assays may not be applicable for NGS HIVDR assays [21]. With such challenges being recognized, themed discussions were carried out on EQA strategies for NGS HIVDR during the 2 nd International Symposium on NGS HIVDR held in Winnipeg, Canada, in September 2019. This article summarizes the proceedings from the discussions specifically on the logistical challenges and considerations for establishing an EQA program for NGS HIVDR assays.

EQA Strategies for NGS HIVDR Assays: Logistic Challenges and Considerations
Recognizably, EQA for NGS HIVDR testing is a new field for which limited knowledge and experience are available currently, and extensive research and development efforts are still required. An operational EQAP that executes such EQA functionalities has yet to be established. Most of the research efforts in this regard have been devoted to the development of effective data assessment and scoring criteria for the evaluation of laboratory competence in performing such assays [21,22,24]. However, the establishment of such an EQAP requires a comprehensive effort of administerial management, financial operation, PT support, data management, and subsequent reporting and follow-up actions.
Like EQAPs for other biomedical assays, the operation of an EQAP for NGS HIVDR testing may be divided into six main task areas, including (1) organization and administration, (2) laboratory recruitment, (3) reference material preparation and distribution, (4) data collection, (5) data assessment, and (6) EQA reporting ( Figure 1). Accordingly, Table 1 summarizes the major logistical challenges one may encounter within each of these areas and some general issues applicable for any operational program (listed as "other challenges"), the successful experiences from EQA for SS HIVDR testing in addressing such challenges (taking the NIAID VQA program as an example), and the suggested considerations and recommendations for the establishment and operation of an EQAP for laboratories conducting NGS HIVDR assays. Based on the experiences from a pilot study that evaluated the potential of using existing VQA PT specimens for NGS HIVDR EQA [23,29], and comparing the performance of different bioinformatics pipelines [22], some strategies that may facilitate a smooth transition from a SS-to a NGS-based HIVDR testing era are also advised ( Table  1). It is noteworthy that, while the NIAID VQA program is taken as an exemplar EQAP for SS-based HIVDR assays in Table 1, most of the NGS HIVDR considerations and recommendations based on the VQA experience should be applicable or adaptable for other alike EQAPs such as TAQAS, QCMD, JEQS, or existing similar programs.

Conclusions
As an exemplar "disruptive" technology, NGS can revolutionize the conventional SS-based HIVDR genotyping practice and can enable sensitive and quantitative MRV detection. Many commercial and in-house-developed NGS HIVDR assays have been developed together with  VQA fits well in undertaking this task; however, extra funding support may be required, and the number of participating laboratories may be limited for operational reasons. When possible, joint efforts between VQA and regional/national/global quality assurance programs or agencies are recommended for managerial and financial considerations. Collaborative data assessment between VQA and partner(s) with relevant NGS bioinformatics expertise would be beneficial.

Who participates?
NIH-funded network laboratories and programs. WHO-designated HIVDR laboratories. Other laboratories approved by NIAID VQA contracting officer.
Laboratories from the NIAID clinical trial networks and with appropriate NGS capacity would potentially be early adopters for NGS HIVDR technologies.
HIVDR laboratories from the current VQA program, WHO HIVResNet, and PHAC/PAHO collaborative network showed interest in participating in a NGS EQA. Gradual expansion is foreseeable while NGS HIVDR is adopted more broadly worldwide.

Who funds?
NIAID VQA contract supports PT panel distribution, data collection, and assessment for EQA purposes. Costs related to specimen processing and data submission are self-funded by the client laboratories.
Adding a NGS HIVDR component into existing HIVDR EQA programs would be preferable. However, extra funding support may be required for VQA or other existing EQA programs to cover NGS HIVDR testing. Commercial non-network laboratories could be self-funded, for cost recovery purposes. Funding from collaborating regional/national quality assurance programs or agencies could be sought when possible.

Basic infrastructure requirements
Availability of laboratory facility and equipment required for SS-based HIVDR genotyping.
Availability of NGS sequencing equipment and accessories required for HIVDR genotyping. Availability of instruments for RNA/DNA quality assessment and quantification, which are required for NGS wet laboratory procedures, test quality control, and troubleshooting uses. An on-site evaluation might be required for official accrediting applications.

Sample processing capacity requirement
Commercial or in-house SS-based HIVDR assay(s) in place. Experienced staff for SS HIVDR sample processing and data management.
Commercial or in-house NGS-based HIVDR assay in place. Experienced staff is available for HIV sample processing and NGS sequencing.

Bioinformatics capacity requirement
Availability of software and expertise required for SS sequence data processing, HIVDR interpretation, and reporting.
Availability of expertise and steady access to bioinformatics pipeline(s) for NGS data processing, HIVDR interpretation, and reporting. If using a pipeline validated for HIVDR applications, specialized bioinformatics support may not be required. Step 1: Assessment Panels (APs): The existing VQA or similar panels may serve this need with no specific modification or extensive characterization required. The subsequent data assessment could be based on NGS consensus sequences, and the current SS-based EQA strategies would apply.
Step 2: Validation Panels (VPs): Well-characterized, comprehensive wet panels with ground truth on HIV DRMs and their exact frequencies are required. Such panels may serve the needs for both EQA and NGS HIVDR assay validation in individual laboratories.

Panel specimen types
Plasma/serum or dried blood spot specimens consisting of -Donor specimens. -Clinical isolates. -Viruses generated from infectious molecular clones.
Plasma/serum or dried blood spot specimens consisting of: -All those listed for SS methods. -Pedigreed plasmids and plasmid mixtures that help for the assessment of the gross error rate and the MRV detection sensitivity of the assay.
* VP specimens should be characterized for the DRMs they contain and their exact frequencies.
* Initial VP panels may focus on plasma specimens at viral loads of ≥1,000 copies/mL. For logistical and practical reasons, assessing different assay capacities or potentials in accommodating diverse specimens with alternate panels is advised to avoid single large panels.

Panel distribution
Biannual distribution for NIAID supported laboratories; Biannual or annual distribution for WHO HIVResNet laboratories of different categories.
More frequent panel distributions rather than larger panels may be beneficial for timely identification of issues and remedial actions A biannual distribution could be a good start considering the high NGS costs to the laboratories.
HIV gene targets PR, RT, and IN Same as SS Panel size Data files of a different quality from~100 specimens.
The size of such dry panels could be flexible. Any data that highlight potential quality assurance (QA) issues and contributes to NGS HIVDR data processing pipeline validation and refinement could be incorporated. Proper categorization (based on QA issues) and annotation of the files would be required.

Data access
Restricted to HIVResNet designated and candidate laboratories (to date) Open access through the public domain is recommended.

Potential application
Such panels involve no sample processing in the laboratory and may serve the needs for:

HIV DRM / Variation reports
Qualitative reports on HIV DRMs present in the specimens.
HIVDR interpretation and reporting based on consensus at a defined threshold using Stanford HIVdb, REGA, ANRS, or other established algorithms for SS-like data assessment [26]. Qualitative HIVDR interpretation and reporting for all DRMs when their frequencies are >=5% or >=15% [27]. A comprehensive AAVF report covering all detected amino acid variations (DRMs or non-DRMs) and frequencies is recommended for cross-pipeline and inter-laboratory comparisons [27].

Raw sequencing data
Except for data from the ViroSeq assay, original raw SS files are not collected.
The collection of anonymized raw NGS data (in Fastq format) is encouraged for potential data validation, troubleshooting, or cross-pipeline comparison purposes.

Information on the protocols applied
Categorical protocol information only (in-house, ViroSeq, or TruGene), with minimal laboratory protocol details collected. Data analysis (base-calling) and reporting software.
Collection of laboratory protocol and data analysis pipeline information is highly recommended for potential data validation, troubleshooting, or cross-pipeline comparison purposes. A standardized documentation template with all required protocol items (e.g., HIV RNA/DNA extraction, PCR amplification procedures, NGS library preparation kits, NGS sequencing kits, NGS platform, data analysis pipeline, and HIVDR interpretation algorithms) should be applied. Differences in sample processing and data analysis protocols should be considered for EQA.

Data collection approach
VQA data submission portal Data submission portal for small-size files, i.e., consensus sequences, AAVF and DR reports. Cloud sharing of coded raw NGS data files of larger sizes (i.e., Fastq) would be beneficial.
Well-defined EQA data assessment guidelines remain to be established [23]; EQA assessment based on NGS consensus sequences may oversimplify the complexity of NGS HIVDR data, which detect HIV DRMs both qualitatively and quantitatively.

EQA Tasks Logistical Issues Sanger Experiences (VQA as an Example) NGS HIVDR EQA Considerations and Recommendations
Data assessment parameters [24] Concordance with consensus from the group (mismatch counts in the examined HIV genomic regions for each sample and target gene regions).
Error counts in identifying amino acid changes at DRM codons. Scoring based on established criteria [8,24].
A two-part assessment is recommended while transitioning from SS to NGS HIVDR testing: Part 1: Simplified data assessment using NGS consensus at a threshold of 20% following the current VQA strategies [24]. Part 2: In-depth, NGS-specific EQA data assessment [21,23] AAVF files containing all amino acid variations, instead of DRMs only, at a wide range of frequencies, may be more informative for cross-laboratory comparisons. Focus more on the ability to detect DRMs over the desired threshold (e.g., 5%) for scoring, while the accuracy of the DRM frequency readouts should also be assessed.

Scoring strategies
Proficiency scores are based on the number of disagreements from the consensus sequence. Performance is assessed by assigning a p-value to the observed number of disagreements in a data set for each sample and gene region. Performance Scores and certifying criteria are well established [24].
Traditional EQA parameters (sensitivity, specificity, linear range, etc.) may not be directly applicable for NGS HIVDR assays due to the uncertain thresholds for MRV detection. New or redefined parameters are required and meaningful reference, target values, or acceptable ranges for such parameters need to be better defined for NGS HIVDR data assessment and scoring [22,23]. Proper strategies for identifying "outlier" laboratories and scoring the inconsistencies in DRM detection and frequency readouts among the laboratories have yet to be better defined. [21,23] Weighting strategies based on the two-part assessment process should be developed for the final laboratory scoring. A two-part assessment report is recommended: Part 1: A simplified assessment report based on NGS consensus at a threshold of 20% following existing SS strategies. Part 2: In-depth data assessment based on AAVF files and DRM reports collected from participating laboratories against the newly-defined, NGS-specific assessment and scoring criteria: -Spreadsheets, statistical analysis, and graphs showing the performance of the laboratory as compared to its peers. - When required, certification recommendations should be provided. -Issues identified and potential corrective or remedial action recommendations should be provided when possible.

Assessment, data distribution and retention/archival
The assessment reports are emailed to the focal contacts of each laboratory. VQA retains the assessment documents for the life of the program contract.
The assessment data should be reported back to the focal contacts of the laboratories via email or the data submission portal. The assessment documents may be retained for the life of the EQA program for subsequent data re-evaluation or validation purposes. The original NGS data collected from the laboratories may be retained a determined amount of time (e.g., up to 6 months) for cross-checking and re-examination purposes. Data retention policies from a specific bioinformatics pipeline may also apply. Guidelines for responsible data sharing among relevant stakeholders should be established.

EQA Strategies for NGS HIVDR Assays: Logistic Challenges and Considerations
Recognizably, EQA for NGS HIVDR testing is a new field for which limited knowledge and experience are available currently, and extensive research and development efforts are still required. An operational EQAP that executes such EQA functionalities has yet to be established. Most of the research efforts in this regard have been devoted to the development of effective data assessment and scoring criteria for the evaluation of laboratory competence in performing such assays [21,22,24]. However, the establishment of such an EQAP requires a comprehensive effort of administerial management, financial operation, PT support, data management, and subsequent reporting and follow-up actions.
Like EQAPs for other biomedical assays, the operation of an EQAP for NGS HIVDR testing may be divided into six main task areas, including (1) organization and administration, (2) laboratory recruitment, (3) reference material preparation and distribution, (4) data collection, (5) data assessment, and (6) EQA reporting ( Figure 1). Accordingly, Table 1 summarizes the major logistical challenges one may encounter within each of these areas and some general issues applicable for any operational program (listed as "other challenges"), the successful experiences from EQA for SS HIVDR testing in addressing such challenges (taking the NIAID VQA program as an example), and the suggested considerations and recommendations for the establishment and operation of an EQAP for laboratories conducting NGS HIVDR assays. Based on the experiences from a pilot study that evaluated the potential of using existing VQA PT specimens for NGS HIVDR EQA [23,29], and comparing the performance of different bioinformatics pipelines [22], some strategies that may facilitate a smooth transition from a SS-to a NGS-based HIVDR testing era are also advised ( Table 1).
It is noteworthy that, while the NIAID VQA program is taken as an exemplar EQAP for SS-based HIVDR assays in Table 1, most of the NGS HIVDR considerations and recommendations based on the VQA experience should be applicable or adaptable for other alike EQAPs such as TAQAS, QCMD, JEQS, or existing similar programs.

Conclusions
As an exemplar "disruptive" technology, NGS can revolutionize the conventional SS-based HIVDR genotyping practice and can enable sensitive and quantitative MRV detection. Many commercial and in-house-developed NGS HIVDR assays have been developed together with sophisticated bioinformatics pipelines. Meanwhile, the gradual cost reductions for both NGS instruments and related consumables have converted NGS from a high-end research tool into an affordable and accessible technology for general HIVDR laboratories. NGS may soon become the new standard for HIVDR testing in research and surveillance, as well as clinical monitoring purposes. Therefore, appropriate EQAPs will become imperative for ensuring the quality of data from the laboratories performing such assays. Due to the uniqueness of NGS HIVDR assays and the complexity of data derived from such tests, the existing EQA strategies and EQAPs targeting SS-based HIVDR genotyping are not optimal for these new assays. Technical and logistical challenges involved in the development and implementation of NGS-specific EQAPs remain to be resolved and require additional research. design, data collection and analyses, manuscript preparation, and the decision-making for the publication of this work.

Conflicts of Interest:
The authors declare no conflict of Interest.