Novel Approach to Proficiency Testing Highlights Key Practice Variations in Cancer Biomarker Delivery

Abstract

Biomarkers are fundamental to modern oncology practice, forming a close link to pathology practice. Pathology results must be accurate, timely, comprehensive, and comprehendible. External proficiency testing is a key tool in maintaining biomarker quality. Here, we demonstrate the feasibility and utility of a novel end-to-end proficiency testing exercise exploring accuracy, turnaround time, and communication. Challenge specimens were made using resected colon cancer tissue, each paired with a fictional clinical vignette, and distributed to participants who were asked to provide all molecular testing required and return a final report for each case upon completion. Reports were redistributed to an assessor team including medical oncologists, each of whom was asked to recommend a systemic therapy based on each lab’s biomarker report. Participants were graded based on their ability to guide oncologists to the correct treatment. Eight laboratories participated. Three laboratories were found to have suboptimal results, two leading oncologists to incorrect therapeutic prescriptions, and one withdrawn. Turnaround time ranged from 6 to 86 days (median 24). Substantial qualitative reporting differences were identified. This study demonstrates the feasibility of end-to-end proficiency testing. The approach provides considerable value beyond analytic accuracy, including specimen management, turnaround time, and communication of results. Results suggest that reporting differences may lead to treatment disparities. This style of quality assurance will help reinforce good practices critical to the delivery of precision cancer care.

1. Introduction

Laboratory-based biomarker testing is a mainstay of precision cancer care. In many disease sites, treatment decisions are heavily predicated on biomarker data [1,2,3]. The complexity of cancer biomarkers has increased dramatically and includes modalities such as comprehensive next-generation sequencing and immunohistochemistry, among others [4,5].

The accuracy of biomarker data is critical, and systematic errors can be devastating. To this end, external quality assurance (EQA) programs have emerged as a critical tool for laboratories to maintain their quality and accuracy. Many EQA programs are currently offering biomarker proficiency testing both internationally and at local levels [6,7]. Typically, programs will provide ‘unknown’ samples, which will be tested at local laboratories, with participants’ results compared to a reference standard [5,6,7]. The ‘unknown’ samples typically consist of DNA or RNA when testing modalities such as next-generation sequencing, and these unknowns may contain several key alterations mixed together within the same sample. Traditional EQA testing in immunohistochemistry might include unstained slides with several sections or tissue microarrays where participants would stain and score each tissue element locally, comparing their results to reference results [8].

While all EQA activities vary in their design to some extent, to date, the focus of EQA for laboratory biomarker testing has been on the analytic accuracy of the methods used. While accuracy remains critical, there are additional factors that may further impair the translation of biomarker results into optimal therapy prescriptions, thereby curbing the efficacy of laboratory biomarker delivery. Turnaround time is often cited as a major barrier to precision cancer care [9]. The readability of molecular reports also remains another major barrier to the routine delivery of precision medicine [10]. Furthermore, treatment for many disease sites requires multimodal biomarker data [11]. In this report, advanced colorectal cancer is explored due to its treatment requiring genetic information such as KRAS, NRAS, and BRAF status (typically assayed via next-generation sequencing (NGS)) as well as mismatch repair protein status (typically assayed via immunohistochemistry (IHC)) or alternatively microsatellite stability status (typically assayed via polymerase chain reaction (PCR)), with additional biomarkers such as HER2 or NTRK, which may be tested utilizing both IHC and gene sequencing technologies [12,13,14]. The multimodal nature of colon cancer biomarkers can make it difficult for oncologists to ascertain all of these data at the time of treatment decisions. Given these challenges and the evolving landscape of precision oncology, a novel approach to cancer-related EQA was developed. The design and results of the inaugural run are described in this report.

2. Materials and Methods

Formalin-fixed paraffin-embedded (FFPE) blocks of colorectal cancer (CRC) from the archives of the Canadian Pathology Quality Assurance (CPQA) with known mismatch repair (MMR) results were utilized to construct “unknown” samples. FFPE scrolls for each of the three cases were sent to an external College of American Pathologists (CAP)-accredited and Clinical Laboratory Improvement Amendments (CLIA)-certified reference laboratory for next-generation sequencing to confirm genotype.

Each unknown consisted of an FFPE block with duplicate (redundant) 2 mm punches of colonic adenocarcinoma. Each case was matched with a clinical vignette describing an advanced or metastatic colon cancer patient requiring biomarker testing to guide further management decisions. The participating laboratories were sent three cases, each consisting of a corresponding FFPE block and clinical vignette. Participants were provided information via an optional pre-exercise meeting, as well as written instructions in advance via email and again at the time of case delivery.

The participants were Canadian hospital laboratories from tertiary or quaternary centers, providing routine molecular diagnostic services to support the care of advanced colorectal cancer. All laboratories were providing biomarker services and were in good standing with local regulatory bodies and traditional EQA programs.

Participants were asked to accession the material as if it were being received from an outside hospital and then to follow local protocol to deliver a complete set of biomarkers. Laboratories would need to perform any morphologic analysis, microtomy, immunohistochemistry, nucleic acid extraction, sequencing or gene testing, variant analysis, and interpretation, among others, that they deemed necessary. Laboratories were asked to digitally upload their final reports to the CPQA website upon completion.

Participants were made aware that turnaround time was being measured. Turnaround times were calculated based on the commercial shipment-tracked delivery date, as well as the date samples were accessioned based on the report until the date the final reports were submitted to CPQA via online submission. The exercise was considered complete when a lab uploaded all three final reports.

An assessment committee including 3 medical oncologists and 3 molecular pathologists with subspecialty interest in colorectal cancer was convened. The committee approved the clinical vignettes as well as an evaluation rubric for each of the participating laboratories. A gold standard treatment for each case was decided by the committee based on reference results and clinical vignettes.

Following completion of the exercise, final reports were de-identified and redistributed to the evaluation committee. Assessors were asked to complete a survey for each biomarker report where they could provide qualitative information about the reports through either drop-down Yes/No or open-text responses. The clarity of reports was evaluated based on the ease of extracting the necessary information from the reports needed for guiding patient management, as well as the relevancy and accuracy of the provided annotations on the results. Annotations on result interpretations were evaluated following the Canadian Consensus Guidelines for CRC biomarker testing [12]. In addition, the medical oncologist assessors were asked to prescribe a systemic therapy to the vignette patient predicated on each participating laboratory’s biomarker report.

Assessor reports were collected by the CPQA, and summary statistics were calculated for quantitative variables. Additional qualitative analysis on reporting readability was performed through the identification of significant phrases with similar meanings extracted from assessor open-text responses and categorized into common themes. The concordance of the resulting prescribed systemic therapies was evaluated for each case based on consensus between the three medical oncologists. The concordance of the report clarity results was determined based on consensus between at least five of the six assessors.

A final assessment summary report was issued to all participating laboratories, including individualized performance letters for each participant. The methodology of the exercise is summarized in Figure 1.

3. Results

Out of the eight laboratories that enrolled, seven laboratories (88%) completed the challenge, and one withdrew. All participants were Canadian hospital laboratories, currently providing routine molecular diagnostic services to support the care of advanced colorectal cancer, and in good standing with local regulatory bodies and traditional EQA programs. The three challenge cases, including clinical vignettes, reference results, and consensus ideal treatments, are summarized in

Table 1. Challenge specimen feature summary denoting clinical vignette, MMR status and reference genotype, and gold standard treatment recommendation.

Sample	Clinical Vignette	MMR Status	Genotype	Gold Standard Treatment
1	The patient is a 52-year-old, previously healthy woman who has presented to the emergency room with a perforated colon adenocarcinoma and evidence of distant metastases. She has been treated with a right hemicolectomy and is now being assessed by a medical oncologist for further management. This case is being referred for biomarker testing.	Loss of MLH1/PMS2	KRAS G13D (c.38G>A, p.Gly13Asp)	Immune Checkpoint Inhibitor
2	The patient is a 42-year-old female with a history of metastatic colon adenocarcinoma and a lifetime history of ulcerative colitis, sclerosing cholangitis, and uveitis who is currently on biologic therapy. Her treatments to date include a total proctocolectomy and a pulmonary metastectomy, followed by systemic therapy with FOLFOX + bevacizumab and then second line FOLFIRI. No previous molecular testing has been performed. An archival block from her pulmonary metastasis has been referred for molecular testing to help inform the next course of therapy.	Loss of MLH1/PMS2	RAS/RAF Wildtype	Anti-EGFR mAb
3	A 66-year-old male with advanced colorectal carcinoma and peritoneal carcinomatosis. He has received 6 cycles of FOLFIRI + panitumumab; however, his oncologist notes worsening of his ascites, and a CT scan shows definitive disease progression on therapy. The tumor sample was referred for molecular testing to inform his next line of therapy.	MMR Intact	BRAF V600E (c.1799T>A, p.Val600Glu)	BRAF Inhibitor + Anti-EGFR mAb

MMR—mismatch repair; mAb—monoclonal antibody.

.

Turnaround time measured from the accession date to the finalized report ranged between 6 and 84 (median 20) days. The time between the courier receipt date and accession date ranged from 0 to 15 (median 2) days. The total turnaround time from sample receipt to finalized report ranged from 6 to 86 (median 24) days. Laboratories were also asked for their self-reported typical turnaround time for biomarker requests on newly diagnosed metastatic CRC patients; this ranged from 3 to 21 (median 13) days. Turnaround time statistics are summarized in Figure 2.

Out of seven participating labs, three labs (43%) produced reports that guided expert medical oncologists to the correct treatment decision in all three cases. One lab failed to provide appropriate genotyping data on case 2, leading assessors to an incorrect treatment decision. One lab failed to incorporate MMR/MSI data into their report, leading to suboptimal therapy in case 1. Four of seven labs (57%) had incomplete or sub-optimally communicated report annotations, leading to heterogeneity in therapeutic decision-making for case number 3. Reviewer prescription concordance, overall report clarity, and turnaround time are summarized in Figure 3. The NGS assay and platform methodology used by each lab are summarized in

Table 2. Laboratories’ NGS assay and platform methodology used in the assessment and whether the IHC and NGS results were integrated into the same report.

Lab ID	NGS Platform	IHC and NGS Results Integrated into The Same Report?
1	Not available	N/A
2	TruSight Tumor 15 Panel—Illumina	Yes
3	Oncomine Comprehensive Assay v3—ThermoFisher	No
4	Oncomine Precision Assay GX—ThermoFisher	No *
5	Oncomine Comprehensive Assay v3—ThermoFisher	Yes
6	Oncomine Comprehensive Assay v3—ThermoFisher	No
7	Oncomine Comprehensive Assay v3—ThermoFisher	No
8	Oncomine Precision Assay GX—ThermoFisher	Yes

* No IHC results submitted.

.

Four major themes regarding reporting clarity and readability were extracted from assessor responses:

I—Delays (long turnaround times detract from report efficacy). For example: “the turnaround time was so long, unfortunately, that it did not really matter what the results were. They came far too late to be useful”—Assessor 4, case 3.

II—Interpretation (difficulty interpreting the reports). For example: “inclusion of a statement that RAS wildtype status before the BRAF narrative in the interpretation could have led to erroneous anti-EGFR single-agent use by some clinicians without BRAF-directed therapy”—Assessor 2, case 3.

III—Organization (inability to locate relevant results). For example: “the reports were found to be very challenging to interpret. MMR IHC was found in a separate report from the genetic findings, and these were separated in time by over 1 month. In addition, the results of the MMR were not clear and easy to find, as they were buried below hypothetical interpretations”—Assessor 1, case 1.

IV—Terminology (preference for colloquial variant reporting over formal HGVS nomenclature). For example: “using the terminology for a subset of KRAS mutations familiar to clinicians, i.e., KRAS G13D, G12C, etc., is very much appreciated so we do not have to look them up”—Assessor 3, case 1.

4. Discussion

This report highlights a first-of-its-kind proficiency testing program. While traditional programs of this nature aid in maintaining the analytic accuracy of biomarker tests, this exercise is designed to capture many of the additional pre- and post-analytic issues that can be most challenging to oncologists when treating their patients.

4.1. Turnaround Time

Turnaround time was a major focus of this exercise and is often cited as one of the major barriers to precision cancer care [15]. In this study, only a single laboratory met local guideline recommendations with a turnaround time of less than 14 days. Three laboratories (38%) failed to provide a biomarker report within 2 months of receiving the test samples. Many assessors noted that the actual results provided were meaningless, given the timeframe.

There are many contributing factors to biomarker turnaround time. One in this exercise is the time between receiving a sample at an institution and the time at which it is accessioned within the laboratory. In one instance, this was shown to take 15 days. This exercise provides a sobering reminder to laboratory practitioners that the process of biomarker testing includes the receiving docks, mail room and sorting, and clerical staff performing accessioning duties. For several labs in this exercise, shortening the time spent from the mail room to the laboratory would significantly improve biomarker turnaround time. To this end, turnaround time statistics reported as time “from receipt in the molecular laboratory” may be considered to have limited clinical relevancy.

While the lengthy time for clinicians and patients to receive biomarker results is often attributed to the complex nature of genetic testing, this study highlights that the actual turnaround time is not related to the exact NGS methodology. For instance, lab 8 demonstrated the fastest turnaround time (6 days), while lab 4 demonstrated the slowest turnaround time (86 days) despite utilizing the same panel and gene sequencer. The stark contrast in biomarker delivery between these two groups suggests that the panel and gene sequencer themselves play a minimal role in the determination of turnaround time.

Notably, labs 2, 5, and 8 demonstrated some of the fastest turnaround times in this exercise, with results in 24, 20, and 6 days, respectively, using different NGS assays and/or platforms. In addition, these three labs were noted to have integrated reports, as depicted in Table 2, combining IHC findings with NGS findings into a single biomarker document, and were ultimately the most successful in consistently steering oncologists to the correct treatments.

One limitation of this study with respect to turnaround time is that the post-analytic period was reduced by allowing participants to upload their final reports. In true clinical practice, this function is often served via facsimile or digital report transmission, both of which can lead to additional delays in report receipt by end-users.

4.2. Reporting Clarity

This exercise also focused on the reporting and presentation of biomarker data from different modalities. Assessors commented on a range of reporting styles, spanning ‘easy to understand’ to ‘difficult and misleading’. While there was no defined formatting or criteria for reports, the assessors were looking for overall clarity and ease in locating pertinent information within the report, in addition to the accuracy of the results and annotations to support treatment in adherence to current Canadian guidelines [12].

A variety of reporting styles was seen from the participants; however, many of the reports were deemed acceptable in terms of clarity and readability. Note that recommendations outlining the minimal report content requirements have been previously established in the literature and by accreditation bodies [16,17]. This was not explicitly assessed in this exercise; instead, a focus was placed on the information pertinent to an oncologist for guiding therapeutic management.

One recurring reporting critique commented on by assessors was the incompleteness of interpretations and annotations. Due to the advanced/metastatic nature of the patients depicted in the challenge vignette, an emphasis should have been placed on the immediate therapeutic implications of the results rather than the hereditary or prognostic implications. In some reports, the hereditary implications of the result (particularly MMR deficiency) were at the forefront and were considerably lengthy, ultimately detracting from the immediate clinical implications needed for patient therapeutic management (sensitivity to immune checkpoint inhibitor therapy).

In addition, as the field progresses at a rapid pace, variant annotations and interpretations should be periodically updated according to current guidelines in order to provide the most relevant information to oncologists. The importance of this is seen in case 3, wherein the annotation for the BRAF V600E mutated colon cancer provided in some reports was not informed by the most up-to-date treatment recommendations for the use of combination BRAF inhibitor and EGFR monoclonal antibody therapy [12,18,19]. The results of this study suggest that varying reporting practices may be associated with resultant treatment discrepancies.

Complex and lengthy reports are known to detract from the actionability of the findings [10]. Overall, the report length ranged between two and six pages and averaged approximately four pages. For many reports, the genetics and IHC results were typically found separated, sometimes with a referral to other reports or addenda.

A recommendation to address the length of the reports made by the assessor panel was to combine pathology IHC and molecular genetics results prominently in a single summary section. This would allow the oncologists to have easy access to the biomarker results rather than having to read through multiple separate reports where important results may be overlooked.

From both the assessor committee and the literature, there is a clear desire for a concise and simple summary of the relevant findings, which can facilitate the readability of the reports for oncologists, with additional information (i.e., methodology, transcripts, gene lists, etc.) included in subsequent pages if deemed absolutely necessary [20,21].

Although the laboratory participants were able to correctly identify the specimen genotype(s), many of the oncologist assessors noted difficulty interpreting the results. For example, for the NM_004333.6(BRAF):c.1799T>A, p.(Val600Glu) mutation, the colloquial term “BRAF V600E” is strongly preferred by the oncologist assessors over any other form of reporting. This finding aligns with recent studies showing that while the standard HGVS three-letter amino acid notation is typically recommended, physicians are more familiar with the single-letter amino acid codes [10,16].

4.3. Systemic Therapy Concordance

This study suggests that otherwise identical patients could be receiving different treatments as a result of heterogeneity within biomarker practices.

Despite the overall analytical accuracy of test results submitted by participants, incorrect therapy prescriptions were observed in laboratory participants with exceedingly long turnaround times, incomplete reports (missing either IHC or genotyping data), or obscure reports. This finding implies that these pre- and post-analytic features of biomarker results are critical in the delivery of correct and optimal therapeutics to patients, thereby justifying their inclusion within EQA exercises.

5. Conclusions

This study showcases a novel approach to proficiency testing, capturing additional key elements of biomarker reporting such as turnaround time, specimen handling, communication, and oncologist interpretation. The data presented here indicate that factors beyond the analytic accuracy of biomarker tests can lead to suboptimal treatments for cancer patients. While additional studies are needed, there are several early conclusions that can be drawn from this inaugural exercise:

(1) Turnaround time is a critical metric for maintaining biomarker quality. Self-reported turnaround time is insufficient to monitor and enforce best practices. Protracted turnaround time should be regarded similarly to critical false-negative or false-positive results, as they will lead to similar mistreatment of patients. The underlying causes of delayed turnaround times are complex and multifactorial. One key component highlighted in this study is the length of time it takes from delivery to a hospital receiving dock to accessioning within a molecular laboratory. The data collected and presented here show that specific NGS methodologies or platforms are unlikely to significantly affect turnaround time;

(2) Integrating findings into a single report was identified as a key component in delivering the most optimal therapy to patients. This should be considered when laboratories are reporting complex biomarkers that span multiple modalities, such as IHC and NGS. Utilizing personnel equipped to interpret both modalities can offer considerable advantages to patient care;

(3) While the use of formal genetic nomenclature has advantages in the research setting, this can actually lead to significant patient harm as it is not well understood or easily read by oncologists. Based on the results of this study, the use of single-letter amino acid abbreviations, or ‘colloquial’ nomenclature, is preferred.

Overall, these results highlight significant gaps in biomarker delivery across a publicly funded healthcare system committed to providing equal access to all of its citizens. By addressing gaps in turnaround time, biomarker accuracy, and reporting clarity, optimal systemic treatments will become more accessible to all patients.

End-to-end proficiency testing represents a valuable tool for maintaining biomarker quality and delivering optimal treatment to cancer patients. More studies are needed to show the long-term effects of regular participation in this style of quality assurance exercise.