Evaluation of Navify Mutation Profiler Tertiary Analysis Software Assessing for Hematologic Malignancies ^†

Abstract

Background: Navify^® Mutation Profiler (Navify MP) is a cloud-based, tertiary analysis software that provides curation, annotation, and reporting of somatic genomic alterations and biomarker signatures identified by next-generation sequencing. The Navify MP software leverages Association for Molecular Pathology/American Society of Clinical Oncology/College of American Pathologists (AMP/ASCO/CAP) Somatic Variant Classification Guidelines to provide information on detected somatic genomic variants and associated therapies according to region-specific approvals. Methods: This validation study assessed the accuracy of the Navify MP software and curation process for hematologic malignancies as compared to expert opinion. A total of 86 variants derived from hematologic malignancies (including myeloid and lymphoid leukemias, B cell lymphomas, and multiple myeloma) were used to contrive 12 VCF files. The VCFs were made up of the following classes of genomic alterations: single nucleotide variants, small insertions and deletions, fusions, and copy number alterations. Of the 86 variants, 42 were Tier IA, and 44 were non-Tier IA, based on AMP/ASCO/CAP classification. The study was performed at four sites with seven software users (molecular genetics experts). Results: Tier classification agreement between Navify MP and expert user assignment was 91.34% for Tier IA and 95.02% across all hematologic variants. The agreement on associated therapies for the Navify MP-classified Tier IA hematologic variants was 99.08%. Conclusions: Navify MP is a robust automated solution for genomic variant reporting of hematologic malignancies and remains up to date with evolving regional approvals and medical guidelines.

1. Introduction

Precision oncology has become a key factor in cancer treatment decision-making, and sequencing technologies, such as next-generation sequencing (NGS), are at the forefront of the transition from research use to routine clinical practice. NGS is recommended by organizations such as the National Comprehensive Cancer Network (NCCN) and the European Society for Medical Oncology (ESMO) for certain cancer types for which numerous actionable targets are well characterized and molecular profiling directly informs treatment selection. Additionally, the World Health Organization (WHO) recommends testing hematologic malignancies for certain molecular biomarkers for which NGS is often instrumental, such as FLT3-ITD. A survey of oncology health-care professionals (HCPs) conducted in 2017 showed that three-quarters of HCPs used NGS in their practice to help guide treatment decisions [1]. A more recent evaluation of NGS uptake by HCPs showed that most patients in their review cohort would have benefited from earlier testing, which would have allowed earlier access to targeted treatments [2].

From 2000 to 2022, 573 agents were approved for various cancer indications, approximately half of which were targeted drugs [3]. For example, in leukemia alone, there are currently 36 approved targeted treatments [4]. As the numbers of biomarkers and targeted treatments have rapidly increased, so has the need for tools to help HCPs make time-dependent treatment decisions for their patients. Thus, oncology HCPs are under increased pressure to remain up to date with the increasing number of targeted treatments in already stretched health-care systems [5]. A lack of streamlined reporting and interpretation of NGS variants is a major barrier to treatment access and timely execution of treatment. Currently, most HCPs either operate from prior knowledge or need to spend time looking through many public databases and manually collating the information for each variant and associated targeted therapies. Manual curation is a key challenge necessitating continual updates in order to provide accurate somatic variant interpretation. Clinical decision support (CDS) tools represent a solution to this challenge, incorporating this vast amount of continually changing information into the clinical workflow to provide accurate and timely variant interpretations for HCPs.

One such CDS tool is Navify^® Mutation Profiler (Navify MP): a cloud-based tertiary analysis software that provides curation, annotation, and reporting of somatic genomic alterations and biomarker signatures identified by NGS. Navify MP is approved for in vitro diagnostic use in the European Union (EU) and for research use only (RUO) in the United States (US). Navify MP starts by processing variant input files from secondary analysis, which are produced downstream of sample preparation, target enrichment, and sequencing (Figure 1). The software filters variants based on quality metrics and variant database prevalence information. Those variants that pass filters are annotated with scientific and therapeutic information, including variant tier classifications. The software leverages tier classification guidelines based on the joint consensus recommendation of the Association of Molecular Pathology, American Society of Clinical Oncology, and College of American Pathologists (AMP/ASCO/CAP) [6,7]. These guidelines categorize variants into four distinct tiers, each defined by clinical impact. Tier I represents the highest clinical relevance and is subdivided into Tiers IA and IB, depending on local regulatory approval status and clinical evidence. Tier IA includes biomarkers that have approved therapies for that specific cancer type. These biomarkers have both a proven clinical utility and an approved therapeutic application. Tier IB includes variants supported by well-powered studies with consensus from experts in the field. Tier II variants have preliminary or emerging evidence of clinical relevance. While the evidence for these variants may not be as robust as for Tier I, they can still provide actionable insights, particularly in specific clinical contexts or for investigational treatments. Tier III variants, on the other hand, represent cases where the clinical impact remains uncertain due to limited or conflicting evidence. These may include novel or uncharacterized mutations in cancer-associated genes that lack sufficient data to inform treatment decisions. Finally, Tier IV variants are benign or likely benign, with no association with cancer biology or clinical outcomes. These are generally incidental findings, such as germline polymorphisms, which do not influence cancer progression, prognosis, or treatment decisions.

The Navify MP software leverages this variant classification system, dynamically adapting tier classifications to align with regulatory approvals and clinical evidence specific to the region of the input sample. By remaining up to date on regional regulatory changes and the evolving evidence landscape, the software generates reports that inform providers, offering guidance based on detected somatic genomic variants and their clinical relevance.

Although this classification system is applicable to both solid and hematologic malignancies, there are important differences in its application due to the distinct biological and clinical characteristics of each. Hematologic malignancies, in particular, present unique challenges, such as the frequent presence of multiple co-occurring variants and a greater reliance on diagnostic and prognostic biomarkers. Therefore, it is essential to conduct a clinical evaluation specifically assessing the software’s performance for hematologic malignancies.

In this study, we conducted an evaluation to validate the agreement between Navify MP-based variant tier classifications and those determined by clinical experts, specifically for hematologic malignancies. It is known that there are distinct differences in variant interpretation and tiering of certain variant–therapy combinations in hematologic cancers [8,9,10]. We performed a comprehensive agreement study using variants with specific relevance to hematologic malignancies and an updated version of the Navify MP software that integrates tiering and therapeutic recommendations specifically for hematologic malignancies. We assessed percent agreement by comparing Navify MP’s classifications to test participants’ Navify MP output using a set of contrived variant call format (VCF) files analyzed by laboratories in three geographic regions with different tier classifications based on the specific recommendations of their governing and advisory bodies.

2. Materials and Methods

2.1. Study Design

This study was designed to evaluate Navify MP’s annotation, tier classification, and curation process by assessing the percent agreement between Navify MP’s tier classifications and output compared to expert users’ tier reclassifications using Navify MP software. Seven software users across three regions (Canada, EU, US) processed fourteen contrived VCF files (Figure 2). The study used Navify MP content released in March 2020, which contained curation for 21,715 variants and 169,495 biomarker profiles. The software users received training on Navify MP v.2.0 (Roche Sequencing Solutions, Santa Clara, CA, USA) and on the study before testing. Training records were completed for each user. For each case, the software processed a VCF file and generated a report with variants classified according to ASCO/AMP/CAP tiers.

The concordance between the output from the software and that generated by the expert users was established by comparing expert user reclassifications to the Navify MP-determined classifications for the contrived dataset. Seven expert users across three different regions were included in the study to mitigate bias due to region and experience. The Navify MP users were laboratory directors, either pathologists or clinical geneticists who sign out NGS reports in a clinical laboratory setting. Testing laboratories were located in various regions around the world and were experienced in processing NGS cases. The expert users were involved in neither the development of the software nor the curation of the content.

2.2. Test Methods

Concordance between tier classifications and therapy matches was evaluated solely using Navify MP-assigned Tier IA variants and genomic signatures, with either “therapies approved/guidelines recommended in: matching indication” or “no approved therapies”, which are not re-categorized by the user. These stipulated categories are primarily based on the differential risk to a patient from the correct treatment options at Tier IA, which are supported by drug labels or medical professional guidelines. Treatment options associated with variants in Tier IB or lower have not reached a threshold supported by drug labels or professional medical guidelines. Concordance was determined collectively for both indicated and contraindicated treatment options.

The test methodology is shown in Figure 2. Starting with contrived data files containing known hematologic variants, the expert users created cases in Navify MP and uploaded the files for analysis. Navify MP generated tier classifications and annotations for the variants and genomic signatures. The expert users reviewed the Navify MP-generated reports and evaluated tiers and therapeutic annotations. If the user disagreed with the assigned tier and/or associated therapies, the user was instructed to review the Navify MP output and reclassify the tiers and associated therapies as needed.

The following acceptance criteria for the primary objectives were determined:

• With the exact same filter set applied, not allowing for any deviations, the reproducibility of the software’s curation of tier classifications and treatment options of hematologic malignancies was expected to be 100%. The output from Navify MP generated by the users should match the reference result.
• Given a pre-defined filter set, the accuracy of the software’s curation of tier classifications (Tier IA vs. non-Tier IA) of individual variants from hematologic malignancies was expected to be ≥85%. This threshold of 85% was chosen due to the known concordance from the AMP’s VITAL challenge, which found an overall concordance between AMP member classifications and expected classifications of 81% [11], the subjectivity of tier IA classifications [12], and the limited total number of Tier IA variants in hematologic malignancies. As Navify MP enables expert users to reclassify variants according to their judgment, it is functionally robust to disagreement between user and software-determined classification. The goal of the acceptance criteria is to ensure the user is not overwhelmed with large numbers of disagreements, rather than to penalize the software and databases for the occasional disagreement.
• The accuracy of treatment options (Navify MP-assigned Tier IA with either “therapies approved/guidelines recommended in: matching indication” or “no approved therapies” without reclassification) for individual variants from hematologic malignancies was expected to be ≥85%. Again, 85% was chosen due to the subjectivity involved in the determination of treatment options, which are not completely prescriptive as per guidelines and ultimately rely on professional expert judgment [12].

Concordance of tier classification was defined as user agreement with the Navify MP-assigned tier classification versus reclassifications between the Navify MP Tier IA and non-Tier IA group. For that concordance analysis, the specific tiering of the non-Tier IA classifications was not relevant, as the concordance was focused on Tier IA variants. However, concordance based on all specific tiers was assessed in a separate analysis. Concordance of treatment options was defined as user agreement with the Navify MP-assigned treatment option(s) for Tier IA variants and genomic signatures. This analysis included agreement of both indicated and contraindicated treatment recommendations.

2.3. Variant Validation and Test Articles

The variants evaluated in the study were established by the internal development team at Roche, which selected a range of variants from commonly mutated genes chosen by a medical science expert, who in turn approved the selected variants. In total, 86 individual variants from curated cancers spread across 14 contrived VCFs (12 from hematologic malignancies and 2 from solid tumors) were included. The users were not involved in the development of the software, including the curated content.

The contrived VCFs contained somatic sequence variants and, in some cases, genomic signatures (specifically microsatellite instability [MSI] and tumor mutation burden [TMB]) selected for their high clinical significance across a representative sample of solid and hematologic malignancies (Supplemental Table S1). The hematologic malignancies included various types of leukemia (acute myeloid [AML], acute lymphoblastic [ALL], chronic myeloid [CML]) and lymphoma (diffuse large B cell [DLBCL] and follicular [FL]) as well as multiple myeloma (MM). Variants included the following classes of genomic alterations: single nucleotide variants, small insertions and deletions, fusions and rearrangements, copy number variants, and multi-nucleotide variants. Concordance of the variant and genomic signature tier classifications of variants were categorized by tier group: Tier IA versus non-Tier IA. Non-Tier IA variants included Tiers IB, IIC, IID, and III. Tier IV variants are non-pathogenic and were therefore excluded. Users, however, had the option of reclassifying a variant as Tier IV. The rationale for such concordance criteria was based primarily on the differential risk to a patient of receiving a therapy decision guided by information related to a variant assessed incorrectly at the Tier IA level.

2.4. Replication Test of Usability Across Users

Roche provided each user access to Navify MP v2.0 after the user had been trained on the software. Roche created an assay that contained realistic NGS filters used for testing the VCFs. Each user was provided test instructions and executed the testing independently. The test instructions encompassed the creation of cases, uploading VCFs, and review of the Navify MP output. Users submitted test results, Navify MP reports, and screen shots to Roche in accordance with the test instructions.

2.5. Statistical Analysis

In this study, a total of 86 independent variants were evaluated by 7 independent expert users from 3 different regions, resulting in a total of 602 independent evaluations of hematologic variants. From a sample size precision analysis, for a 90% positive percent agreement, 280 (=40 variants × 7 users) samples were needed to achieve a lower bound of two-sided 95% CI of >=85%. Therefore, a variant count of approximately 40 Tier IA variants and approximately 40 non-Tier IA (Tiers IB, IIC, IID, and III) from clinical curated cancers were selected for the comparison. All variants were specifically tiered based upon specific hematologic malignancy annotations.

An agreement analysis was performed on tier assignment between reference results and the outputs of Navify MP generated by the users from each testing site for the variants and genomic signatures. The reference results were generated by Roche operators. Agreement analyses were performed separately for the following conditions:

• For replication testing: Using the exact same filter set applied without tier reclassification
• For agreement: Using pre-defined filters with tier reclassification as decided by the software users

The same analysis was performed when the treatment assignments of the software output and that of the experts were in agreement for Navify MP-assigned Tier IA variants.

Separate exploratory analyses were performed to study tier and treatment option agreements for the MSI and TMB genomic signatures as well as for variant combinations. In addition, tier agreement was also assessed between Navify MP software and the expert majority result (i.e., the majority result determined among the expert users for Tier IA and non-Tier IA for all variants evaluated).

3. Results

3.1. Hematologic Malignancies: Agreement Based on Concordance

The agreement between software users’ assignments and reference results was evaluated to determine concordance (Supplemental Table S1). Expert users were allowed to modify the tier classification or treatment assignments reported by the software based on their own expertise/knowledge if they did not agree with those assignments. The expert users could reclassify tiers, delete or exclude treatment options, and record additional treatment options, as needed, after reviewing the output from the software.

3.1.1. Individual Variants

The results of agreement analyses for the tier classification of individual hematologic variants—between user reclassifications and reference results—are shown in

Table 1. Agreement of Variants by Tier.

Tier	Total Variants *	Variants in Agreement	Agreement% (95% CI)
IA	254	232	91.34% (87.18–94.49%)
Non-IA	348	340	97.70% (95.52–99.00%)
IB	26	21	80.77% (60.65–93.45%)
IIC	175	161	92.00% (86.94–95.56%)
IID	21	19	90.48% (69.62–98.83%)
III	126	122	96.83% (92.07–99.13%)
Overall (IA vs. non-IA)	602	572	95.02% (92.96–96.61%)

* Overall agreement between all individual hematologic variants was 92.19% (555/602). CI: confidence interval.

. Of 602 total variants, users agreed with the tier classification (IA vs. non-IA) for 572 and reclassified 30 variants, resulting in an overall percent agreement of 95.02% (95% CI: 92.96–96.61%). This met the acceptance criteria of ≥85% agreement on tier classification. Agreement analysis was also performed for individual hematologic variants in a setting where users were allowed to reclassify disputed results based on the specific tier classifications (Table 1), accounting for differences in expert user opinion regarding tier IB/IIC/IID/III classifications. The overall agreement was 92.19% (555/602) between the reference results and the tiers produced through the software by the users after tier reclassification.

3.1.2. Subgroup Analysis of Biomarkers

Table 2. Subgroup Analysis of Agreement for Tier Classification (IA and Non-IA) for Individual Hematologic Variants (with Tier Reclassification).

		For Tier IA		For Non-Tier IA
		Total Variants *	Agreement % (n/N; 95% CI)	Total Variants *	Agreement % (n/N; 95% CI)
Tumor Type	ALL	52	86.54% (45/52; 74.21–94.41%)	53	98.11% (52/53; 89.93–99.95%)
	AML	92	98.91% (91/92; 94.09–99.97%)	55	100.0% (55/55; 93.51–100.0%)
	CML	42	85.71% (36/42; 71.46–94.57%)	35	100.0% (35/35; 90.00–100.0%)
	DLBCL	21	100.0% (21/21; 83.89–100.0%)	70	98.57% (69/70; 92.30–99.96%)
	FL	12	100.0% (12/12; 73.54–100.0%)	93	95.70% (89/93; 89.35–98.82%)
	MM	35	77.14% (27/35; 59.86–89.58%)	42	95.24% (40/42; 83.84–99.42%)
Region	Canada	64	93.75% (60/64; 84.76–98.27%)	108	95.37% (103/108; 89.53–98.48%)
	EU	64	100.0% (64/64; 94.40–100.0%)	108	100.0% (108/108; 96.64–100.0%)
	US	126	85.71% (108/126; 78.37–91.31%)	132	97.73% (129/132; 93.50–99.53%)
Software User	User 1	32	100.0% (32/32; 89.11–100.0%)	54	90.74% (49/54; 79.70–96.92%)
	User 2	32	87.50% (28/32; 71.01–96.49%)	54	100.0% (54/54; 93.40–100.0%)
	User 3	32	100.0% (32/32; 89.11–100.0%)	54	100.0% (54/54; 93.40–100.0%)
	User 4	32	100.0% (32/32; 89.11–100.0%)	54	100.0% (54/54; 93.40–100.0%)
	User 5	42	64.29% (27/42; 48.03–78.45%)	44	95.45% (42/44; 84.53–99.44%)
	User 6	42	100.0% (42/42; 91.59–100.0%)	44	100.0% (44/44; 91.96–100.0%)
	User 7	42	92.86% (39/42; 80.52–98.50%)	44	97.73% (43/44; 87.98–99.94%)

* Variant classifications are presented according to how they are assigned based on Navify MP software output and used as the reference for reference–user agreement calculations. ALL: acute lymphoblastic leukemia; AML: acute myeloid leukemia; CI: confidence interval; CML: chronic myeloid leukemia; DLBCL: diffuse large B-cell lymphoma; EU: European Union; FL: follicular lymphoma; MM: multiple myeloma.

presents the subgroup analyses of agreement between reference software output and users (in a setting where users were allowed to modify results) for tier classification by tumor type, region, and software user. The agreement for DLBCL and FL tumor type variants was 100% for Tier IA classification. For the EU region, the agreement for tier classification (IA vs. non-IA) was 100%. Four of seven software users were in 100% agreement on tier classification for Tier IA variants. Differences in the variant counts between users 5, 6, and 7 and those of the other users is due to regional differences, as these three users were located in the United States, while the other users were in Canada (users 1 and 2) and the European Union (users 3 and 4). The number of Tier IA and non-Tier IA variants were selected based on the tier classification in the US region, with 42 Tier IA variants and 44 non-Tier IA variants. The tier classification (IA vs non-IA) is different across different regions, which resulted in 32 Tier IA variants and 54 non-Tier IA variants in the EU and Canada regions.

3.1.3. Variant Combinations

Navify MP also annotates and determines tier classifications for variant combinations, meaning multiple variants that contribute to a single recommendation and which have their own tier classifications. Expert users agreed with the Navify MP-determined tiers in 100% (133/133; 95% CI: 97.26–100%) of variant combinations relevant to hematologic malignancies included in the study (

Table 3. Exploratory Agreement on Tier Classification (IA and Non-IA) for Hematologic Variant Combination (with Tier Reclassification).

Tier	Total Variants *	Variants in Agreement	Agreement % (95% CI)
IA	17	17	100.0% (80.49%, 100.0%)
Non-IA	116	116	100.0% (96.87%, 100.0%)
Overall	133	133	100.0% (97.26%, 100.0%)

* Variant classifications are based on Navify MP software output and used as the reference for agreement. CI: confidence interval.

). Although the expert users were in complete agreement with the Navify MP-determined tier classifications for the variant combinations, they agreed with only 53 of the 56 treatment options assigned by Navify MP, representing an agreement of 94.64% (95% CI: 85.13–98.88%) (

Table 4. Exploratory Agreement on Treatment Assignment for Tier IA Hematologic Variant Combinations (with Tier Reclassification).

	Total Treatment Options *	User-Assigned Treatment Options	Agreement on Treatment Assignment
			Agreement% (n/N)	95% CI
Navify MP-Determined Tier IA Variant Combinations (n = 17)	56	53	94.64% (53/56)	(85.13–98.88%)

* Total numbers of treatment options were based on both the software and the user. More than one treatment option can be assigned per variant. CI: confidence interval; Navify MP: Navify Mutation Profiler.

).

A similar analysis was performed for the variants identified as Tier IA by both Navify MP and the users. The percent agreement for treatment for Navify MP-assigned and user-assigned Tier IA variant combinations was 94.64%, the same as that for Navify MP-assigned Tier IA variant combinations (Table 4).

3.2. Treatment

For 254 Navify MP-assigned Tier IA individual variants, a total of 325 treatment options were either identified by the Navify MP software (n = 324) or assigned by the software users (n = 1). Of the 325 treatment options, users agreed with 322, resulting in a percent agreement in treatment assignment of 99.08% (95% CI: 97.33–99.81%), which met the acceptance criteria of ≥85% (

Table 5. Agreement of Variants based on Treatment Assignment.

	Total Variants	Total Treatment Options *	Agreement on Treatment Assignment
			Agreement % (n/N)	95% CI
All Navify MP-Assigned Tier IA Individual Variants	254	325	99.08% (322/325)	(97.33–99.81%)
Individual Navify MP- and User-Assigned Tier IA Variants,	232	303	99.67% (302/303)	(98.17–99.99%)

* Total numbers of treatment options were based on both the software and the user. More than one treatment option can be assigned per variant. CI: confidence interval; Navify MP: Navify Mutation Profiler.

).

Treatment agreement analysis was also performed for the variants identified as Tier IA by both Navify MP and the software users. In this analysis, the agreement for treatment was 99.67% (302/303; Table 5).

3.3. Solid Tumors: Agreement Based on Concordance

For solid tumors, the overall agreement for tier assignment (IA vs. non-IA) between the reference results and users was 100% for TMB and MSI biomarker signatures and 98.76% (159/161; 95% CI: 95.58–99.85%) for all individual variants (

Table 6. Agreement on Tier Classification (IA and Non-IA) for Solid Tumors (with Tier Reclassification).

	Tier	Total Variants *	Variants in Agreement	Agreement % (95% CI)
All Individual Variants	IA	67	67	100.0% (94.64–100.0%)
Navify MP and User-Assigned Tier IA Variant Combinations	Non-IA	94	92	97.87% (92.52–99.74%)
	Overall	161	159	98.76% (95.58–99.85%)
TMB	IA	5	5	100.0% (47.82–100.0%)
	Non-IA	9	9	100.0% (66.37–100.0%)
	Overall	14	14	100.0% (76.84–100.0%)
MSI	IA	3	3	100.0% (29.24–100.0%)
	Non-IA	11	11	100.0% (71.51–100.0%)
	Overall	14	14	100.0% (76.84–100.0%)

* Variant classifications are based on Navify MP software output and used as the reference for agreement. CI: confidence interval; MSI: microsatellite instability; TMB: tumor mutation burden.

).

Solid Tumors: Treatment

In total, 15 treatment options were identified by Navify MP for the five Navify MP-assigned Tier IA TMB biomarkers, and three treatment options were identified for the three Navify MP-assigned Tier IA MSI biomarkers. The users were in 100% agreement with the Navify MP-assigned treatment options for these Tier IA TMB and MSI biomarkers (

Table 7. Exploratory Agreement on Treatment Assignment for Tier IA Variants from Solid Tumors (with Tier Reclassification).

		Total Variants	Total Treatment Options *	Agreement on Treatment Assignment
				Agreement % (n/N)	95% CI
Navify MP-Assigned Tier IA Variants	All Individual Variants	67	154	96.75% (149/154)	(92.59–98.94%)
	TMB	5	15	100.00% (15/15)	(78.20–100.00%)
	MSI	3	3	100.00% (3/3)	(29.24–100.00%)
Navify MP and User- Assigned Tier IA Variants	All Individual Variants	67	154	96.75% (149/154)	(92.59–98.94%)
	TMB	5	15	100.00% (15/15)	(78.20–100.00%)
	MSI	3	3	100.00% (3/3)	(29.24–100.00%)

* Total number of treatment options was based on both the software and the user. More than one treatment option can be assigned per variant. CI: confidence interval; MSI: microsatellite instability; Navify MP: Navify Mutation Profiler; TMB: tumor mutation burden.

). The agreement for treatment for all individual variants from solid tumors was 96.75%.

A similar analysis was performed for the variants identified as Tier IA by both Navify MP and the users. In this analysis, the agreement in treatment for Navify MP- and user-assigned Tier IA variants was 100% for both TMB and MSI biomarkers (Table 7)—the same values as were seen for Navify MP-assigned Tier IA variants.

3.4. Replication Test of Usability Across Users

The replication test was used to evaluate the agreement between the software users and the reference results, in an environment where software users use the same filters set by the Navify MP software without any changes.

In the replication test, the agreement between the reference results and the tiers produced through the software across seven users was 100% (602/602; 95% CI: 99.39–100%) for all individual variants. The results of tier agreement based on Tier IA versus non-Tier IA classification met the acceptance criteria of 100% agreement for reproducibility.

Table 8. Replication Agreement of Users Versus Navify MP Output.

Tier	Total Variants *	Variants in Agreement	Replication Agreement % (95% CI)
IA	254	254	100.0% (98.56–100.0%)
Non-IA	348	348	100.0% (98.95–100.0%)
IB	26	26	100.0% (86.77–100.0%)
IIC	175	175	100.0% (97.91–100.0%)
IID	21	21	100.0% (83.89–100.0%)
III	126	126	100.0% (97.11–100.0%)
Overall	602	602	100.0% (99.39–100.0%)

* Variant classifications are based on Navify MP software output and used as the reference for agreement. CI: confidence interval.

outlines the primary agreement analyses of users for the Navify MP-assigned tier classifications for individual hematologic variants regarding software output compared with reference results. A similar analysis was also performed on tier classification for individual hematologic variants based on the non-Tier IA categories (Table 8). The agreement between the reference results and the tiers produced through the software by the users was 100% (602/602; 95% CI: 99.39–100%) for all the categories.

4. Discussion

Navify MP accurately annotated 95% of all variants and 91.34% of Tier IA variants, across multiple users’ hematologic variants. The replication test was consistent across global users with 100% replication on all systems. In the Tier I individual tier classifications, agreement was reached for 91.34% of variants at Tier IA and 80.77% in Tier IB. The sample set for Tier IB was substantially smaller than for Tier IA and, in general, there is discordance within the molecular pathology community about Tier IB variants. Tier I variants are divided into A and B by ASCO/AMP/CAP classification guidelines, with Tier IA requiring regulatory approval or inclusion in professional guidelines. Tier IB requires well-powered studies and/or consensus from experts in the field. As a result, Tier IB had more natural variability in consensus than Tier IA, which is more defined [11]. We have previously shown that subjectivity in the terms used to define tiers, especially Tier IB (e.g., “well-powered studies” and “consensus”), results in a reduction in consistent annotation of Tier IB variants between different tertiary analysis software [12]. Excluding Tier IB, agreement for all other lower tiers was above 90%, showing strong alignment with Navify MP. The agreement on treatment assignment was 99.08% between users and the treatment output by Navify MP. This result shows Navify MP provided clinically meaningful and accurate annotations, supporting its use for matching patients to targeted treatments.

It is estimated that hematologic malignancies make up approximately 11.5% of all cancers globally [13] and have their own distinct biomarkers and associated therapies [14]. Due to the complexity involved in some of the hematologic pathways, analysis specifically into hematologic malignancies was identified as important by stakeholder engagement of HCPs. However, in a broader context, particularly in large community-based oncology centers, large volumes of patients with both solid and hematologic malignancies are seen, increasing the need for CDS in an HCP’s workflow [15,16]. Although some solid malignancies were included in this analysis, future studies should explore both solid and hematologic malignancies in the same analysis. Another concern was that users may have encountered a Navify MP-determined tier classification and agreed without independently validating the determination, leading to confirmation bias. However, most of the users in this study reclassified variants based on their independent expertise. Inclusion of a number of false cases in future studies might be a good mechanism to assess this bias. Lastly, Navify MP has the capacity to provide outputs across different geographic regions, in accordance with the latest drug approvals and guidelines from agencies such as the European Medicines Agency, the United States Food and Drug Administration, Health Canada, National Institute for Health and Care Excellence, and Swissmedic. However, because this study only assessed classifications based on the US, Canada, and EU regions, agreement percentages that may occur in other regions are beyond the scope of this study.

Navify MP incorporates approval status from regulatory bodies in various geographic regions into tier assignments. The oncogenicity of a variant or approved therapy may vary based on levels of evidence in different regions. A major benefit of Navify MP is the ability to capture a large range of publicly available evidence. The complexity of drug-resistance pathways and assessing variant combinations is an area where device CDS tools such as Navify MP may have utility [15,16]. It has been shown that with an increasing number of variants and associated therapies, there are challenges with interpreting NGS output in an accurate and consistent manner. Differences in the way that variants are annotated has been shown to result in inconsistent variant interpretation and associated therapy options [17,18]. Moreover, evidence from AMP’s VITAL study has shown that there is disagreement between AMP members about the tier classification of variants, with just 81% of respondents (1119/1379, 95% CI = [79–83%]) in that study agreeing with the intended tiering [11]. It can be anticipated that consistent interpretation of variants will only become more difficult as the number of variants and drug therapies in oncology NGS continues to increase. In the context of increasing variants and associated drug therapies, device CDS systems such as Navify MP have a unique role in offering a ‘one-stop shop’ for laboratories and HCPs to stay up to date with changes in recommendations, new variants and variant combinations, and associated drug approvals, as well as with new findings from scientific publications [19]. Rare actionable genes may not be as familiar to laboratory professionals and HCPs. Therefore, Navify MP may be useful for identifying patients for matched therapies that otherwise might be missed and may potentially improve patient survival. Overall, automation provided by tools such as Navify has the potential to reduce errors and increase access to novel variants with associated therapies.

Accurate and consistent interpretation of variants and automating the steps in this process are vital developments in the evolution of precision oncology. In our validation study with seven external users across four institutions, Navify MP output showed high agreement with molecular genetics expert opinion in tier assignments and associated therapies for 86 hematologic variants. CDS tools such as Navify MP and other CDS systems can aid in the streamlining of oncology NGS variant reporting and associated therapies, while reducing error and saving time in busy clinical practices.