Prostate MRI PI-RADS Scoring by the Machine Learning Software Quantib^® Prostate: A Retrospective Agreement Pilot Study

Abstract

Objective: Quantib^® Prostate is a commercially available machine learning software (MLS) used for prostate magnetic resonance imaging (MRI), approved by the Food and Drug Administration (FDA) and Conformité Européenne (CE). We aimed to assess the agreement between biparametric MRI (bpMRI) interpretations produced by radiologists and by Quantib^® alone. Material and Methods: This single-centre, retrospective, and observational study included 188 bpMRI scans from the Innlandet Hospital Trust, Norway. Radiologists’ Prostate Imaging Reporting and Data System (PI-RADS) scores were compared with scores produced by the Quantib^® Prostate PI-RADS, which was used as an autonomous software. Additionally, for exploratory purposes, the PI-RADS scores were compared with biopsy findings, using the radiologist’s scorings to determine biopsy indications. Results: For PI-RADS scores generated by radiologists vs. those produced by Quantib^® Prostate, the weighted kappa was 0.49 (95% CI: 0.37–0.59), and 46% of the PI-RADS scores differed by at least one unit. In the sample, 125 men (66%) had a prostate biopsy, and for male patients with PI-RADS scores of 4–5, the probability of detecting cancer grades 2–5 via biopsies was 64% (95% CI: 56–71) for radiologists and 58% (95% CI: 54–62) for Quantib^® Prostate. The main limitations of this study are the retrospective, single-centre design, the small sample size, the use of Quantib^® Prostate in an autonomous manner, and verification bias, as the biopsy decisions were based solely on PI-RADS scoring by radiologists. Conclusions: We identified inter-rater variability between PI-RADS scores generated by Quantib^® Prostate and radiologists’ assessments, and the agreement between the two scoring groups is moderate. Our results suggest that Quantib^® Prostate requires further clinical comparative studies before it is used in clinical routines.

1. Introduction

In recent years, magnetic resonance imaging (MRI) has become an important tool for detection and risk stratification in prostate cancer diagnostics. European guidelines recommend a prostate MRI early in the diagnostic pathway, before deciding whether to perform a prostate biopsy [1]. According to the International Society of Urological Pathology (ISUP)’s five-tier grading system, a challenge facing prostate cancer diagnostics involves avoiding the detection of low-grade cancer—i.e., grade group (GG) 1 [2]. GG 1 prostate cancer progresses slowly and does not metastasise [3]; hence, this type of cancer is best left undiagnosed, to avoid overtreatment. By contrast, GG 2–5 (intermediate to high-grade) cancer may progress more rapidly to lethal cancer and is often considered clinically significant [4]. Since findings on prostate MRI correlate with the presence of GG 2–5 cancers, MRI is a critical part of the diagnostic pathway [5].

Multiple pieces of machine learning software (MLS) have been developed to interpret prostate MRI [6]. However, the positive and negative predictive values of most prostate MRI MLS are not high enough to forgo a radiologist’s interpretation. In addition, there is a need for validation studies before an MLS is implemented.

Prostate MRI is suitable for machine learning software (MLS), since the classification of abnormal findings is based on a commonly defined and limited number of sequences in a relatively small and uniform gland that is not easily accessible by other diagnostic means. Lesions on prostate MRI are categorised using the five-point Prostate Imaging Reporting and Data System (PI-RADS) [7], with a PI-RADS score of 1 indicating that clinically significant prostate cancer is highly unlikely and a PI-RADS score of 5 implying a high probability of significant cancer.

Quantib^® Prostate (Quantib B.V. Rotterdam, The Netherlands) is a Food and Drug Administration (FDA)-approved and Conformité Européenne (CE)-certified MLS that is commercially available for prostate MRI [8,9]. In the studies published thus far, radiologists have used Quantib^® Prostate to interpret both multiparametric and biparametric prostate MRI [10,11]. With the present study, we wanted to test the autonomous use of Quantib^® Prostate on biparametric MRI (bpMRI), independent of radiologist input. For exploratory purposes, we also evaluated PI-RADS scores generated by radiologists versus scores produced by Quantib^® Prostate by assessing how well Quantib^® Prostate detected GG ≥ 2 cancer (i.e., significant cancer)—to do so, we correlated Quantib^® Prostate PI-RADS scores with biopsy-based detection in men with PI-RADS scores of 4 or 5 (as determined by radiologists).

2. Materials and Methods

This is a single-centre, retrospective, observational, exploratory agreement study by Innlandet Hospital Trust, Norway. We requested approval for the re-evaluation of data from 300 consecutive patients who underwent prostate bpMRI examines from 1 March 2022 to 7 June 2022, all with the same MRI protocol. In total, 188 eligible men consented to participate.

The prostate examinations were performed with biparametric sequences on a 1.5 Tesla Siemens MAGNETOM Sola (Siemens Healthineers, Erlangen, Germany) without an endorectal coil. Three sequences were used: T2-weighted images, the apparent diffusion coefficients (ADCs), and a diffusion-weighted imaging (DWI) series with a calculated b-value of 1500. The suppliers of Quantib^® Prostate confirmed that their software could be utilised with these specifications.

The MRI scans were transferred from the hospital’s picture archiving and communication system (PACS) to compact discs (CDs). A data protection impact assessment (DPIA) was prepared specifically for this study; for security reasons, direct transfer between the PACS and the research server was not permitted. Staff from the radiology department were responsible for copying the CDs, with one CD used for each patient. The images were then uploaded to a research-dedicated cloud server and analysed with Quantib^® Prostate version 3.2 (Quantib BV, Rotterdam, The Netherlands).

Subsequently, a single urologist (AK) with no specific radiology training used the software on a local stationary PC to analyse the entire dataset. AK is both a general surgeon and urologist who received his speciality in urology in 2022; his fields of expertise are kidney surgery and general urology, and he has limited experience with prostate MRI. Prior to starting the analysis, he received one hour of instruction on how the software worked from technicians at Quantib^® Prostate. First, Quantib^® Prostate defined the boundaries and the segmentation of the prostate. Then, AK used the biparametric combination image thresholding feature to locate “regions of interest”, and then he scored lesions according to the PI-RADS v2.1 grade suggested by the scoring manual in the software. Since this study compares the Quantib ^® Prostate software with radiologist assessments, care was taken to ensure that the analysis was performed by the software, with the urologist merely accepting its suggestions. The software was easy and intuitive to use, and the analysis took place between 2 February 2024, and 11 March 2024, with 10–30 cases analysed per session.

Figure 1 shows a region of interest set by Quantib^® Prostate. The software differentiated lesions in the peripheral, transitional, and central zones, with findings in each zone having different bearings on the PI-RADS score, in line with international PI-RADS criteria built into the software. If multiple lesions were present, we used the highest PI-RADS score in the analyses. Lesions smaller than 4 mm were ignored as a study-specific criterion.

The MLS results were compared to the radiologists’ results. The original interpretations were performed by eight radiologists whose experience in MRI prostate analysis ranged from 0 years to more than 10 years. Three of the radiologists were considered experienced, with more than 1000 prostate cases each, and the other five were less experienced, with case numbers ranging from 0 to approximately 500. If the radiologist had limited experience, the images were also assessed by a radiologist with extensive experience with prostate MRI.

Each upload from the CDs to the cloud took approximately 4 min, and pre-processing in the cloud required approximately 3–4 min (depending on the size of the prostate). The bpMRI analysis by the urologist took less than 3 min to perform for each case. Finally, the generation of the results took 2 min.

The software workflow is illustrated in Figure 2. The data were pseudonymised, and the urologist who used Quantib^® Prostate did not have access to the prior interpretations of the MRI or clinical data. The CDs were destroyed after the images were uploaded to the local workstation; likewise, the information uploaded to the cloud server was deleted after the analysis was performed.

The necessary data required for this study (i.e., PI-RADS scores determined by radiologists and information about GG on biopsies) were obtained from electronic medical records. The hospitals’ guideline for prostates was 3–5 targeted biopsies, and only with lesions categorised as PI-RADS scores of 4 or 5. Lesions with a PI-RADS score ≤ 3 were also biopsied (either targeted or systematic; the latter only if no lesions were detected on MRI) if the prostate-specific antigen (PSA) density (PSA divided by prostate volume) was greater than 0.15 ng/mL/cm³; exceptions to the latter routine were palpation suspect of malignancy and history of prostate cancer in close family. Each core of a biopsy with confirmed prostate cancer was separately graded. The biopsy indication was defined using the radiologists’ PI-RADS scores.

3. Ethics

This study was approved by the data protection officer at our institution and the Regional Ethics Committee (REC) (reference number: 625398, 5 September 2023). Since sensitive information was uploaded to a cloud server, a DPIA was prepared by the data protection officer. In addition, an agreement between our institution and Quantib^® Prostate was established to ensure that sensitive data were protected.

4. Statistics

The agreement between the PI-RADS scores assigned by radiologists and those assigned by Quantib^® Prostate was calculated as a weighted kappa with a 95% confidence interval (CI). The weighted kappa is the most common measure of agreement between two ordinal scales. Moreover, quadratic weighting was applied in order to weight scores farther apart more heavily than scores closer together. Bowker’s test [12] for table symmetry was performed to assess whether the potential disagreement between the PI-RADS scores from Quantib^® Prostate and the radiologists was caused by a systematic difference between the approaches.

For the calculation of conditional probabilities, PI-RADS scores were divided into three groups: PI-RADS ≤ 3, PI-RADS 4, or PI-RADS 5. The GG in the biopsy was dichotomised into GG = 1 vs. GG ≥ 2; with this, a ratio between the number of men with PI-RADS scores of 4 or 5 + a GG > 1 and the total number of men with PI-RADS 4 or 5 was calculated, and it is presented with corresponding Wald-type 95% CIs. This ratio represents the probability that a man with a PI-RADS score of 4 or 5 will have a cancer GG > 1 in the biopsy. Sensitivity was defined as the proportion of men with a PI-RADS score of 4 or 5 + a GG > 1 among all men with a cancer GG > 1 detected in the biopsy. The exact Clopper–Pearson 95% CIs for sensitivity were presented, and the analyses were performed in Stata/SE v18 (StataCorp LLC, College Station, TX, USA).

5. Results

No technical failures for autonomous segmentation were observed. The observed agreement in the interpretation of prostate MRI results between radiologists and Quantib^® Prostate was 46%, meaning that almost half of the PI-RADS scores differed by at least one unit (

Table 1. Agreement between PI-RADS scores produced by Quantib^® Prostate (qPI-RADS) and those from radiologists (rPI-RADS), presented with frequencies and percentages n = 188).

qPI-RADS	rPI-RADS
	≤3	4	5	Total
≤3	34 (18)	8 (4)	2 (1)	44 (24)
4	49 (26)	33 (18)	7 (4)	89 (47)
5	11 (6)	9 (5)	35 (18)	55 (29)
Total	94 (50)	50 (27)	44 (23)	188 (100)

). The weighted kappa was 0.49 (95% CI: 0.37–0.59), indicating moderate agreement. In general, Quantib^® Prostate assigned a higher PI-RADS score, and the results of Bowker’s test for table symmetry were highly significant (p < 0.001), implying systematic differences between the approaches.

Of the 188 men in the sample, 125 (66%) had undergone prostate biopsies, with the decision of whether to perform a biopsy made after PI-RADS scoring by radiologists; 97 men were diagnosed with prostate cancer. Figure 3 shows the distribution of low-, intermediate-, and high-grade prostate cancer (by GG) together with the PI-RADS scores produced by radiologists and the PI-RADS scores generated retrospectively by Quantib^® Prostate.

The probability of having significant cancer upon (urologist-performed) biopsy with a high PI-RADS score (4–5) was 64% (95% CI: 56–71) for radiologist scoring and 58% (95% CI: 54–62) for Quantib^® Prostate scoring. Biopsies were avoided in 34% of men after radiologist scoring. The sensitivity of the PI-RADS for the detection of a GG ≥ 2 was estimated to be 96% (95% CI: 88–99) for Quantib^® Prostate and 84% (95% CI: 73–92) for radiologists. Figure 4 illustrates the distribution of significant cases of prostate cancer with PI-RADS scores ≤ 3 and 4–5.

6. Discussion

We identified a moderate agreement between the PI-RADS scores determined by Quantib^® Prostate and those generated via radiologist assessments. Quantib^® Prostate tended to assign higher scores to lesions than the radiologists. The conditional probabilities for detecting clinically significant prostate cancer (exploratory analysis) within the limitations of the study design were similar between Quantib^® Prostate’s and radiologists’ MRI assessments.

Although PI-RADS was developed to standardise reporting and reduce variability, inter-rater variability of prostate MRI remains a major clinical challenge [13], as interpretation varies across radiologists—both experienced and inexperienced—and across different MRI protocols [14]. Artificial intelligence (AI) and MLS have the potential to standardise PI-RADS reporting, making it more objective, reducing the risk of human error, and reducing the learning curve for radiologists. Software like Quantib^® Prostate may also increase the precision and quality of care further along the diagnostic pathway, such as with fusion biopsies. When image fusion biopsies are performed, either radiologists or (as at our institution) the urologists mark the suspicious lesion; this process could be aided with MLS, increasing the precision of cancer detection. Nevertheless, the most important objective of prostate MRI is to detect significant prostate cancer and to rule out non-significant prostate cancer.

A previous study on Quantib^® Prostate evaluated its capacity for cancer detection [10] and found slightly higher sensitivity when non-experienced radiologists used the MLS, with experienced radiologists’ scoring used as the reference. In addition, a recent systematic review on the diagnostic accuracies of different AI programmes used for prostate MRI interpretation showed promising results [15]. Our study differs from previous studies on Quantib^® Prostate because we used the software autonomously, not exclusively as a support for radiologists. The possible upside—if the diagnostic performance is comparable—is that less time is spent on image interpretation, and the diagnostic pathway is more efficient. Standalone AI models have already been reported to demonstrate superior diagnostic performance compared to radiologists [16]. Similar to our study, the PI-CAI trial investigated bpMRI. Its strengths were its multicentre design and large sample size. Its limitations include a retrospective design, different biopsy performance routines, and the range in years of experience for both radiologists and pathologists (1–21 years in both groups).

Although multiparametric MRI (mpMRI) is the gold standard for prostate assessment, bpMRI without endorectal coils has been the standard of care for more than a decade at our institution. Notably, bpMRI costs less than mpMRI and takes less time. The diagnostic performance and detection of significant prostate cancer with bpMRI have been proven to be effective: recently, prospective trials and a comparative systematic review have demonstrated that prostate cancer detection using bpMRI, reported using the latest version of the PI-RADS, is comparable to that of mpMRI [17,18,19].

An evaluation of the time spent on image interpretation and cost-effectiveness was beyond the scope of this study, but the urologist found the software easy to use. The introduction to Quantib^® Prostate was conducted digitally. As the urologist merely used the image interpretation provided by Quantib^® Prostate, we anticipate that operator bias did not impact the results. However, since Quantib^® Prostate is approved as a decision-support tool rather than as standalone software, our methodological choice may have introduced automation bias and influenced the results.

Increased effectiveness is a potential benefit of introducing AI into radiology: decreases in the times for image interpretation reduce workload and give radiologists more time to carry out other tasks. Presumably, the introduction of MLS for prostate MRI will likely improve cost-effectiveness, especially as most MLS are suited for bpMRI rather than mpMRI, but this needs to be studied specifically.

The objective of this study was to test an available MLS together with the specific bpMRI protocol used at our institution. Several other MLS for prostate MRI are available on the market [9]; these systems should also be tested in different institutions and with different MRI protocols. Most existing MLS for prostate MRI are non-autonomous, although software for autonomous interpretation (without the assistance of radiologists or urologists) already exists [16,20]. Autonomous MLS for prostate MRI could possibly produce even more objective interpretations, and this should be considered when choosing an MLS. One difficulty is that MLS will presumably be adopted and used in clinical practice before prospective—and eventually randomised—controlled studies are performed. Quantib^® Prostate was compared with radiologist assessments, but we did not address radiologists’ individual performance; instead, the aim was to compare it with “real-world” MRI data already used for triage and diagnostic purposes. One difference between the groups is that the radiologists had access to clinical data, such as PSA results. This represents a potential source of non-equivalence between the two approaches. However, we believe that this is unlikely to have materially affected the overall results. A subgroup analysis of radiologist inter-variability was not performed due to the small sample size and the fact that each analysis by an inexperienced radiologist was checked and signed off by an experienced radiologist, with or without corrections.

There are several limitations of this study, including its single-centre, retrospective, observational design and its small sample size. Although the retrospective design was adequate for comparing PI-RADS scores, no biopsies were performed when the Quantib^® Prostate software generated a biopsy indication, unless the radiologist’s score also identified a biopsy indication. This limits this study’s assessment of prostate cancer detection. Moreover, since the decision to perform a biopsy was based solely on the radiologists’ assessments, lesions might have been assigned higher or lower scores by Quantib^® Prostate, introducing verification bias (leading to both false-negative and false-positive results). The radiologists scored 94 out of 188 patients with PI-RADS 4 or 5. The urologist using Quantib^® Prostate scored 144 out of 188 (Table 1). Compared with the number of confirmed cancers in the cohort, the false-positive rate in the Quantib^® Prostate group was considerably higher than that of the radiologists. If we assume that the cancers detected by radiologists represent true-positive cases, interpreting PI-RADS scores with Quantib^® Prostate would generate a substantially higher number of unnecessary negative biopsies. As this was a retrospective study, no biopsies were performed following Quantib^® Prostate analysis; therefore, this remains uncertain, as it is unknown whether those who were not biopsied harboured cancer. Nevertheless, given the magnitude of the discrepancy, it is possible that the use of the software alone would result in a higher number of negative biopsies.

In addition, in line with the hospital’s routines, some men with PI-RADS scores of 3 or lower on their MRI had prostate biopsies—for instance, if the PSA density was high. Due to small group sizes, we did not include a subgroup analysis; consequently, some information about the diagnostic performances of both Quantib^® Prostate and radiologists might be missing. Finally, for men treated with radical prostatectomy, an evaluation against the histopathology of the prostatectomy specimen was not performed. A true and more valid evaluation requires comparison with the prostatectomy specimen, which is regarded as the gold standard, and this is a limitation of the current study.

Moreover, the prostate imaging quality (PI-QUAL) scoring system was not used in this study, and we cannot rule out the possibility that the image quality influenced the results. A revised version of PI-QUAL, which can be applied to both mpMRI and bpMRI, was released in 2024, and PI-QUAL should be assessed in future studies [21]. Quantib^® Prostate was primarily trained on 3.0T MRI scanners, although the MRI sequences and scanner strength we used were communicated to the suppliers before we conducted this study, and they confirmed that the software could be applied with these specifications. In addition, the lack of data on patients’ clinical characteristics is a weakness; nevertheless, because the patients constituted a consecutive sample from routine clinical care at a centre with a defined catchment area, the results may be valid in most other countries with similar PSA testing habits and prostate cancer incidence. Finally, a commercially available MLS has not been trained on the institutions’ own images, which limits the possibility of local feedback and learning.

In summary, prospective, ideally randomised, trials with larger sample sizes should be carried out to assess Quantib^® Prostate’s performance in cancer detection. In addition, the results may not be applicable to other institutions using other MRI protocols; consequently, our results have limited generalisability and need to be confirmed. Multicentre, prospective studies with different available MLS with large study populations are warranted to provide more robust data.

7. Conclusions, Clinical Implications, and Future Directions

In this retrospective study, there was inter-reader variability in PI-RADS scores between Quantib® Prostate and radiologists, with Quantib^® Prostate assigning higher scores. Nevertheless, the overall agreement was moderate. With regard to Quantib^® Prostate’s use in interpreting prostate MRI, we consider the results acceptable, but our findings suggest that further clinical and comparative studies are required before this software can be used in clinical routine. At this time, we cannot claim superiority or inferiority of MLS compared with radiologists, but we plan a prospective randomised trial to assess this.