Next Article in Journal
Liquid Biopsy in Early-Stage Lung Cancer: Current and Future Clinical Applications
Previous Article in Journal
Efficacy and Safety of High-Dose Chemotherapy with Treosulfan and Melphalan in Multiple Myeloma
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Cancers
Volume 15
Issue 10
10.3390/cancers15102700
cancers-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Machine Learning Improves the Prediction of Responses to Immune Checkpoint Inhibitors in Metastatic Melanoma

by
Azadeh Tabari
1,2,*,
Meredith Cox
2,
Brian D’Amore
2,3,
Arian Mansur
2,
Harika Dabbara
4,
Genevieve Boland
2,3,
Michael S. Gee
1,2 and
Dania Daye
1,2
1
Department of Radiology, Massachusetts General Hospital, 55 Fruit Street, Boston, MA 02114, USA
2
Harvard Medical School, Boston, MA 02215, USA
3
Department of Surgery, Massachusetts General Hospital, 55 Fruit Street, Boston, MA 02114, USA
4
Boston University Chobanian & Avedisian School of Medicine, 72 East Concord Street, Boston, MA 02118, USA
*
Author to whom correspondence should be addressed.
Cancers 2023, 15(10), 2700; https://doi.org/10.3390/cancers15102700
Submission received: 14 March 2023 / Revised: 12 April 2023 / Accepted: 27 April 2023 / Published: 10 May 2023
Browse Figures
Review Reports Versions Notes

Abstract

:

Simple Summary

Lactate dehydrogenase (LDH) levels prior to treatment are a known biomarker to predict advanced melanoma’s response to immune checkpoint inhibitors (ICI). In this study, we evaluated the ability of machine learning-based models to predict responses to ICI and complement LDH for in predicting the outcomes of metastatic melanoma. A machine learning algorithm was developed using radiomics, and further analysis helped select the most important predictive features and variables. The machine learning model that combined both features extracted from images (radiomics) and pretreatment LDH levels resulted in better predictions of cancer response to ICI than models that only use radiomics features or LDH levels alone.

Abstract

Pretreatment LDH is a standard prognostic biomarker for advanced melanoma and is associated with response to ICI. We assessed the role of machine learning-based radiomics in predicting responses to ICI and in complementing LDH for prognostication of metastatic melanoma. From 2008–2022, 79 patients with 168 metastatic hepatic lesions were identified. All patients had arterial phase CT images 1-month prior to initiation of ICI. Response to ICI was assessed on follow-up CT at 3 months using RECIST criteria. A machine learning algorithm was developed using radiomics. Maximum relevance minimum redundancy (mRMR) was used to select features. ROC analysis and logistic regression analyses evaluated performance. Shapley additive explanations were used to identify the variables that are the most important in predicting a response. mRMR selection revealed 15 features that are associated with a response to ICI. The machine learning model combining both radiomics features and pretreatment LDH resulted in better performance for response prediction compared to models that included radiomics or LDH alone (AUC of 0.89 (95% CI: [0.76–0.99]) vs. 0.81 (95% CI: [0.65–0.94]) and 0.81 (95% CI: [0.72–0.91]), respectively). Using SHAP analysis, LDH and two GLSZM were the most predictive of the outcome. Pre-treatment CT radiomic features performed equally well to serum LDH in predicting treatment response.

1. Introduction

The incidence of metastatic melanoma, which has a 5-year survival rate of 32%, is estimated to double every decade [1,2]. The survival benefit of targeted therapies as well as immunotherapy, including ipilimumab (CTLA-4 checkpoint inhibitor), nivolumab and pembrolizumab (PD-1 checkpoint inhibitors), has been proven [3]. The main limitations of the use of immune checkpoint inhibitors (ICI) in these patients include individual variations in treatment response and survival improvement, and the absence of objective, accurate predictive biomarkers of response [1,2,3].
Serum lactate dehydrogenase (LDH) is the main biomarker of response to immunotherapies; however [4,5,6], an increase of LDH serum level cannot differentiate pseudoprogression (defined as transient size increase followed by tumor response) from true progression. Therefore, therapeutic changes cannot be always initiated based on the changes in LDH serum level [7,8]. Identifying reliable predictive biomarkers for better monitoring of patients undergoing ICI therapy and identifying the patients likely to benefit from ICI therapy would be a major advance in pairing patients with beneficial therapies [9,10,11,12,13]. A recent study showed that melanoma tumors that have high imaging heterogeneity exhibited varied treatment responses and decreased survival [14,15]. Currently, changes in therapy are made according to imaging size-based response assessment, RECIST 1.1 (response evaluation criteria in solid tumors), and on follow-up CT examinations [16,17]. Interpretations of most malignant tumors with heterogenous growth and response patterns remains an issue of clinical decision, especially because of the limitations of RECIST 1.1 [18,19,20,21]. Intra-tumoral heterogeneity and tumor infiltration are newer imaging features that can be spatially assessed on imaging using quantitative measures of image signal heterogeneity and may improve prognostication compared with size assessment alone [22,23]. In routine practice iRECIST criteria, defined according to immune-related response criteria, have been introduced to partially overcome the heterogenous response problem and assess response to ICI therapy on contrast-enhanced CT scans, although these criteria have not been validated and atypical responses have been observed [23,24].
Several studies have evaluated the prognostic value of blood and imaging biomarkers, but the prognostic potential of combined imaging and blood markers has not been investigated in metastatic melanoma [11,25,26,27,28]. We hypothesized that baseline CT-based quantitative radiomics solely and combined with serum LDH can be predictive of responses to immunotherapy. In this study, we assessed the role of machine learning-based radiomics in predicting responses to ICI and in complementing LDH for prognostication of metastatic melanoma.

2. Materials and Methods

This study was approved by the Massachusetts General Brigham Institutional Review Board (IRB), and was conducted in accordance with the Health Insurance Portability and Accountability Act (HIPAA) guidelines for research. The requirement for patient informed consent was waived by the Massachusetts General Brigham IRB. The study was performed in accordance with the Declaration of Helsinki.

2.1. Patients and Clinical Data

Abdominal CT studies over a 14-year period (January 2008–January 2022) were reviewed in a retrospective manner for the presence of secondary malignant hepatic tumors from metastatic melanoma in the 1-month period prior to initiation of ICI. Clinical data collected at follow-up included medical and surgical interventions, and disease response to treatment (clinical outcomes). Inclusion criteria consisted of (a) age greater ≥18 years, (b) histopathology-confirmed melanoma cancer, (c) presence of biopsy-proven liver metastases on the arterial phase images of abdomen CT performed within 1 months prior to starting treatment, (d) available pre-immunotherapy serum LDH level, and (e) patients treated by an oncologist in our institution’s Cancer Center to ensure the availability of a response to immunotherapy and follow-up data. The presence of a response to immunotherapy was defined using RECIST version 1.1. At 3 months post-treatment evaluation, treatment response was defined by stable disease, and partial or complete response. According to RECIST criteria, the absence of a treatment response was defined by progression. Patients were then designated to 1 of the 2 groups: (1) Responders and (2) Non-responders [17].
Seventy-nine eligible patients (F: M 28:51; mean age 62 ± 17 years; range, 27–90 years) with 168 liver metastases were identified. None of the patients received treatment (surgery, immunotherapy, or targeted therapies) for their liver metastases before CT examinations.
Clinicopathological data included patient age, patient gender, pre-immunotherapy LDH level, tumor NRAS and BRAF mutation status, primary site of disease, tumor staging, RECIST data, and ICI type received, and this data were obtained from the electronic clinical and histological records by the study coordinator who was blinded to the radiological data.

2.2. CT Acquisition

All patients underwent a contrast enhanced abdominal exam on a 64-row (Discovery HD 750; GE Healthcare, Chicago, IL, USA) or 128-row (Somatom Definition Flash, Siemens Healthineer, Erlangen, Germany) multidetector CT scanner. A volume of 2 mL/kg of non-ionic contrast material (iomeprol, 350 mg iodine/mL; Iomeron 350, Bracco) was injected into an antecubital vein at a flow rate of 3.5 mL/s, followed by 50 mL of saline solution at the same flow rate. Acquisition parameters were as follows: tube voltage: 120 and 100 kVp; automatic tube current modulation: 64 × 1.25 mm and 128 × 1.25 mm; slice thickness: 5 mm.

2.3. Segmentation and Image Processing

Images were reviewed on the Picture Achieving and Communication System (PACS) by a radiologist, blinded to all clinical information, to identify metastatic lesions >1 cm in greatest diameter. From one to nine target metastases for each patient were selected following RECIST criteria. A volumetric image analysis software (3D slicer; http://www.slicer.org, accessed on 14 October 2022) was used to perform manual delineations and volume-based hepatic lesion segmentation on DICOM images [29] by a diagnostic radiology instructor (AT, 6 years of experience in image segmentation), on the original arterial phase contrast-enhanced CT images. An experienced fellowship-trained radiologist reviewed the segmentations (DD, 8 years of experience).

2.4. Extraction of Radiomic Features and Machine Learning Model Training

The 116 radiomic features, including first-order, 2D and 3D shape features, gray level co-occurrence matrix (GLCM) features, gray level size zone matrix (GLSZM), gray level run time length matrix (GLRLM) features, neighboring gray tone difference matrix (NGTDM) features, and gray level dependence matrix features (GLDM), were extracted from 168 hepatic metastatic lesions [30,31]. The image analysis pipeline is summarized in Figure 1.
A random forest machine learning algorithm using a selected subset of all radiomics features was used to predict the tumor response to immunotherapy. Maximum relevance minimum redundancy (mRMR) was used to select features that are both correlated with the outcome and non-redundant with other features in the set. Features with over 90% missing values were removed. The dataset was divided into a training, validation, and testing set at a ratio of 3:1:1. To avoid bias from excluding images with missing feature values, optimal imputation [32] was used to impute missing values. Feature sets of sizes 1 to the total number of features were selected with mRMR on the training data, and for each feature set, a new random forest model was developed using training data and evaluated using validation data. To balance the data for model development, synthetic cases of the minority class were generated using ADASYN [33]. The model that exhibited the best performance utilized fifteen features. The square root of the number of features indicated the depth of the random forest algorithm. Area under the receiver operating characteristic (AUC) analysis, confusion matrix metrics, and Brier score were used to assess and compare classification performance. The Brier score measures the mean of the squared difference between the predicted score and the occurrence of response to ICI. A lower Brier score indicates a more accurate model [34]. Youden’s index was used to select an optimal threshold for generating a confusion matrix [35].
Logistic regression analysis was used for the LDH model. Additionally, a combined machine learning algorithm based on all radiomics features and pre-treatment LDH was used to predict the response to immunotherapy.
To understand which of the selected features most strongly influenced model predictions, Shapley additive explanations (SHAP) was applied with our model. SHAP is a game-theoretic approach to interpreting predictions of machine learning models and an extension of Shapley values, indicating the average marginal contribution of each feature over all combinations of features [36]. This allows for both local and global explanations of the model, indicating both feature importance and the direction of the relationship between a feature and model output. Python 3.8 (https://www.python.org/, accessed on 1 February 2023) was used to perform the statistical analyses and machine learning models.

3. Results

3.1. Patients and Tumor Characteristics

From 2008 to 2022, ICI were initiated in 303 patients with metastatic melanoma. Of these patients, 113 underwent contrast-enhanced abdomen CT within 3 months prior to start of the ICI. The final population who met our inclusion criteria and had hepatic metastatic disease was comprised 79 patients with 168 hepatic metastatic tumors. Figure 2 shows the study flowchart.
The patients’ characteristics are demonstrated in Table 1. Serum LDH level was elevated (>the normal upper limit i.e., 248 UI/L) in 36/79 patients (45.5%).
A total of 168 metastases were analyzed. The mean number of target metastases was 2.1 [1–9]. The average sum of lesions diameters was 24.5 [4.4–126.1] mm on pre-treatment CT. Disease progression was observed in 23/79 patients (29%) at three months after initiation of ICI.
The most common driver mutation in our cohort was BRAF, which was found in 23 patients (29%). Other relevant mutations included NRAS in 8 patients (10%). The majority of patients had cutaneous melanoma (72%), but several patients (28%) had other histologic subtypes of melanoma.

3.2. ICI Response Kinetics for the Whole Population

Progression of disease was assessed for each tumor on follow-up CT at 3 months following treatment initiation using RECIST criteria. Of the 168 tumors, 100 (59.4%) were treated with ipilimumab, 30 (17.7%) with pembrozilumab, 25 (15%) with nivolumab, and 13 (2.8%) with a combination of ipilimumab and nivolumab as the first line treatment. All of the patients had an LDH assessment taken prior to start of ICI. The results showed that in non-responders, LDH levels were significantly higher when compared to the responding patients (p <0.01). This was in agreement with prior publications [18,19,23,24].
One hundred and twelve tumors (66.6%) responded to the ICI at 3 months follow-up (responding tumors) and 56 (33.3%) showed no response to ICI (progressing tumors).

3.3. Comparison of Baseline Characteristics between Responding and Progressing Tumors

Of the 112 responding tumors, associated with 56 patients, 48 (42.6%) were treated with ipilimumab, 26 (23%) with pembrozilumab, 16 (14.6%) with nivolumab, and 22 (31.8%) with other ICI treatments. Fifty percent of patients (28/56) had multiple (>1) liver tumors. The volume (mean: 20.4 ± 13 [4.1–92] mL, p = 0.022) and maximum dimension (mean: 22 ± 14 [4.5–107] cm, p = 0.022) of the segmented tumors were significantly higher in responding tumors. The target lesion number ranged from 1 to 7. Although not significant, patients who failed to respond to ICI tended to be older than the responders (mean age 60 ± 15 [27–90] years vs. 63 ± 15 [22–89] years, p = 0.27).
Of the 56 tumors associated with 23 patients that failed to respond to immunotherapy, 28 (50%) received ipilimumab, 13 (23.2%) nivolumab, 11 (19.6%) pembrozilumab, and 4 (7.2%) received other ICI treatments. The volume and maximum dimension of the segmented tumors was 37.3 ± 75 [4.4–562] mL and 29 ± 26 [4.5–126] in non-responding tumors, respectively. Eleven patients (48%) had >1 liver tumors. The target lesion number ranged from 1 to 9. Based on the tumor volume and number of liver lesions, the average overall liver disease burden was 0.013 ± 0.008 [0.002–0.06] in responders and 0.024 ± 0.05 [0.002–0.37] in non-responders.
Serum LDH levels were significantly lower in responders when compared to non-responders (mean 318 ± 214 [108–1242] UI/L vs. 914 ± 672 [160–2751] UI/L, p = 0.01). Fifty percent of responders were females, while most of those with progressing tumors (78%) were male (p = 0.01) (Table 2).

3.4. Predictive Power of Individual LDH, Radiomic and Combined Models

We included all radiomic features in the machine learning pipeline to predict treatment response in the study population. The dataset was divided into training (n = 100 tumors), validation (n = 34), and testing (n = 34 tumors) sets, with tumors from the same patients in the same set. Missing values were imputed separately in each set, and features were selected with mRMR. mRMR resulted in the selection of fifteen radiomics features, including 4 GLSZM and 4 GLDM, 3 GLCM, and 3 first-order and 1 shape, as shown in Figure 3A,B.
In the testing dataset, a random forest-based machine learning algorithm that included the above radiomic features yielded an AUC of 0.81 (95% CI: 0.65–0.94) (Figure 4A). Confusion matrix analysis revealed a sensitivity of 58%, a specificity of 91%, and an accuracy of 76%.
The logistic regression model based on the pretreatment LDH alone resulted in an AUC of 0.81 (95% CI: 0.72–0.91) in the testing dataset (Figure 4B) with a sensitivity of 70%, a specificity of 87%, and an accuracy of 76%. The machine learning model that included the combination of the selected radiomic features and the pretreatment LDH level achieved higher performance on all performance metrics including AUC, sensitivity, specificity, and accuracy measures, with an AUC of 0.89 (95% CI: 0.76–0.99), sensitivity of 75%, a specificity of 95%, and an accuracy of 85% in the testing dataset (Figure 4C) (Table 3).
The Brier score for the combined model was 0.121, lower than the models that include radiomics features alone (0.174) or LDH alone (0.155), indicating the higher accuracy of the model that combines both the radiomics features and LDH. Brier scores for the three models were obtained using the same independent test set.

3.5. Radiomics-only and Combined Machine Learning Models Interpretation

In the radiomics-only model, the most important feature as determined by SHAP was the Diagnostic Image Maximum, which is a first order statistic. For the combined model, in addition to LDH, the most important feature was GLSZM large area emphasis. This feature is a second order statistic that reports the quantity of homogeneous connected areas inside the volume, of a certain size and intensity, thus describing heterogeneity in the region of interest. A gray level zone is defined as the amount of connected voxels that share the same gray level intensity within the image region defined by the mask. The Large Area Emphasis feature calculates the maximum value in the normalized segmented image array. All features in order of importance, are pictured in the Shapley summary plot in Figure 5A,B, respectively.

4. Discussion

Our results illustrate that an explainable machine learning model that includes pre-treatment radiomic features and pre-treatment LDH provides better performance (AUC of 0.89) for predicting treatment responses to ICI. Additionally, our study has shed light on the independent predictive value of pre-treatment LDH and CT-based radiomics for early response to treatment according to RECIST criteria in patients with metastatic melanoma undergoing immunotherapy.
Despite multiple previous translational and clinical research studies, there is still a critical need for more valid blood and imaging biomarkers for real-time assessments of responses to ICI, because once metastatic tumors have developed resistance to ICI by bypassing the host’s immune system control, the disease progression can be rapid.
For immunotherapy, prior studies have reported that elevated baseline LDH serum levels are correlated with poor prognosis and ICI response rates [5,37]. This is in agreement with our results: patients who responded to ICI had lower levels of LDH before initiation of the immunotherapy compared to non-responding patients (p = 0.00). However, there is evidence showing that the early response to ICI therapy and overall survival have not always been correlated with baseline serum LDH levels; this finding potentially broadens treatment accessibility to patients excluded on the basis of marker levels at baseline. Additional imaging biomarkers that can identify real-time changes in the disease status and predict the response to ICI for active monitoring are thus needed to allow for timely and personalized therapeutic adjustments.
Radiomic features analysis can provide voxel-scale information about the compositional distribution profile within a tumor beyond anatomic size. In this context, several studies have highlighted the application of radiomics analysis to evaluate the efficacy and successful prediction of response to immunotherapy in variety of malignancies [38,39]. A recent study in patients with advanced NSCLC who were treated with ICI presented an approach that evaluates dynamic changes in specific tumor radiophenotypic attributes between baseline and post-treatment CT scans [40]. In addition, a few studies have extracted imaging biomarkers to assess the correlation between pretreatment CT texture parameters and survival prediction in patients with metastatic melanoma who received anti-PD-1 monoclonal antibodies [41,42,43,44]. Study findings by Trebeschi et al. on a small melanoma cohort proved an association between tumor textural radiomic patterns and responses to immunotherapy [45]. Durot et al. proposed that tumor skewness extracted from pre-treatment CT using texture analysis has been reported as a possible predictive biomarker of overall survival in metastatic melanoma patients who underwent anti-PD1 monoclonal antibody therapy [46].
In a study by Sun et al., an MRI-based radiomics model correlated with response to immunotherapy was developed and validated on 3 independent cohorts with diverse image acquisition parameters [47]. Both serum biomarkers and imaging findings are routinely investigated in patients with metastatic melanoma [48], but the combined predictive power of these markers is seldom investigated. To our knowledge, this is the first study that highlights the independent predictive value of combined LDH and machine learning-based radiomics in the assessment of response to ICI in these patients. Our analysis shows that such a multidisciplinary and multifaceted approach may have the potential to improve the treatment strategies and prognostic performance.
This study has a number of limitations, including the relatively small number of patients and the lack of multi-institutional validation. Additionally, the reproducibility of texture analysis has been commonly problematic in the field of radiomics. The imaging data was normalized prior to analysis. Although separate training, testing, and validation sets were used, the study was trained, tested, and validated in a study population from the same institution. We seek to increase the diversity and volume of training and testing data and separately verify our model, prospectively and across multiple sites.
In the next phase of this study, we will prospectively (1) review the CT images at two time points (before and 6–8 weeks after the start of ICI) to assess dynamic imaging biomarker change during treatment, (2) validate the results using an independent external dataset across multiple sites, and (3) reduce the inter-reader reliability by including >2 readers and automated segmentation to ensure the ability of producing the same results on a larger patient population, other scanners, and different image acquisition protocols.

5. Conclusions

This study identifies key radiomic features from a single pre-treatment CT scan that predict response to ICI in patients with metastatic melanoma using a machine learning-based model. The radiomics-only model performed with similar predictability as the model based solely on pretreatment serum LDH levels. Importantly, we showed that the combined machine learning-based model outperformed the models based on radiomic features and pretreatment serum LDH levels alone, and, therefore, has the potential to further optimize clinical practice through more precise and individualized treatment decisions.

Author Contributions

Conceptualization, A.T. and D.D.; methodology, A.T., B.D., M.C. and D.D.; software, A.T. and B.D.; validation, A.T., B.D., M.S.G., G.B. and D.D.; formal analysis, A.T., B.D., M.C. and D.D.; investigation, A.T. and D.D.; resources, A.T. and D.D.; data curation, A.T., B.D., A.M. and D.D.; writing—original draft preparation, A.T. and D.D.; writing—review and editing, A.T., B.D., H.D., G.B., M.S.G. and D.D.; visualization, A.T. and D.D.; supervision, A.T. and D.D.; project administration, A.T. and H.D.; funding acquisition, D.D. All authors have read and agreed to the published version of the manuscript.

Funding

RSNA Seed grant: 2021.

Institutional Review Board Statement

This retrospective study was approved by the IRB (protocol #2017P000043). The affiliation: Mass general Brigham (MGB).

Informed Consent Statement

The requirement for informed consent was waived by the IRB.

Data Availability Statement

The data presented in this study are available on request from the corresponding author.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. National Cancer Institute Surveillance Epidemiology and End Results Program. Cancer Statistics Cancer Stat Facts: Melanoma of the Skin. 2014. Available online: https://seer.cancer.gov/statfacts/html/melan.html (accessed on 11 March 2018).
  2. Available online: https://www.cancer.org/cancer/melanoma-skin-cancer/detection-diagnosis-staging/survival-rates-for-melanoma-skin-cancer-by-stage.html (accessed on 13 March 2023).
  3. Wolchok, J.D.; Chiarion-Sileni, V.; Gonzalez, R.; Rutkowski, P.; Grob, J.J.; Cowey, C.L.; Lao, C.D.; Wagstaff, J.; Schadendorf, D.; Ferrucci, P.F.; et al. Overall survival with combined nivolumab and ipilimumab in advanced melanoma. N. Engl. J. Med. 2017, 377, 1345–1356. [Google Scholar] [PubMed]
  4. Pelizzari, G.; Basile, D.; Zago, S.; Lisanti, C.; Bartoletti, M.; Bortot, L.; Vitale, M.G.; Fanotto, V.; Barban, S.; Cinausero, M.; et al. Lactate Dehydrogenase (LDH) Response to First-Line Treatment Predicts Survival in Metastatic Breast Cancer: First Clues for a Cost-Effective and Dynamic Biomarker. Cancers 2019, 11, 1243. [Google Scholar] [CrossRef]
  5. Diem, S.; Kasenda, B.; Spain, L.; Martin-Liberal, J.; Marconcini, R.; Gore, M.; Larkin, J. Serum lactate dehydrogenase as an early marker for outcome in patients treated with anti-PD-1 therapy in metastatic melanoma. Br. J. Cancer 2016, 114, 256–261. [Google Scholar] [CrossRef]
  6. Van Wilpe, S.; Koornstra, R.; Den Brok, M.; De Groot, J.W.; Blank, C.; De Vries, J.; Gerritsen, W.; Mehra, N. Lactate dehydrogenase: A marker of diminished antitumor immunity. Oncoimmunology 2020, 9, 1731942. [Google Scholar] [CrossRef] [PubMed]
  7. Ma, Y.; Wang, Q.; Dong, Q.; Zhan, L.; Zhang, J. How to differentiate pseudoprogression from true progression in cancer patients treated with immunotherapy. Am. J. Cancer Res. 2019, 9, 1546–1553. [Google Scholar] [PubMed]
  8. Jia, W.; Gao, Q.; Han, A.; Zhu, H.; Yu, J. The potential mechanism, recognition and clinical significance of tumor pseudoprogression after immunotherapy. Cancer Biol. Med. 2019, 16, 655–670. [Google Scholar] [CrossRef] [PubMed]
  9. Dromain, C.; Beigelman, C.; Pozzessere, C.; Duran, R.; Digklia, A. Imaging of tumour response to immunotherapy. Eur. Radiol. Exp. 2020, 4, 1–15. [Google Scholar] [CrossRef]
  10. Garutti, M.; Bonin, S.; Buriolla, S.; Bertoli, E.; Pizzichetta, M.A.; Zalaudek, I.; Puglisi, F. Find the Flame: Predictive Biomarkers for Im-munotherapy in Melanoma. Cancers 2021, 13, 1819. [Google Scholar] [CrossRef] [PubMed]
  11. Rizk, E.M.; Seffens, A.M.; Trager, M.H.; Moore, M.R.; Geskin, L.J.; Gartrell-Corrado, R.D.; Wong, W.; Saenger, Y.M. Biomarkers Predictive of Survival and Response to Immune Checkpoint Inhibitors in Melanoma. Am. J. Clin. Dermatol. 2019, 21, 1–11. [Google Scholar] [CrossRef]
  12. Bai, R.; Lv, Z.; Xu, D.; Cui, J. Predictive biomarkers for cancer immunotherapy with immune checkpoint inhibitors. Biomark. Res. 2020, 8, 1–17. [Google Scholar] [CrossRef]
  13. Darvin, P.; Toor, S.M.; Sasidharan Nair, V.; Elkord, E. Immune checkpoint inhibitors: Recent progress and potential biomarkers. Exp. Mol. Med. 2018, 50, 1–11. [Google Scholar] [CrossRef]
  14. Lin, Z.; Meng, X.; Wen, J.; Corral, J.M.; Andreev, D.; Kachler, K.; Schett, G.; Chen, X.; Bozec, A. Intratumor Heterogeneity Correlates with Reduced Immune Activity and Worse Survival in Melanoma Patients. Front. Oncol. 2020, 10, 596493. [Google Scholar] [CrossRef] [PubMed]
  15. Rauwerdink, D.J.W.; Molina, G.; Frederick, D.T.; Sharova, T.; van der Hage, J.; Cohen, S.; Boland, G.M. Mixed Response to Immunotherapy in Patients with Metastatic Melanoma. Ann. Surg. Oncol. 2020, 27, 3488–3497. [Google Scholar] [CrossRef] [PubMed]
  16. Nishino, M.; Jackman, D.M.; Hatabu, H.; Yeap, B.Y.; Cioffredi, L.-A.; Yap, J.T.; Jänne, P.A.; Johnson, B.E.; Abbeele, A.D.V.D. New Response Evaluation Criteria in Solid Tumors (RECIST) Guidelines for Advanced Non–Small Cell Lung Cancer: Comparison with Original RECIST and Impact on Assessment of Tumor Response to Targeted Therapy. Am. J. Roentgenol. 2010, 195, W221–W228. [Google Scholar] [CrossRef]
  17. Schwartz, L.H.; Litière, S.; de Vries, E.; Ford, R.; Gwyther, S.; Mandrekar, S.; Shankar, L.; Bogaerts, J.; Chen, A.; Dancey, J.; et al. RECIST 1.1-Update and clarification: From the RECIST committee. Eur. J. Cancer 2016, 62, 132–137. [Google Scholar] [CrossRef]
  18. Subbiah, V.; Chuang, H.H.; Gambhire, D.; Kairemo, K. Defining Clinical Response Criteria and Early Response Criteria for Precision Oncology: Current State-of-the-Art and Future Perspectives. Diagnostics 2017, 7, 10. [Google Scholar] [CrossRef] [PubMed]
  19. Aykan, N.F.; Özatlı, T. Objective response rate assessment in oncology: Current situation and future expectations. World J. Clin. Oncol. 2020, 11, 53–73. [Google Scholar] [CrossRef] [PubMed]
  20. Ko, C.-C.; Yeh, L.-R.; Kuo, Y.-T.; Chen, J.-H. Imaging biomarkers for evaluating tumor response: RECIST and beyond. Biomark. Res. 2021, 9, 1–20. [Google Scholar] [CrossRef]
  21. Radziuviene, G.; Rasmusson, A.; Augulis, R.; Grineviciute, R.B.; Zilenaite, D.; Laurinaviciene, A.; Ostapenko, V.; Laurinavicius, A. In-tratumoral Heterogeneity and Immune Response Indicators to Predict Overall Survival in a Retrospective Study of HER2-Borderline (IHC 2+) Breast Cancer Patients. Front Oncol. 2021, 11, 774088. [Google Scholar] [PubMed]
  22. Yoshimura, K.; Tsujikawa, T.; Mitsuda, J.; Ogi, H.; Saburi, S.; Ohmura, G.; Arai, A.; Shibata, S.; Thibault, G.; Chang, Y.H.; et al. Spatial Profiles of Intratumoral PD-1+ Helper T Cells Predict Prognosis in Head and Neck Squamous Cell Carcinoma. Front. Immunol. 2021, 12, 4554. [Google Scholar] [CrossRef] [PubMed]
  23. Laurinavicius, A.; Rasmusson, A.; Plancoulaine, B.; Shribak, M.; Levenson, R. Machine-Learning–Based Evaluation of Intratumoral Heterogeneity and Tumor-Stroma Interface for Clinical Guidance. Am. J. Pathol. 2021, 191, 1724–1731. [Google Scholar] [CrossRef] [PubMed]
  24. Seymour, L.; Bogaerts, J.; Perrone, A.; Ford, R.; Schwartz, L.H.; Mandrekar, S.; Lin, N.U.; Litière, S.; Dancey, J.; Chen, A.; et al. iRECIST: Guidelines for response criteria for use in trials testing immunotherapeutics. Lancet Oncol. 2017, 18, e143–e152, Erratum in Lancet Oncol. 2017, 18, e143–e152. [Google Scholar] [CrossRef]
  25. Weide, B.; Martens, A.; Hassel, J.C.; Berking, C.; Postow, M.A.; Bisschop, K.; Simeone, E.; Mangana, J.; Schilling, B.; Di Giacomo, A.M.; et al. Baseline Biomarkers for Outcome of Melanoma Patients Treated with Pembrolizumab. Clin. Cancer Res. 2016, 22, 5487–5496. [Google Scholar] [CrossRef] [PubMed]
  26. Walenta, S.; Mueller-Klieser, W.F. Lactate: Mirror and motor of tumor malignancy. Semin. Radiat. Oncol. 2004, 14, 267–274. [Google Scholar] [CrossRef] [PubMed]
  27. Suh, S.-Y.; Ahn, H. Lactate dehydrogenase as a prognostic factor for survival time of terminally ill cancer patients: A preliminary study. Eur. J. Cancer 2007, 43, 1051–1059. [Google Scholar] [CrossRef] [PubMed]
  28. Burgermeister, S.; Gabryś, H.S.; Basler, L.; Hogan, S.A.; Pavic, M.; Bogowicz, M.; Martínez Gómez, J.M.; Vuong, D.; Tanadini-Lang, S.; Foerster, R.; et al. Improved Survival Prediction by Combining Radiological Imaging and S-100B Levels into a Multivariate Model in Metastatic Melanoma Patients Treated with Immune Checkpoint In-hibition. Front Oncol. 2022, 12, 830627. [Google Scholar] [CrossRef] [PubMed]
  29. Fedorov, A.; Beichel, R.; Kalpathy-Cramer, J.; Finet, J.; Fillion-Robin, J.-C.; Pujol, S.; Bauer, C.; Jennings, D.; Fennessy, F.; Sonka, M.; et al. 3D Slicer as an image computing platform for the Quantitative Imaging Network. Magn. Reson. Imaging 2012, 30, 1323–1341. [Google Scholar] [CrossRef] [PubMed]
  30. Van Griethuysen, J.J.M.; Fedorov, A.; Parmar, C.; Hosny, A.; Aucoin, N.; Narayan, V.; Beets-Tan, R.G.H.; Fillion-Robin, J.-C.; Pieper, S. Aerts HJWL: Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017, 77, e104–e107. [Google Scholar] [CrossRef] [PubMed]
  31. Daye, D.; Tabari, A.; Kim, H.; Chang, K.; Kamran, S.C.; Hong, T.S.; Kalpathy-Cramer, J.; Gee, M.S. Quantitative tumor heterogeneity MRI profiling improves machine learning–based prognostication in patients with metastatic colon cancer. Eur. Radiol. 2021, 31, 5759–5767. [Google Scholar] [CrossRef] [PubMed]
  32. Bertsimas, D.; Pawlowski, C.; Zhuo, Y.D. From predictive methods to missing data imputation: An optimization approach. J. Mach. Learn. Res. 2017, 18, 7133–7171. [Google Scholar]
  33. He, H.; Bai, Y.; Garcia, E.A.; Li, S. ADASYN: Adaptive synthetic sampling approach for imbalanced learning. In Proceeding of the IEEE International Joint Conference on Neural Networks, Hong Kong, China, 1–8 June 2008; pp. 1322–1328. [Google Scholar] [CrossRef]
  34. Available online: https://statisticaloddsandends.wordpress.com/2019/12/29/what-is-a-brier-score/ (accessed on 13 March 2023).
  35. Schisterman, E.; Faraggi, D.; Reiser, B.; Hu, J. Youden Index and the optimal threshold for markers with mass at zero. Stat. Med. 2007, 27, 297–315. [Google Scholar] [CrossRef] [PubMed]
  36. Rodríguez-Pérez, R.; Bajorath, J. Interpretation of machine learning models using shapley values: Application to compound potency and multi-target activity predictions. J. Comput. Mol. Des. 2020, 34, 1013–1026. [Google Scholar] [CrossRef] [PubMed]
  37. Nosrati, A.; Tsai, K.K.; Goldinger, S.M.; Tumeh, P.; Grimes, B.; Loo, K.; Algazi, A.P.; Nguyen-Kim, T.D.L.; Levesque, M.; Dummer, R.; et al. Evaluation of clinicopathological factors in PD-1 response: Derivation and validation of a prediction scale for response to PD-1 monotherapy. Br. J. Cancer 2017, 116, 1141–1147. [Google Scholar] [CrossRef] [PubMed]
  38. Park, B.W.; Kim, J.K.; Heo, C.; Park, K.J. Reliability of CT radiomic features reflecting tumour heterogeneity according to image quality and image processing parameters. Sci. Rep. 2020, 10, 3852. [Google Scholar] [CrossRef] [PubMed]
  39. Deng, W.; Gopal, Y.N.; Scott, A.; Chen, G.; Woodman, S.E.; Davies, M.A. Role and therapeutic potential of PI3K-mTOR sig-naling in de novo resistance to BRAF inhibition. Pigment. Cell Melanoma Res. 2012, 25, 248–258. [Google Scholar] [CrossRef] [PubMed]
  40. Khorrami, M.; Prasanna, P.; Gupta, A.; Patil, P.; Velu, P.D.; Thawani, R.; Corredor, G.; Alilou, M.; Bera, K.; Fu, P.; et al. Changes in CT Radiomic Features Associated with Lymphocyte Distribution Predict Overall Survival and Response to Immunotherapy in Non–Small Cell Lung Cancer. Cancer Immunol. Res. 2020, 8, 108–119. [Google Scholar] [CrossRef] [PubMed]
  41. Guerrisi, A.; Russillo, M.; Loi, E.; Ganeshan, B.; Ungania, S.; Desiderio, F.; Bruzzaniti, V.; Falcone, I.; Renna, D.; Ferraresi, V.; et al. Exploring CT Texture Parameters as Predictive and Response Imaging Biomarkers of Survival in Patients with Metastatic Melanoma Treated With PD-1 Inhibitor Nivolumab: A Pilot Study Using a Delta-Radiomics Approach. Front. Oncol. 2021, 11. [Google Scholar] [CrossRef]
  42. Peisen, F.; Hänsch, A.; Hering, A.; Brendlin, A.S.; Afat, S.; Nikolaou, K.; Gatidis, S.; Eigentler, T.; Amaral, T.; Moltz, J.H.; et al. Combination of Whole-Body Baseline CT Radiomics and Clinical Parameters to Predict Response and Survival in a Stage-IV Melanoma Cohort Undergoing Immunotherapy. Cancers 2022, 14, 2992. [Google Scholar] [CrossRef] [PubMed]
  43. Bonnin, A.; Durot, C.; Barat, M.; Djelouah, M.; Grange, F.; Mulé, S.; Soyer, P.; Hoeffel, C. CT texture analysis as a predictor of favorable response to anti-PD1 monoclonal antibodies in metastatic skin melanoma. Diagn. Interv. Imaging 2022, 103, 97–102. [Google Scholar] [CrossRef] [PubMed]
  44. Wang, Z.-L.; Mao, L.-L.; Zhou, Z.-G.; Si, L.; Zhu, H.-T.; Chen, X.; Zhou, M.-J.; Sun, Y.-S.; Guo, J. Pilot Study of CT-Based Radiomics Model for Early Evaluation of Response to Immunotherapy in Patients with Metastatic Melanoma. Front. Oncol. 2020, 10. [Google Scholar] [CrossRef] [PubMed]
  45. Trebeschi, S.; Drago, S.G.; Birkbak, N.J.; Kurilova, I.; Cǎlin, A.M.; Pizzi, A.D.; Lalezari, F.; Lambregts, D.M.J.; Rohaan, M.W.; Parmar, C.; et al. Predicting response to cancer immunotherapy using non-invasive radiomic biomarkers. Ann. Oncol. 2019, 30, 998–1004. [Google Scholar] [CrossRef] [PubMed]
  46. Durot, C.; Mulé, S.; Soyer, P.; Marchal, A.; Grange, F.; Hoeffel, C. Metastatic melanoma: Pretreatment contrast-enhanced CT texture parameters as predictive biomarkers of survival in patients treated with pembrolizumab. Eur. Radiol. 2019, 29, 3183–3191. [Google Scholar] [CrossRef] [PubMed]
  47. Sun, R.; Limkin, E.J.; Vakalopoulou, M.; Dercle, L.; Champiat, S.; Han, S.R.; Verlingue, L.; Brandao, D.; Lancia, A.; Ammari, S.; et al. A radiomics approach to assess tumour-infiltrating CD8 cells and response to an-ti-PD-1 or anti-PD-L1immunotherapy: An imaging biomarker, retrospective multicohort study. Lancet Oncol. 2018, 19, 1180–1191. [Google Scholar] [CrossRef] [PubMed]
  48. Madonna, G.; Masucci, G.V.; Capone, M.; Mallardo, D.; Grimaldi, A.M.; Simeone, E.; Vanella, V.; Festino, L.; Palla, M.; Scarpato, L.; et al. Clinical Categorization Algorithm (CLICAL) and Machine Learning Approach (SRF-CLICAL) to Predict Clinical Benefit to Immunotherapy in Metastatic Melanoma Patients: Real-World Evidence from the Istituto Nazionale Tumori IRCCS Fondazione Pascale, Napoli, Italy. Cancers 2021, 13, 4164. [Google Scholar] [CrossRef] [PubMed]
Cancers 15 02700 g001 550
Figure 1. The image analysis pipeline used in this study consisted of tumor segmentation, feature extraction, and random forest-based machine learning for ICI therapy response prediction. Contrast-enhanced CT images in the arterial phase were used for analysis.
Figure 1. The image analysis pipeline used in this study consisted of tumor segmentation, feature extraction, and random forest-based machine learning for ICI therapy response prediction. Contrast-enhanced CT images in the arterial phase were used for analysis.
Cancers 15 02700 g001
Cancers 15 02700 g002 550
Figure 2. Flow diagram of data set.
Figure 2. Flow diagram of data set.
Cancers 15 02700 g002
Cancers 15 02700 g003 550
Figure 3. Top 15 radiomic features determined by mRMR for (A) radiomics only and (B) combined models.
Figure 3. Top 15 radiomic features determined by mRMR for (A) radiomics only and (B) combined models.
Cancers 15 02700 g003
Cancers 15 02700 g004a 550Cancers 15 02700 g004b 550
Figure 4. Receiver operating characteristic curves of the models based on the (A) Pre-treatment LDH, (B) Radiomic features, and (C) Pre-treatment LDH + radiomics feature.
Figure 4. Receiver operating characteristic curves of the models based on the (A) Pre-treatment LDH, (B) Radiomic features, and (C) Pre-treatment LDH + radiomics feature.
Cancers 15 02700 g004aCancers 15 02700 g004b
Cancers 15 02700 g005 550
Figure 5. Shapley summary plot generated with Shapley additive explanation (SHAP) to illustrate the relative importance of each feature on (A) Radiomics-only and (B) combined model predictions.
Figure 5. Shapley summary plot generated with Shapley additive explanation (SHAP) to illustrate the relative importance of each feature on (A) Radiomics-only and (B) combined model predictions.
Cancers 15 02700 g005
Table
Table 1. Patient, tumor, and treatment characteristics.
Table 1. Patient, tumor, and treatment characteristics.
Variablesn (%)
Mean Age a (years ± SD)62.1 ± 17
Gender
 Male51 (64.6%)
 Female28 (35.4%)
NRAS mutation
 Positive8 (10%)
 Negative71 (90%)
BRAF V600 mutation
 Positive23 (29%)
 Negative56 (71%)
Primary Site of Disease
 Skin57 (72%)
 Other22 (28%)
LDH > 248 (UI/L)36 (45.5%)
Cancer Stage4
Number of Liver Lesions
 141 (52%)
 219 (24%)
 39 (11.3%)
 43 (3.7%)
 53 (3.7%)
 73 (3.7%)
 91 (1.6%)
ICI Type
 Ipilimumab47 (59.4%)
 Pembrozilumab14 (17.7%)
 Nivolumab12 (15%)
 Other ICI6 (7.9%)
RECIST status at three months
 Progression23 (29%)
 Stable, partial or complete response56 (71%)
a At the time of CT. ICI: immune checkpoint inhibitor; LDH: lactate dehydrogenase.
Table
Table 2. Demographics features for the responders and non-responders.
Table 2. Demographics features for the responders and non-responders.
VariableRespondersNon-Responders p Value
Mean ± SDMin–MaxMean ± SDMin–Max
Age (years)63 ± 1522–8960 ± 1529–900.27
Gender (M:F)28:2818:50.01
LDH (UI/L)318 ± 214108–1242914 ± 672160–27510.00
Maximum dimension (cm)22 ± 144.5–10729 ± 264.5–1260.027
Tumor volume (mL)20.4 ± 134.1–92.737.3 ± 754.4–5620.022
Table
Table 3. Performance metrics of the three models.
Table 3. Performance metrics of the three models.
MetricLDH onlyRadiomics OnlyCombined
Sensitivity0.700.580.75
Specificity0.870.910.95
Accuracy0.760.760.85
AUC0.81
CI: [0.72–0.91]		0.81
CI: [0.65–0.94]		0.89
CI: [0.76–0.99]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Tabari, A.; Cox, M.; D’Amore, B.; Mansur, A.; Dabbara, H.; Boland, G.; Gee, M.S.; Daye, D. Machine Learning Improves the Prediction of Responses to Immune Checkpoint Inhibitors in Metastatic Melanoma. Cancers 2023, 15, 2700. https://doi.org/10.3390/cancers15102700

AMA Style

Tabari A, Cox M, D’Amore B, Mansur A, Dabbara H, Boland G, Gee MS, Daye D. Machine Learning Improves the Prediction of Responses to Immune Checkpoint Inhibitors in Metastatic Melanoma. Cancers. 2023; 15(10):2700. https://doi.org/10.3390/cancers15102700

Chicago/Turabian Style

Tabari, Azadeh, Meredith Cox, Brian D’Amore, Arian Mansur, Harika Dabbara, Genevieve Boland, Michael S. Gee, and Dania Daye. 2023. "Machine Learning Improves the Prediction of Responses to Immune Checkpoint Inhibitors in Metastatic Melanoma" Cancers 15, no. 10: 2700. https://doi.org/10.3390/cancers15102700

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop