Review Reports - Development and Validation of Prognostic Models for Treatment Response of Patients with B-Cell Lymphoma: Standard Statistical and Machine-Learning Approaches

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

1. Add cutoff values for each indicator to Table 1, indicating the "high/low" category.
2. I would recommend that the authors provide an example of using the nomogram, perhaps under letter B next to the empty nomogram figure for clarity.
3. Also, carefully compare the values of the parameters included in the model in the low-risk, low intermediate, high intermediate, and high-risk groups. I recommend that the authors add such a table to the text of the manuscript, rather than to supplementary materials.
4. The model achieves an overall accuracy of no higher than 0.7, which is generally quite low. The authors do not discuss the obtained values or compare them with similar indicators for existing models; I believe this should be corrected. Even if there are no data on lymphomas, similar models for other cancer types can be examined and their prediction accuracy compared. Add this information to the Discussion section.
5. Which indicators, in your opinion, could improve the quality of the prediction if they could be included in the model?

Author Response

Dear reviewer,

We would like to thank you for your constructive comments on our manuscript titled “Development and validation of prognostic models for treatment response of patients with B-cell lymphoma: Standard statistical and machine-learning approaches.” Your feedback has been invaluable in improving the quality and clarity of our work. We have carefully addressed each of your comments and revised the manuscript to strengthen its rigour, transparency, and completeness. A point-by-point response to all recommendations and concerns is provided below.

Comments and response

Reviewer #1

Comment 1: Add cutoff values for each indicator to Table 1, indicating the "high/low" category.

Response 1: Thank you. We have added cutoff values obtained by the ROC curve for each indicator in the revised manuscript.

Comment 2: I would recommend that the authors provide an example of using the nomogram, perhaps under letter B next to the empty nomogram figure for clarity.

Response 2: Thank you for insightful suggestion. To improve clarity of the nomogram, we have added a personalized working example below Figure 4. Moreover, in the text, we provided instructions on how to assign points for each variable and calculate the total score from the nomogram (Page 14, Lines: 327-339).

Comment 3: Also, carefully compare the values of the parameters included in the model in the low-risk, low intermediate, high intermediate, and high-risk groups. I recommend that the authors add such a table to the text of the manuscript, rather than to supplementary materials.

Response 3: We appreciate the reviewer’s suggestion. In the original manuscript, Supplementary Table 7 presents treatment response rates across risk groups for our nomogram and existing prognostic models (IPI, R-IPI, NCCN-IPI). To enhance clarity and accessibility, we included this table in the main text as Table 3 in the revision manuscript. Moreover, to ensure balanced representation of sample across risk categories, we used quantiles instead of ROC coordinates to identify risk groups from the nomogram score.

Comment 4: The model achieves an overall accuracy of no higher than 0.7, which is generally quite low. The authors do not discuss the obtained values or compare them with similar indicators for existing models; I believe this should be corrected. Even if there are no data on lymphomas, similar models for other cancer types can be examined and their prediction accuracy compared. Add this information to the Discussion section.

Response 4: We appreciate the reviewer for the valuable comment. In general, AUC values in the range of 0.7-0.8 are considered as acceptable, 0.8-0.9 as good to excellent, and above 0.9 as outstanding. We have expanded the discussion by presenting the obtained AUC values and comparing our model’s performance with established prognostic indices (IPI, R-IPI, NCCN-IPI) and prior nomograms or models used in predicting lymphoma outcomes ((Page 19 & 20, Lines: 409-437).

Comment 5: Which indicators, in your opinion, could improve the quality of the prediction if they could be included in the model?

Response 5: Thank you. Based on the literature, we evaluated several inflammatory and nutritional indicators, including SII, SIRI, PNI, PLR, NLR, and MLR, for their association with treatment response. Univariable logistic regression showed that all these indicators were significantly associated with treatment response (Page 9, Lines: 268-271, Supplementary Table 3). Moreover, their individual discriminative ability, as reflected by ROC curve AUC values, was broadly comparable (Page 14, Lines: 257-260).

To develop final model, we followed established practices for feature selection in prediction model development in order to ensure methodological rigour and avoid overfitting. Based on our methodology, only SII was retained in the final model.

As the overall discriminatory power of our final models remains limited, in our opinion, incorporating additional molecular parameters (e.g., β2-microglobulin, FISH, MYC) and transcriptomic signatures could improve the quality of the prediction. We have suggested this (Page 23, Lines: 540-542).

Reviewer 2 Report

Comments and Suggestions for Authors

This study made a bioinformatics exercise to define prognostic values in non-Hodgkin lymphoma using a dataset from Australia. Overall, the relevance of the study for the clinical used remains unclear. The study is mixing different lymphoma subtypes, prognosis of follicular is very different from mantle cell lymphoma; please confirm if analysis should be stratified for each lymphoma subtype.

Additional comments:

(1) Please provide website link of teh LaRDR.

(2) Please describe which version of WHO classification was used.

(3) Please describe Lugano 2014 criteria.

(4) Apart from CR, was overall surival or progression-free survival evaluated?

(5) Please explain why BCL2 and BCL6 expression were considered as prognostic variables?

(6) Is it correct to mix DLBCL, FL, MCL, and Burkitt lymphoma? The survival of these lymphoma subtypes is quite different.

(7) Section 2.3. Please provide more details regarding the statistical analyses. What packages and what methods were used in R?

(8) Please describe the method to avoid averfitting and the logic behind it.

(9) Please describe Boruta algorithm.

(10) Please describe Gini index.

(11) Please describe ROSE and SMOTE techniques, also please cite appropriate references.

(12) In Figure 1. Please use same font in all sections.

(13) Is ROC the best technique to identify cutoff? The AUCs ranged from 0.574 to 0.555. Is this AUC good enough (it si pretty low)?

(14) Regarding Table 1. This is a univariate analysis. Could you please calculate a binary logistic regression, backward conditional? What variables were the most relevant?

(15) In figure 2. Does the Boruta algorithm rates the importance of the variables? Does Burata importance variable correlate with the hazard-risk or odds ratio of binary logistic regression?

(16) Regarding attributs (variables) of Figure 2. Did the variables correlate between them?

(17) Table 2. The AUCs and other performance parameters are low. What usefulness for clinical practice those values provide? Please confirm.

(18) Should lymphoma subtype be included in the nomogram?

(19) Please describe in the footnote the interpretation of Figure 5 so you may help potential readers.

Author Response

Dear reviewer,

Comments and responses

Reviewer 2:

Comment 1: Please provide website link of the LaRDR.

Response 1: Thank you. We have added the official website link of the LaRDR (https://lardr.org) in the Methods section.

Comment 2: Please describe which version of WHO classification was used.

Response 2: Thank you. We have provided version of WHO classification with citation in the Methods section (Page 4, Lines: 131-132).

Comment 3: Please describe Lugano 2014 criteria.

Response 3: Thank you. We have briefly described Lugano 2014 criteria in the revised manuscript (Page 5, Lines: 138-144).

Comment 4: Apart from CR, was overall survival or progression-free survival evaluated?

Response 4: Thank you for this insightful comment. We did not evaluate Overall survival (OS) and progression-free survival (PFS) in this study. We understand that OS and PFS are key endpoints in lymphoma care. Existing valuable prognostic tools, including IPI, R-IPI and NCCN-IPI, are specifically designed to predict OS and PFS. Thus, in this study, our primary focus was on predicting treatment response to first-line therapy, as predicting the likelihood of complete response before initiating therapy provides an early and actionable indicator of treatment effectiveness. We aimed to develop and validate a new model, as well as test the performance of existing tools in predicting treatment response.

Comment 5: Please explain why BCL2 and BCL6 expression were considered as prognostic variables?

Response 5: Thank you for the query. Studies have established the prognostic value of BCL2 and BCL6 expression in B-cell lymphomas, particularly in diffuse large B-cell lymphoma (DLBCL). Consistent with previous studies, our study found that BCL2 expression was significantly associated with a higher rate of incomplete treatment response (as discussed in Page 22, lines: 503-516), whereas BCL6 did not show consistent predictive value and was not retained in the final model. Thus, these markers were considered based on their prognostic value and availability in our dataset.

Comment 6: Is it correct to mix DLBCL, FL, MCL, and Burkitt lymphoma? The survival of these lymphoma subtypes is quite different.

Response 6: Thank you for bringing this key concern to our attention. We agree that these subtypes exhibit distinct survival outcomes. However, our analysis showed that lymphoma subtype was not a significant predictor of treatment response in our cohort, suggesting that the included features may capture common determinants of response. To evaluate the performance of our model, which was developed regardless of subtypes, we assessed its performance specifically within the DLBCL cohort, and our model offers slightly improved discriminative ability within this subtype (Page 17&18, lines: 365-369). To address this issue further, we have proposed evaluating model performance within rare subtypes in future work, using larger cohorts (Page 20&21, Lines: 465-470). Moreover, we acknowledged that the predominance of DLBCL cases may influence the model’s generalizability (Page 22, Lines: 521-525).

Comment 7: Section 2.3. Please provide more details regarding the statistical analyses. What packages and what methods were used in R?

Response 7: We thank the reviewer for this suggestion. We have provided details of the R packages in Sections 2.3, 2.4, and 2.7. Specifically, pROC for ROC curve analysis, mice for multiple imputation, randomForest, caret, gbm, e1071, and xgboost for machine learning models, pec for calibration and rmda for decision curve analysis (Page 6, line: 173 & Lines: 184-185, Page 8, Lines:243-245).

Comment 8: Please describe the method used to avoid overfitting and the logic behind it.

Response 8: Thank you. To mitigate model overfitting, we employed two strategies. First, we employed feature selection (Boruta algorithm followed by logistic regression and Random Forest ranking) to retain only variables that consistently contributed to the prediction. This decreases model complexity, thereby reducing the risk of overfitting. Second, we used 10-fold cross-validation, in which the data are randomly divided into ten subsets; each subset is used once for validation while the remaining subsets are used for training. We have expanded the description of cross-validation to minimise overfitting in the revised manuscript (Page 6, Lines: 185-190).

Comment 9: Please describe Boruta algorithm.

Response 9: We thank the reviewer for this comment. We have expanded the description of the Boruta algorithm in the methods section in the revised manuscript (Page 6, Lines: 192-202).

Comment 10: Please describe Gini index.

Response 10: We thank the reviewer for this suggestion. In the revised manuscript, we have expanded our description of the Gini index in the methods section (Page 7, Lines: 205-213).

Comment 11: Please describe ROSE and SMOTE techniques, also please cite appropriate references.

Response 11: We thank the reviewer for this comment. We have added descriptions of the ROSE and SMOTE methods in the Methods section with appropriate references (Page 7, Lines: 219-224).

Comment 12: In Figure 1. Please use same font in all sections.

Response 12: Thank you. We have used the same font accordingly.

Comment 13: Is ROC the best technique to identify cutoff? The AUCs ranged from 0.574 to 0.555. Is this AUC good enough (it is pretty low)?

Response 13: The ROC curve is widely used in clinical research to assess the discriminative ability of continuous variables and to identify thresholds that best separate binary outcomes. In this study, this approach was primarily used to determine optimal cut-off points for inflammatory and nutritional indicators to facilitate dichotomisation for clinical simplicity, rather than for assessing the predictive value of these markers alone. We acknowledge that the AUC values for these markers (ranging from 0.555 to 0.574) are poor, indicating that these variables cannot be used as standalone predictors. Thus, we evaluated the prognostic value of these variables alongside other clinical features using multivariable logistic regression and machine learning approaches.

Comment 14: Regarding Table 1. This is a univariate analysis. Could you please calculate a binary logistic regression, backward conditional? What variables were the most relevant?

Response 14: Thank you for the suggestion. Table 1 presents chi square test. We have included the results of univariable logistic regression in Supplementary Table 3, as recommended. Several factors were significantly associated with incomplete response in variable analysis, including advanced stage, poor performance status, elevated LDH, anemia, and inflammatory markers (PNI, SII, SIRI, MLR, PLR, and NLR) (Page 9, Lines: 268-271). Most of the variables (age, sex, BMI, subtype, BCL6 experession, etc) rejected by the Boruta algorithm were insignificant in univariable logistic regression analysis. Moreover, based on the strength of the odds ratio, stage and performance status are more relevant, which are also the most influential variables, as determined by the Boruta algorithm. In line with the PROBAST tool (https://doi.org/10.7326/M18-1377) and best practices for developing prediction models, we did not use univariable analysis for feature selection. Instead, we used a robust feature selection strategy using the Boruta algorithm, followed by multivariable logistic regression.

Comment 15: In figure 2. Does the Boruta algorithm rates the importance of the variables? Does Boruta importance variable correlate with the hazard-risk or odds ratio of binary logistic regression?

Response 15: We thank the reviewer for the query. Boruta provides importance scores, reflecting the contribution of each variable to classification accuracy within the Random Forest framework. However, these scores are not equivalent to odds ratios or hazard ratios, which quantify the magnitude and direction of association. We have clarified this in the revised manuscript and in the Figure 2 caption. Importance reflects contribution to prediction accuracy within the Random Forest framework and is not directly comparable to odds ratios or hazard ratios.

Comment 16: Regarding attributes (variables) of Figure 2. Did the variables correlate between them?

Response 16: Thank you. Attributes are expected to exhibit some levels of correlation between themselves. Boruta is based on Random Forests, which are relatively robust to multicollinearity and can retain correlated predictors if each contributes predictive information. However, the algorithm does not indicate whether variables are correlate or not.

Comment 17: Table 2. The AUCs and other performance parameters are low. What usefulness for clinical practice those values provide? Please confirm.

Response 17: Thank you for the critical comment. We agree that the discriminative ability (AUC) for our models is modest but acceptable. While the AUC values are moderate, our model still offers better discrimination compared to existing valuable prognostic indices. Moreover, our nomogram demonstrated better net benefit in decision curve analysis compared to IPI-based systems, suggesting improved clinical utility in guiding treatment decisions. We expanded this issue in the discussion section. Moreover, we hope we did not overestimate our model performance, as we have concluded our results as "overall discriminatory power of our model is limited and we recommend developing a better model."

Comment 18: Should lymphoma subtype be included in the nomogram?

Response 18: Thank you for the suggestion. We understand that lymphoma subtypes, such as DLBCL, FL, MCL, and Burkitt lymphoma, differ in their biological behaviour and survival outcomes. However, our analysis showed that lymphoma subtype was not a significant predictor of initial treatment response in our cohort. Regardless of its significance, we tested model performance by incorporating the subtype into the final model, and this did not improve the model’s discriminative ability. Thus, we excluded the subtype from the final model to maintain model simplicity and parsimony. Additionally, we validated the model specifically within the DLBCL cohort (a subtype with a large sample size), where our model demonstrated slightly improved performance compared to existing prognostic indices.

We acknowledged the limitation of our model as "the predominance of DLBCL cases may influence generalizability" and proposed evaluating model performance within rarer subtypes in future studies using larger cohorts (Page 22, Lines: 521-525).

Comment 19: Please describe in the footnote the interpretation of Figure 5 so you may help potential readers.

Response 19: Thank you. We have provided footnote as “IPI; International Prognostic Index, R-IPI; Revised-IPI, NCCN-IPI; National Comprehensive Cancer Network-IPI”.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have amended the manuscript in accordance with the reviewer's comments. However, I have a minor comment regarding Table 3. I recommend reversing the order of the rows and columns to make it more visually informative. That is, the vertical columns should correspond to the categories low-risk (<138 points), low-intermediate (138-188 points), high-intermediate (188-225 points), and high-risk (>225 points), and the rows should list the compared characteristics of the subgroups.

Author Response

Comment: The authors have amended the manuscript in accordance with the reviewer's comments. However, I have a minor comment regarding Table 3. I recommend reversing the order of the rows and columns to make it more visually informative. That is, the vertical columns should correspond to the categories low-risk (<138 points), low-intermediate (138-188 points), high-intermediate (188-225 points), and high-risk (>225 points), and the rows should list the compared characteristics of the subgroups.

Response: Thank you for your suggestion. We have revised Table 3 accordingly. The order of rows and columns has been reversed as recommended; the columns now correspond to the four risk groups in the revised manuscript (see Table 3).

Reviewer 2 Report

Comments and Suggestions for Authors

Thank you for the answers and revised version

Author Response

Comment: Thank you for the answers and revised version

Response: Thank you. We appreciate the time and constructive comments you provided to help improve the manuscript.