Skip Content
You are currently on the new version of our website. Access the old version .
MDPIMDPI
by
  • Angela Lombardi1,2,
  • Nicola Amoroso2,3 and
  • Raffaella Massafra5,†
  • et al.

Reviewer 1: Vivek Kumar Singh Reviewer 2: Md Islam Shariful Reviewer 3: Anonymous

Round 1

Reviewer 1 Report

In this paper, the authors proposed a machine learning pipeline to

explore the effects of unbalanced populations on the performance and stability of feature ranking for sentinel lymph node status classification in breast cancer.

The manuscript is organized well and easy to follow. However, I have a few minor comments:

  1. The manuscript needs an English proof-reading
  2. Please, define the acronym in the abstract AUC, should be the area under the curve?
  3. In section 3, how data split was performed?
  4. The authors should correct the typos error in subsection 3.2.1 'Pr' and 'yi'.
  5. Subsection 3.3 is not-relevant since it is very common in the scientific community.
  6. In section 4, the authors should describe the subcaption of Figures 2 and 3 in text or figure caption itself.
  7. Did the author try to compare their method to existing deep learning-based techniques?
  8. What is the advantage of the proposed method in terms of computational time and complexity?

Author Response

In this paper, the authors proposed a machine learning pipeline to explore the effects of unbalanced populations on the performance and stability of feature ranking for sentinel lymph node status classification in breast cancer.The manuscript is organized well and easy to follow. However, I have a few minor comments:

We thank Reviewer #1  for the following comments, which allowed us to improve the overall quality of the manuscript.

 

1)     The manuscript needs an English proof-reading

We proofread the entire manuscript to correct grammatical mistakes.

 

2)     Please, define the acronym in the abstract AUC, should be the area under the curve?

We apologize for the lack of this information. Yes, AUC represents the area under the ROC curve. We defined the acronym in the abstract in the new version of the manuscript.

 

3)     In section 3, how data split was performed?

We randomly selected 100 samples from the entire dataset (approximately 20% of the total) for the hold-out test set. We performed a repeated k-fold validation (N = 10 iterations, k = 10 folds) on the training set.

4)     The authors should correct the typos error in subsection 3.2.1 'Pr' and 'yi'.

Yes, we corrected the typos in the new version of the manuscript.

5)     Subsection 3.3 is not-relevant since it is very common in the scientific community.

Although we agree with Reviewer #1 about the “redundancy” of introducing performance metrics, we believe it helps the manuscript clarity to provide their definition also because of the broad and varied readership of the scientific journal Applied Sciences.

6)     In section 4, the authors should describe the subscription of Figures 2 and 3 in text or figure caption itself.

We integrated the descriptions in the captions of Figures 2 and 3 in the new version of the work.

7)     Did the author try to compare their method to existing deep learning-based techniques?

We thank Reviewer #1 for this interesting point. Advanced methods such as those based on neural networks were avoided in preference for two simpler methods with embedded feature selection strategies such as LASSO and RF. Such methods are rather well established in the literature for classification tasks in various clinical contexts and offer the advantage of providing the most important features for the algorithm's decisions. An important consequence is the higher explainability of the impact of predictors on the overall performance. This issue is particularly relevant as it enables the integration of the perspectives of both clinical specialists and algorithm developers, allowing synergy between the different entities and improving the machine learning paradigms.

We integrated this issue in the new Section “Limitations and future perspective” of the work.

 

8)     What is the advantage of the proposed method in terms of computational time and complexity?

Although the proposed framework is fully scalable and would allow a larger number of machine learning algorithms combined with different sampling strategies to be compared, it should be noted that the computational complexity and the execution time increase with the number of algorithms to be compared.

We have argued this point in Section “Limitations and future perspectives” in the new version of the manuscript.

Reviewer 2 Report

1. From the Abstract result, its not clear the final findings of this paper. Instead of using "ML framework to explore the effects of unbalanced populations on the performance and stability of feature ranking for sentinel lymph node status classification in breast cancer." use "specific output results"

 

2. The authors should add the data visualization and phylogenetic analysis tree in the results section. Also, the score based detection is required to identify the Lymph Node Status Classification in Breast Cancer. 

 

3. Overall this manuscript is important and need further modifications with proper justification for the classification of breast cancer. 

Author Response

We thank Reviewer #2  for the valuable comments. We tried to address all the points raised by Reviewer #2.

  1.     From the Abstract result, its not clear the final findings of this paper. Instead of using "ML framework to explore the effects of unbalanced populations on the performance and stability of feature ranking for sentinel lymph node status classification in breast cancer." use "specific output results"

We appreciate this suggestion who helped us to improve the clarity of our findings in Section Abstract. We added the quantitative information as suggested by Reviewer #2.

  1. The authors should add the data visualization and phylogenetic analysis tree in the results section. Also, the score based detection is required to identify the Lymph Node Status Classification in Breast Cancer.

We thank Reviewer #2 for this advice.

In our study, we employed clinical data, for which the recommended analysis may not be precisely appropriate, even though phylogenetic analysis trees are valuable and interesting research tools for molecular data.

It will be noteworthy to take into consideration the Reviewer's recommended analysis in the future, whenever molecular data will also be accessible.

Regarding the quantification of the score, we did not provide any quantitative indication of the characteristics of the patients in the previous version of the work. We integrated the quantitative information for each categorical feature in the new Table 1.

  1. Overall this manuscript is important and need further modifications with proper justification for the classification of breast cancer.

Agree. We have tried to better discuss the work in relation to other state-of-the-art studies in the new Section Landscape.

Reviewer 3 Report

 

 The main limitation of current study is that the dataset is merely small scale, and methods used are over-simplified. There are a large number of classification methods, but only two (old) methods have been considered. Both logistic LASSO and RF can select important features associated with class label, so why not consider more advanced feature selection methods?

 The authors described LASSO as it can handle multicollinearity, which is not accurate. LASSO is well known for ignoring correlations among features and only assumes independence. More appropriate methods, such as network constrained regularization for classification (Ren et al., PMID: 28511641) that use network to model interconnections among features, should be considered for comparison. The R package regnet is available on CRAN.

 Stability of selected features has been discussed. However, this is given such a small scale methods and two simple models. When number of feature increases, robust feature selection (Wu and Ma PMID: 25479793) is critical for the successful performance since these methods usually have better stability and reproducibility. It is premature to reach the conclusions presented in the study with limited scale of data and few methods compared. In fact, with high dimensionality of the features and presence of complicated data structure, model misspecification happens quite often (Wu and Ma PMID: 25479793). These critical factors have all been overlooked. The conclusions obtained may only be restricted to simple settings and cannot be extended to the general cases.

Author Response

We thank Reviewer #3  for the following comments, which allowed us to better clarify in the manuscript the rationale of our choices.

1) The main limitation of current study is that the dataset is merely small scale, and methods used are over-simplified. There are a large number of classification methods, but only two (old) methods have been considered. Both logistic LASSO and RF can select important features associated with class label, so why not consider more advanced feature selection methods?

We agree with Reviewer #3 on the relatively small size of the dataset and therefore advanced methods such as those based on neural networks were avoided in preference to two simpler methods with an embedded feature selection strategy such as LASSO and RF. Such methods are rather well established in the literature for classification tasks in various clinical contexts. Moreover, these methods offer the advantage of providing the most important features for the algorithm's decisions and therefore exhibit a higher explainability of the impact of predictors on the overall performance. This issue is particularly relevant as it enables the integration of the perspectives of both clinical specialists and algorithm developers, allowing synergy between the different entities and improving the machine learning paradigms in each specific clinical context.

We have included this information in the new "Limitations and future perspective" section of the new version of the manuscript.

 

2) The authors described LASSO as it can handle multicollinearity, which is not accurate. LASSO is well known for ignoring correlations among features and only assumes independence. More appropriate methods, such as network constrained regularization for classification (Ren et al., PMID: 28511641) that use network to model interconnections among features, should be considered for comparison. The R package regnet is available on CRAN.

We thank Reviewer #3 for suggesting this interesting work. 

Although we believe it is important to point out to the reader that for high-dimensional clinical contexts it is appropriate to examine other approaches in conjunction with Lasso as suggested in the paper for “high dimension settings with p ≫ n”, we argue that for the present analysis such a method is not strictly necessary considering the limited number of features relative to the sample size (p<<n). We have extended the "Limitation future perspective" section with the point raised by Reviewer #3.

3) Stability of selected features has been discussed. However, this is given such a small scale methods and two simple models. When number of feature increases, robust feature selection (Wu and Ma PMID: 25479793) is critical for the successful performance since these methods usually have better stability and reproducibility. It is premature to reach the conclusions presented in the study with limited scale of data and few methods compared. In fact, with high dimensionality of the features and presence of complicated data structure, model misspecification happens quite often (Wu and Ma PMID: 25479793). These critical factors have all been overlooked. The conclusions obtained may only be restricted to simple settings and cannot be extended to the general cases.

 

We certainly agree with Reviewer #3 on the observations concerning the use of more robust feature selection techniques with high dimensional data structures. As motivated in comment 2), we believe it is important to highlight this limitation. We have therefore extended the new section Limitations and future perspective.

Round 2

Reviewer 3 Report

I appreciate the authors' revision and addressing of all my questions.

Author Response

Point 1. I appreciate the authors' revision and addressing of all my questions.

Response 1. We thank the reviewer.