Fusion Model for Classification Performance Optimization in a Highly Imbalance Breast Cancer Dataset

The present work aims to provide evidence for the use of algorithms for accurate diagnosis of breast cancer.

The work is interesting, however there are some points requiring correction and / or clarification, as follows.

1.     There is no reason to give emphasis to Saudi Arabia breast cancer cases. Women may be affected by breast cancer worldwide.

2.     There are several typo/grammar/syntax errors, i.e., Line 189.

Thank you for the compliment.

The authors have replaced the sentence with a global scenario.

The authors have resent the manuscript for another round of proofreading.

Reviewer comments

Action Taken

In this paper, the author proposed a fusion model for classification performance optimization in highly imbalance breast cancer dataset. The organization of the paper is good, and the idea of the paper is overall good. However, there are still some major issues that authors should revise and modify in order to accept the paper.   –

1.     The authors need to include a good literature survey to show exactly what is novel about the proposed approach.

2.     In addition, they should analyze these studies more carefully, and summarize the limitations to address in this study. These limitations also should be highlighted as their contributions, and evaluated by the experiment. Therefore, the problem description logic in the Introduction part should be reorganized.

3.     Moreover, the authors are suggested to provide more existing works. Please refer and cite the recent papers, not limited to

10.1109/TNNLS.2021.3071122

10.1109/TNNLS.2021.3133262 10.1109/TKDE.2020.2985965.  

4.     Some topics are significantly overstressed without any need. The authors need to structure the content better, to facilitate legibility and understanding.  

5.     I miss a section that outlines the limitations of your approach and the possibilities of extension. Are there any disadvantages or limits of the proposed approach?

Thank you for the compliment.

The authors have revised the whole section of the literature review which include new sections as the following: 

1.1.  Overview of Class Imbalanced Issue in Breast Cancer Dataset

1.2.  Overview of Techniques Handling Class Imbalanced

1.3.  Overview of Class Imbalanced Issues in Breast Cancer Studies

1.4.  Research Contribution

Based on the literature review, the authors have highlighted the gaps found in Table 3 and discussed the limitations in section 5, pg. 24.

The authors have cited suggested papers as follows:

Ref# 19

Ref# 21

Ref# 23

and cited more recent works and in the manuscript. Kindly refer to the highlighted reference.

The authors have reconstructed the whole paper for better reading and continuous flow of information.

The authors have added a new section on limitations in section 5, pg. 24.

Reviewer comments

Action Taken

The paper explored various data undersampling and oversampling techniques, and their combinations to handle imbalances in the dataset. The results of multiple experiments with breast cancer dataset using standard machine learning classifiers (J48, NB, RF and LR) are presented. More recent deep learning based classifiers have not been considered in this study.

My greatest concerns are with the novelty of this study. The experiments with unmodified existing methods are presented, however no new method or methodology is proposed. The experiments are presented on a single dataset only, which raises the concerns that the findings will not be transferrable to other datasets.

The solution of authors for selecting the best model is pretty straightforward: they perform experiments with all models and announce the winner with best performance. Such approach is data-dependent.

·        

Motivation for selecting J48, NB, RF and LR classifiers is not clear. These are old methods, explored many times before.

The results are not analysed using statistical methods, therefore it is not clear that the observed differences between method’s performance are statistically significant. Since you used 10-fold cross-validation, you can use performance results from each fold to calculate confidence limits of each model.

The influence of the optimisation of the classifiers’ hyperparameters is not explored. In many cases, tuning hyperparameters can improve the performance notably (Medical internet-of-things based breast cancer diagnosis using hyperparameter-optimized neural networks).

Comparison with original imbalanced dataset is straightforward. Of course, the balanced dataset would give better performance than imbalanced. A comparison with random baseline, i.e., Random Oversampling and Undersampling would be more meaningful.

Data visualization is poor: using interpolated line plots for categorical independent variable data (Figure 6a,d) can lead to wrong conclusions.

There is no critical analysis of the limitations of the  methodology used in this study.

The authors have changed the deployed classifiers based on the gaps found in the literature review (Kindly refer to Table 3, pg.10 and Table 14, pg. 23). The new classifiers are XGBoost (from ensemble learning), ANN (from deep learning) and SVM (from classical machine learning). Furthermore, this paper utilized clinical data which is more suitable for machine learning.

The authors have evident of the paper’s novelty during the comparative analysis with previous studies that deployed the same BCSC dataset. Kindly refer to Table 14, pg. 23.

The authors presented the following analysis before deriving to the best fusion model:

1.    Model classification evaluation

2.    Confusion Matrix analysis

3.    ROC analysis

4.    Model efficiency analysis based on training time and testing time

Furthermore this study approach is to adopt data-level strategy to improve the model performance.

The authors changed the deployed classifiers based on the summary of the literature review. Kindly refer to Table 3, pg. 10.

The authors have included a new section (4.4) about model efficiency on pg. 22.

The research aim is to investigate the resampling techniques optimization to improve the model classification as presented in section 3.

The authors have changed the comparison approach as per the suggestion. Kindly view to section 4, pg. 16-23.

The authors have tried their level best to improve the visualization within the constraint of correction time provided by the editor.

The authors have added the discussion on the limitation of the study in section 5, pg. 24.

Comment from the reviewer

Response

The revised version has fixed successfully my previous concerns and the paper can be accepted at this stage.

Thank you very much.

Comments from the reviewer

Response

- A very high performance result achieved by the authors may be due to overfitting. The authors used a simple train-test split. Why k-fold cross-validation was not used, which allows to mitigate overfitting and achieve an more balanced evaluation of performance?

- Only three studies of other authors were used for comparison in Table 14, which is not sufficient considering a large number of studies using the same benchmark datasets. The authors may check and consider for inclusion and comparison:

·       Appl. Sci. 2022, 12(4), 1957;

·       Future Internet 2022, 14(5), 153;

·       and other relevant studies.

- Modify colorbars in Figure 8: there can be no fractional number of papers.

Based on your suggestion, we considered the k-fold cross validation and added results along with details in the revised version. Kindly refer to Table 5, Table 6, and Table 7. Pg. 16-17.

The authors have included more related research using BCSC dataset as suggested.

This article is included in the comparison because using BCSC (image data).

This article is using Wisconsin Diagnostic Breast Cancer (WDBC) dataset, thus not included.

Other relevant studies using the same BCSC dataset are included in the comparison. Kindly refer to Table 15, pg. 23-24.

The authors have decided to delete Figure 8 because it was cited from:

[58] Werner de Vargas, V.; Schneider Aranda, J. A.; dos Santos Costa, R.; da Silva Pereira, P. R.; Victória Barbosa, J. L. Imbalanced data pre-processing techniques for machine learning: a systematic mapping study. Knowl Inf Syst 2022, 65, 31–57 (2023). https://doi.org/10.1007/s10115-022-01772-8