Round 1

Reviewer 1 Report

This paper proposed a new optimization method for GA to fine tune the hyperparameters of the machine learning models such as random forest or CNN. The paper give sufficient information for methods design and compared results with existing optimization methods to show the strength of the proposed method. The manuscript can be accepted after minor revision with language and to correct small errors. 

Section 2 “sped up” should it be speed up？Please check for other small errors like this

There are some “error reference not found”mark in the manuscript, please check.

Can this optimization method for GA be used for other machine learning models or it is specific for CNN and RF?

Author Response

Response to Reviewer 1

General Comment:

This paper proposed a new optimization method for GA to fine tune the hyperparameters of the machine learning models such as random forest or CNN. The paper give sufficient information for methods design and compared results with existing optimization methods to show the strength of the proposed method. The manuscript can be accepted after minor revision with language and to correct small errors.

Response: Thank you very much for this comment!

 

Comment 1: Section 2 “sped up” should it be speed up? Please check for other small errors like this

Response 1: Thank you so much for your valuable comment! Please kindly note that the entire manuscript has been thoroughly proofread and we have followed your advice and made appropriate changes to our written presentation.

Comment 2: There are some “error reference not found” mark in the manuscript, please check.

Response 2: We are really grateful for this! Apparently, we made some changes to the manuscript which ended up leaving some hanging references behind. We have searched through the manuscript and fixed all the hanging references. Thank you so much!

 Comment 3: Can this optimization method for GA be used for other machine learning models or it is specific for CNN and RF?

Response 3: Thank you for the comment! Yes, the SDSGA can be used to optimize any machine learning model and not just the CNN and RF. However, larger models may take a much longer time before returning reliable results. This is why the need arises to modify existing algorithms to be more efficient in tuning the hyperparameters. Please note that we have made included this statement in the concluding section.

Reviewer 2 Report

This paper has adequately described a Simple Deterministic Selection-based Genetic Algorithm (SDSGA) for hyperparameter tuning of machine learning methods. I suggest the authors further improve this paper to meet the requirements of publication.

• The format problem is quite essential, but it is should be seriously improved. There are too many typos or formatting errors in the current submission. For example, many period symbols are missing; the format on pages 5, 6, and 7 are broken; figure 3 has low quality; pages 12, 14, and 15 have missing references.
• I think the proposed SDSGA method is well presented. But a basic question lies in that the authors only compared some black-box hyperparameter tuning methods (such as random search, grid search, bayesian optimization, and evolutionary strategies), which are NOT SOTA in current machine learning research. I suggest the author investigate the scope of AutoML and describe the relationship to SDSGA in the first two chapters.
• Section 3 also includes some description of experimental contents. The author should change to a new section named “4. experiments” from section 3.3.
• The experiments on MNIST and UCI (CAR, Cancer, Adult, and Letter) achieve good performance. However, both two datasets are small, and there is no description for the change of model hyperparameters. I suggest the author do more experiments on a bigger dataset, and also show the change curve about hyperparameters.

Author Response

Response to Reviewer 2

General Comment: This paper has adequately described a Simple Deterministic Selection-based Genetic Algorithm (SDSGA) for hyperparameter tuning of machine learning methods. I suggest the authors further improve this paper to meet the requirements of publication.

Response: Thank you very much for your comment. The paper has been improved with new experiments performed and reported. Furthermore, the entire manuscript has been thoroughly proofread and corrections have been made across the manuscript.

 

Comment 1: The format problem is quite essential, but it is should be seriously improved. There are too many typos or formatting errors in the current submission. For example, many period symbols are missing; the format on pages 5, 6, and 7 are broken; figure 3 has low quality; pages 12, 14, and 15 have missing references.

Response 1: Thank you so much for your observations! We have thoroughly searched through the manuscript to find all the errors and other similar ones and corrected them. We have added the period symbols where applicable and fixed the formatting problems on the pages. The image in figure three has also been changed to a higher-quality image. The missing references error occurred due to some changes made to the manuscript leaving behind some hanging references. Thank you for calling our attention to it.

 

Comment 2: I think the proposed SDSGA method is well presented. But a basic question lies in that the authors only compared some black-box hyperparameter tuning methods (such as random search, grid search, bayesian optimization, and evolutionary strategies), which are NOT SOTA in current machine learning research. I suggest the author investigate the scope of AutoML and describe the relationship to SDSGA in the first two chapters.

Response 2: Thank you very much for your observation. We have edited the document as applicable and avoided the use of SOTA. We have also added a paragraph to describe the relationship of the SDSGA to AutoML as a whole. Included is the excerpt that captures that, which can be found in pp. 4 of the revised manuscript.

Various AutoML systems have been proposed in the literature for the practical application of hyperparameter optimization of ML models. The main goal is to automatically optimize the ML pipeline, such as ML model selection and hyperparameters. These include Auto-Sklearn [29], Auto-Weka [30], Auto-Keras [31], and Auto-Pytorch [32], which are named after the machine learning packages, they primarily support. Although this study is not about AutoML, we note that because the proposed optimization algorithm is model-free, an AutoML system can be built with little modification.

 

Comment 3: Section 3 also includes some description of experimental contents. The author should change to a new section named “4. experiments” from section 3.3.

Response 3: Thank you very much for this observation. We fully agree that a new section should be created dedicated to the description of the experiments performed. So, we have done that and rearranged the manuscript.

 

Comment 4: The experiments on MNIST and UCI (CAR, Cancer, Adult, and Letter) achieve good performance. However, both two datasets are small, and there is no description for the change of model hyperparameters. I suggest the author do more experiments on a bigger dataset, and also show the change curve about hyperparameters.

Response 4:  Thank you for the comment. In view of your suggestion, we have included a much larger dataset from the UCI repository called “COVER TYPE” dataset. We have also included a table of results that shows the performance of the random forest model and their hyperparameters. This table was missing in the earlier submission. We have also added a figure to show the graphs of the improvements of the accuracy achieved by all the optimization algorithms. 

 

Reviewer 3 Report

The research topic is important. There are some comments to be addressed, before the recommendation of acceptance.

 

Comment 1. Abstract:

(a) It seems that authors compare the proposed work with baseline models. It is expected authors to compare the work with existing works.

(b) Briefly share the key findings/results of the proposed work.

Comment 2. More precise keywords should be included to better reflect the scopes of the paper.

Comment 3. Section 1 Introduction:

(a) Elaborate the importance of the research topic.

(b) Clarify if the focus is towards deep learning algorithms.

(c) The first contribution, should it be “fewer numbers of function evaluations”?

Comment 4. Section 2 Related works:

(a) More latest works (2019-Now) should be included.

(b) Summarize the methodology, results, and limitations of the existing works.

Comment 5. Section 3 Proposed Approach for the Development of SDSGA for Hyperparameter

Tuning:

(a) Equation (3), justify the selection of “30%”.

(b) There is a formatting/conversion error in the paragraph “This challenge was addressed by increasing….”

(c) Algorithm 1, add the inputs and outputs of the algorithm.

(d) In the benchmark function, justify “dimension of 10 was used for all the benchmark functions with a population size of 200 and iteration of 100”.

Comment 6. Section 4:

(a) Ensure high resolutions for all figures.

(b) Correct the conversion error “Error! Reference source not found”.

(c) Show that the proposed method is statistically enhancing the performance compared with the baseline models.

(d)  Compare the proposed work with existing works.

Author Response

Response to Reviewer 3

Comment 1a: It seems that authors compare the proposed work with baseline models. It is expected authors to compare the work with existing works.

Response 1a: Thank you so much for the comment. We have included the Bayesian Optimization algorithm as part of the algorithms studied. This has been utilized by many researchers and we believe that it serves as a good algorithm to be used as an existing work.

Comment 1b: Briefly share the key findings/results of the proposed work.

Response: Thank you again for this comment. We have revisited the discussion of the result and discussed it more deeply as contained in section 5 of the revised manuscript. Furthermore, the concluding sentences in the abstract have summarized that the developed SDSGA achieved an improved performance.

 

Comment 2: More precise keywords should be included to better reflect the scopes of the paper.

Response 2: Thank you for the observation. We have edited the keywords and used more precise ones.

 

Comment 3a: Elaborate the importance of the research topic.

Response 3a: Thank you very much for this comment. This has been included in the manuscript as also documented in the contribution of the manuscript to knowledge.

 

Comment 3b: Clarify if the focus is towards deep learning algorithms.

Response 3b: Thank you very much for your comment. The focus of the work is geared towards developing an optimization algorithm to find the optimal hyperparameters of ML models. Deep learning models are also ML models, which makes it a viable approach in this case too.

 

Comment 3: The first contribution, should it be “fewer numbers of function evaluations”?

Response: Thank you very much for the observation. We have corrected it to “with few numbers of fitness evaluations”. We believe that the word “fitness” will invoke less ambiguity.

 

Comment 4a: More latest works (2019-Now) should be included.

Response 4a: Thank you so much. We have included more latest works from 2019 to 2021 in the related works.

 

Comment 4b: Summarize the methodology, results, and limitations of the existing works.

Response 4b: Thank you so much for the observation. We have followed your suggestion and made sure that these are all captured in the conclusion.

 

Comment 5a: Equation (3), justify the selection of “30%”.

Response 5a: Thank you very much for the comment. The justification has been included into the manuscript. We are basically assuming a value of 30% to describe the operation of a simple selection algorithm. However, to make this clear, we have added a statement explaining how this value will be gotten in practice. Below is the excerpt of the revised manuscript to justify its use.

The number of parents, probability of crossover and probability of mutation are usually given as parameters to the GA algorithm before the optimization run.

 

Comment 5b: There is a formatting/conversion error in the paragraph “This challenge was addressed by increasing….”

Response 5b: Thank you very much for these valuable comments. The grammar and syntax
issues have been reviewed to ensure consistency and accuracy in meeting the required
level of the journal standard.

 

Comment 5c: Algorithm 1, add the inputs and outputs of the algorithm.

Response 5c: Thank you very much for this esteemed observation. The input and output of the algorithm have been added.

 

Comment 5d: In the benchmark function, justify “dimension of 10 was used for all the benchmark functions with a population size of 200 and iteration of 100”.

Response 5d: Thank you very much for your comment. We have included the justification accordingly in pp. 8 of the revised manuscript. An excerpt is provided as follows:

A dimension of 10 was used for all the benchmark functions with a population size of 200 and iteration of 100, which is within the limit of the maximum number of fitness evaluations as noted in [39]. The use of dimension 10 generally introduces some level of complexity and difficulty into the optimization process, with difficulty increasing as dimension increases.

Comment 6a: Ensure high resolutions for all figures.

Response 6a: Thank you so much. The images have been changed to higher resolution versions.

 

Comment 6b: Correct the conversion error “Error! Reference source not found”

Response 6b: This has been corrected in the manuscript. Thank you very much for the observation.

 

Comment 6c: Show that the proposed method is statistically enhancing the performance compared with the baseline models.

Response: The experiments carried out and reported in the results section revealed that the proposed method performed better than the baseline methods, albeit marginally in accuracy terms. However, such small percentage differences may be worthwhile in other application areas with higher performance requirements, such as military applications involving the optimization of unmanned drones for enemy attack purposes. Nonetheless, we have now mentioned in the abstract, also in page 14 and in the conclusion of the revised manuscript, that the performance differences between the different algorithms were marginal within the confines of the application areas considered in this paper. We have also expunged throughout the entire paper such terms as “outperformed”. Finally, we note that our method's better performance, albeit marginally, was achieved at faster computing times than the other methods, emphasizing the proposed method's advantage.

Comment 6d: Compare the proposed work with existing works.

Response: Thank you sincerely! We appreciate your suggestions and comments on how to improve the manuscript. In addition to the baseline algorithms, we have now compared them with the Bayesian optimization technique. Because there are so many MOAs, it is usually impossible to compare them all. Nonetheless, we have considered the most popular ones, and we have now included the BO algorithm as a cutting-edge method in the literature. Where applicable, the obtained results have been incorporated into the revised manuscript's results section. We appreciate your assistance.

Round 2

Reviewer 2 Report

Most of my concerns are well addressed and I have two suggestions for the author.

1. In this revision, the author conducts experiments on the random forest model and explores its "best position". There is some study theoretically analyzing the split rule, split position, or tree numbers, For example: [1] Pattern Recognition 2022. Multinomial random forest. [2] Pattern Recognition 2020. Heterogeneous Oblique Random Forest. It is worthwhile to introduce and describe the difference between them and the proposed method. 

2. The format of this revision is not good-looking enough. Figure 1 is compressed in the vertical direction and should zoom out; The bottom border of Algorithm 1 is missed; Figure 3 should be compressed to one line and not in left-up and right-down; Figure 4 should zoom out to appropriate font size; Table 1 is too width while Table 4 is too narrow, both of them should be adjusted. 

Author Response

Comment 1

In this revision, the author conducts experiments on the random forest model and explores its "best position". There is some study theoretically analyzing the split rule, split position, or tree numbers, For example: [1] Pattern Recognition 2022. Multinomial random forest. [2] Pattern Recognition 2020. Heterogeneous Oblique Random Forest. It is worthwhile to introduce and describe the difference between them and the proposed method.

Response:

Thank you very much for your suggestion. We have addressed this suggestion by describing the RF implementation method that we have used in this research and other potential implementations have also been mentioned. This can be found in pp. 9 of the revised version.

Comment 2

The format of this revision is not good-looking enough. Figure 1 is compressed in the vertical direction and should zoom out; The bottom border of Algorithm 1 is missed; Figure 3 should be compressed to one line and not in left-up and right-down; Figure 4 should zoom out to appropriate font size; Table 1 is too width while Table 4 is too narrow, both of them should be adjusted. 

Response:

Thank you so much for your observations. We have checked the images for any sign of stretching and corrected them as appropriate. Figure 3 has also been corrected to properly align by putting the legends directly at the bottom of the architecture. We have also made sure that the Figure is at the right font size. The width of Table 1 has been narrowed while that of Table 4 has been expanded.

 

Reviewer 3 Report

Authors have improved the quality of the paper. I have some follow-up comments.

Comment 1a: It seems that authors compare the proposed work with baseline models. It is expected authors to compare the work with existing works.

Response 1a: Thank you so much for the comment. We have included the Bayesian Optimization algorithm as part of the algorithms studied. This has been utilized by many researchers and we believe that it serves as a good algorithm to be used as an existing work.

Follow-up comment 1a: Some more existing works have been discussed in Section 2. Some of the works should be included in the performance comparison.

 

Comment 4b: Summarize the methodology, results, and limitations of the existing works.

Response 4b: Thank you so much for the observation. We have followed your suggestion and made sure that these are all captured in the conclusion.

Follow-up comment: This comment has not been addressed. It was referring to Section 2 Related works.

 

Comment 5a: Equation (3), justify the selection of “30%”.

Response 5a: Thank you very much for the comment. The justification has been included into the manuscript. We are basically assuming a value of 30% to describe the operation of a simple selection algorithm. However, to make this clear, we have added a statement explaining how this value will be gotten in practice. Below is the excerpt of the revised manuscript to justify its use.

The number of parents, probability of crossover and probability of mutation are usually given as parameters to the GA algorithm before the optimization run.

Follow-up comment: The justification is not convincing and technical.

 

Author Response

Follow-up comment 1a

Some more existing works have been discussed in Section 2. Some of the works should be included in the performance comparison.

Response:

Thank you very much for your comment. We already included the work of Wu et al in the performance comparison. He worked on Bayesian Optimization (BO) algorithm and compared it with both Random Search and Grid Search. We have chosen to compare our work with that of Bayesian Optimization, which was reported to perform the best in the article.

Follow-up comment 4b

This comment has not been addressed. It was referring to Section 2 Related works.

Response:

Thank you very much for this observation. We have now summarized their methodologies, performance, and limitations of previously documented literature as highlighted in red font in Section 2 of the revised manuscript.

 

 

Follow-up comment 5a

Comment 5a: Equation (3), justify the selection of “30%”.

Response 5a:

Thank you very much for this comment! We really do appreciate it! Please note that the justification has now been included into the manuscript, which can be found in pp. 4 of the revised manuscript. Please note that we were basically assuming a value of 30% in our initial version to describe the operation of a simple selection algorithm. However, to make this clear, we have generalized this concept using a variable P to provide a platform for users to determine the choice of the percentage value they may require. 

Follow-up comment: The justification is not convincing and technical.

Response:

Thank you very much for your comment. In order to accommodate this comment, we have performed additional experiments to justify the choice of 30%. Consequently, we selected some of datasets and used our proposed SDSGA to optimize the ML algorithms. We did the same for the benchmark functions. The results we achieved have been included in the manuscript to justify the use of 30% of the total population as parents. This has been included in pp. 10 -11 of the revised manuscript.

We sincerely appreciate the valuable comments of our esteemed reviewers, and it is our sincere hope the above revisions will be favourably considered.