TPENAS: A Two-Phase Evolutionary Neural Architecture Search for Remote Sensing Image Classification

Round 1

Reviewer 1 Report (Previous Reviewer 1)

The authors improved the paper since the last submission and considered almost all of my remarks.

However, the results presented in Table 6 and Table 8 must be clarified. In both these tables, the authors evaluate the state-of-the-art methods and their proposed TPENAS on NWPU45 dataset. In Table 6, the performance of “TPENAS_V1 (ours)” and “TPENAS_V2 (ours)” according to the „OA“ is 90.38 and 87.79, respectively. However, in Table 8, the performance of “TPENAS (ours)” increases to 95.70. This must be clarified/explained.

Moreover, the recent related publication have to be cited: https://doi.org/10.1080/2150704X.2022.2161847, https://doi.org/10.3390/rs15010091

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report (Previous Reviewer 2)

The authors have addressed the previous comments from this reviewer. The only comment is the need to discuss the implication of using smaller training datasets on the performance of the proposed solution. I understand that the authors based their strategy for the training data size on the state-of-the-art, but what is the implication of that decision on the performance of the solution? At least, this should be discussed in the Discussions section.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

This manuscript is a resubmission of an earlier submission. The following is a list of the peer review reports and author responses from that submission.

Round 1

Reviewer 1 Report

In this paper, the authors proposed a two-stage evolutionary neural architecture search framework.

It was experimentally evaluated on three benchmark datasets. Generally, I think the article interesting, and solves relevant problems. However, some points must be improved.

Main remarks

 ·        Authors have to review/analyze related state-of-the-art works with already proposed similar methods/frameworks, e.g. https://doi.org/10.1109/TNNLS.2022.3220699, https://doi.org/10.1016/j.isprsjprs.2020.11.025, etc., and to identify the advantages/superiority/differences of the proposed one in this article one. It is strange that in this work, authors even cite these related articles as they are the most related ones… Moreover, the experimental investigation should be done by comparing the proposed method/framework with the already similar ones proposed in the literature. 

·        In the Abstract authors put some conclusions “The experimental results on the NWPU45 [1] dataset show that TPENAS improves overall classification accuracy by 4.02% when compared to other NAS algorithms. Furthermore, it reduces the parameters by at least 13 times when compared to classic classification methods”. These sentences are more suitable for the “Conclusions” section.

·        As for the “Conclusions” section, it must be extended. It lucks summarizing conclusions about the obtained results and numerical experiments. Moreover, future works also could be identified.

·        Line 319-320: What was the motivation for splitting datasets at such a percentage rate?

·        The used experimental environment is not described in the paper. Please, describe what computing infrastructure was used.

·        More details about algorithm settings in the experimental investigation must be provided, e.g., the mutation and crossover probability rates.

·        The same mathematical notions used for different things: “N” in line 185 corresponds to individuals, but in line 330 “N” corresponds to all samples; in line 218 “n” represents the number of nodes, however in line 473 “n“ is related with the number of models; “M” in line 192 corresponds to The population size in the second search stage, however in Tables 3-5 “M” used in Param(M).

·        These relevant and recent related publications should be cited: https://doi.org/10.3390/rs15010091, https://doi.org/10.1109/TNNLS.2022.3220699, https://doi.org/10.1016/j.isprsjprs.2020.11.02 

Some clarification in the text is required

·        In lines 85-86, the sentence “Since evolutionary computing is very easy to perform parallel computing…” must be clarified/rewritten.

·        Lines 134-153: formulate contributions in a more compact way; ideally, use one sentence for one contribution.

·        Section “Materials and Methods” should start with 1-2 sentences describing what will be presented in the following subsections.

·        Lines 161-162: cite UCM21 and NWPU45 datasets.

·        Line 161: when citing datasets, please provide a direct link to data (e.g., PatternNet [60] does not have a direct link to data).

·        Line 65 abbreviation NAS is the first time used in the text. While the first time using full words, “Neural Architecture Search“ must also be written.

·        The caption of Figure 16. word “Pareto” should start with a capital letter.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

The authors presented a modification to the NAS algorithm for remote sensing image classification. Based on the presented results, the proposed approach shows overall improvement in classification accuracy. However, this reviewer suggest the following improvement to the manuscript.

1. The key aspect of the proposed method is the two-phase evolutionary multi-objective NAS framework. However, this part of the approach is not well presented and discussed. A major revision is required for the readers to understand the optimization approach.

2. The utility of Figures 8, 11, and 14 is minimal since the matrices are sparse; the authors should find a more insightful way to present the same info.

3. The experiments to evaluate the approach did not account for noisy images which is common in remote sensing applications. A complete evaluation should include the use of noisy images or a detailed explanation of why they are not included.

4. In Section 4.2, the authors discuss about the optimal Pareto front being equal to 5, which is the same as for the first search stage but they did not discuss the rationale behind this.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

The authors proposed a two-stage evolutionary neural architecture search framework, which is effective to search the best suitable model for classification task. However, the problem is not well modeled and the authors directly proposed algorithm for the task. The mathematical model and related analysis to the specific problem should be enhanced before publication. Other comments are as follows:

1.      The mathematical model for the optimization should be proposed in the manuscript. Optimization algorithm should be designed to solve the model.

2.      The unique contrition of the manuscript to existing work should be enhanced. In the current, what is novel to existing works is not clearly presented.

3.      How to set the parameters involved in the manuscript should be introduced.

4.      The structure of the manuscript should be improved. Datasets should be moved to the experimental part. Moreover, Algorithm should be summarized after theoretical introduction of the proposed TPENAS.

5.      Algorithm 1 should be designed related to TPENAS task. Current version is just a two step optimization framework.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 4 Report

Please see the attached file for details.

Comments for author File: Comments.pdf

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

The authors corrected the paper according to almost all of my remarks.

However, the corrections and, even more, the answer to my first remarks (that is the most important according to the scientific novelty) causes uncertainty. 

My initial remark was, “Authors have to review/analyze related state-of-the-art works with already proposed similar methods/frameworks, e.g. https://doi.org/10.1109/TNNLS.2022.3220699, https://doi.org/10.1016/j.isprsjprs.2020.11.025, etc., and to identify the advantages/superiority/differences of the proposed one in this article one. It is strange that in this work, authors even cite these related articles as they are the most related ones… Moreover, the experimental investigation should be done by comparing the proposed method/framework with the already similar ones proposed in the literature. “

 When considering/answering this remark:

- Firstly, authors only cite these suggested two works along with other publications without any description of these relevant works. The suggested publications are very reeled to this submitted paper as they are introducing frameworks/models called SceneNet, E2SCNet that are similar to the proposed one in this paper. These state-of-the-art methods and of course, other related ones must be analyzed deeper in this paper. Authors should investigate the related/similar state-of-the-art methods and identify the advantages/superiority/differences of their proposed TPENAS method compared with other ones.

-        Secondly, that is also very important. In the answers to this remark, the authors provided the results of an experimental investigation where the TPENAS framework is compared with the state-of-the-art SceneNet, E2SCNet models on only the NWPU45 dataset. The performance of the proposed TAPENAS framework according to the „OA“ is 95.7 (that is slightly higher) than at sate-of-the-art SceneNet, E2SCNet (95.22 and 95.23 Accordingly). However, in the paper's initially submitted and revised version, the “OA” of TPENAS is 90.38. That is lower than in SceneNet, E2SCNet.

-        Thirdly, the authors do not include this experimental comparison in the revised version of the paper.

To show/prove scientific novelty, the deep competitive analysis of similar state-of-the-art methods and experimental evaluation on the considered datasets have to be done.

The authors must correct the paper according to my initial remark and the current one.

Author Response

Thank you very much for your kind recognition of our work. We sincerely appreciate your valuable comments and suggestions, which have helped us to improve the quality of our work. We will carefully revise and resubmit the paper.

In the original manuscript, TPENAS used 20% of the NWPU45 dataset as the training dataset, resulting in an OA of 90.38. In comparison with the SceneNet, E2SCNet algorithms, TPENAS used 80% of the NWPU45 dataset as the training dataset and obtained an OA of 95.7%.

Reviewer 2 Report

The authors have made some significant improvements to the manuscript. Nevertheless, this reviewer would like to suggest these additional revisions to the manuscript:

1. This applies to Figs 6, 7, 9, 10, etc. it is difficult to differentiate between the "Nonoptimal solution" and "Nondominant solution" in black and white. The authors should consider a different symbol for one of those.

2. In Figs. 8, 11, and 14, remove the black background in the diagonal; it provides no additional insight, but impair seeing the numbers.

Author Response

Thank you very much for your kind recognition of our work. We sincerely appreciate your valuable comments and suggestions, which have helped us to improve the quality of our work. We will carefully revise and resubmit the paper.

Reviewer 3 Report

Though the authors have addressed most of my comments in previous round, the following issues should be further handled before publication:

1. Eqs. (1) and (2) are not conventional formulas for optimization problems. The left and right part do not equal to each other.

2. Though the authors improve Algorithm 1 by adding 'D: Remote sensing image classification datasets', D should represent Remote sensing image classification problem. Moreover, such simple modification is still hard to follow.

Author Response

Thank you very much for your kind recognition of our work. We sincerely appreciate your valuable comments and suggestions, which have helped us to improve the quality of our work. We will carefully revise and resubmit the paper.

Reviewer 4 Report

1. Line 366, "As a mentioned above in section 2.1", there should not be a "a"?

2.The language is repetitive and not authentic in Line 367-369, please improve them.

Author Response

Thank you very much for your kind recognition of our work. We sincerely appreciate your valuable comments and suggestions, which have helped us to improve the quality of our work. We will carefully revise and resubmit the paper.