Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network

Round 1

Reviewer 1 Report

In this work the authors propose a dynamic convolutional NN, ReConv, to address the issue of redundant information in dynamic convolutional NNs with attention. ReConv reduces spatial information redundancy by reducing the number of feature map channels, and employs adaptive average pooling to retain key information. ReConv's accuracy and computational efficiency are demonstrated empirically.

I think this is a well-structured paper, and the motivation and design of ReConv make sense. I have only minor comments:

I think the paper would be clearer if the background material in Subsection 3.1 was expanded and given its own section separate from the authors' proposed ReConv Networks. E.g. a clearer definition of exactly what the authors mean by lightweight NNs (as opposed to conventional CNNs), dynamic NNs, etc.

p8 "To prevent overfitting[4,42], our image is increased by up to 4px during random horizontal flip and shift." - please give more details of how this is done.

Start of section 4: please add a discussion of why you chose CIFAR10, CIFAR100, and ImageNet-100 for the empirical tests and any limitations of the tests used for drawing general conclusions about your method.

In addition, I think the English/grammar needs a bit more proofreading. Here is a list of minor typos and errors:

abstract:

"the Dynamic Neural Networks (DNNs)" -> "Dynamic Neural Networks (DNNs)"

"It achieves higher accuracy of the network test while requires less network parameters." - fix grammar

Section 1 p1

"the image classification" -> "image classification"

"are typically with" -> "typically have"

p2

"A direct consequence is that it brings a large number of computation and potential interference." - fix grammar

Fig 1 caption: fix grammar

p3

"work are" -> "work is"

"it introduces" -> "we introduce"

p4

sec 3 title: "propose" -> "proposed"

"model of dynamic network" -> "model of dynamic networks"

p5

"accuracy rate will also be change." -> "accuracy rate will also change."

p6

"It shows that when C is consistent with the number of attention cores k , the effect is best by a series of experiments." - grammar

"On the RCR-based method, we optimize it." - reprase this more clearly

p7

"and compare the results of accuracy to compare which pooling layer is more effective." -> rephrase

p8

"of which 1281167 images are used for training and 50,000 images are used for validation, 50,000 images are used for the test." -> "of which 1281167 images are used for training, 50,000 images are used for validation and 50,000 images are used for the test."

p10

"A series of ablation experiments is conducted" -> "A series of ablation experiments are conducted"

Author Response

Please refer to the Word documents for the revision of the manuscript and point-by-point response to the reviewer’s comments. Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Writing:

1. Abbreviation - DNN is a widely accepted abbreviation for Deep Neural Network by the research and academic community. So, it is highly recommended not to use this abbreviation for Dynamic Neural Networks. Probably, use DyNet. 
2. I am not sure how you drive an abbreviation for the dynamic convolutional network as ReConv. Please introduce an easily catchable abbreviation. 
3. Uncommon abbreviations must be introduced when the terminology appears in the manuscript, including the abstract. Once the abbreviation is introduced, then it should not be repeatedly redefined. There are many abbreviations in the manuscript that have been redefined, for example, CNNs. Please fix such issues.
4. Keywords - The first letter of the first keyword must be capitalized while others in non-cap format. Besides, the keywords should reflect the main theme of the research. 
5. It requires a thorough review for English grammar checking. E.g., "3. Proposes ReConv Networks" --> "3. Proposed ReConv Networks".

Model:

1. Complete a failure mode experiment and present quantitative and qualitative analysis.
2. How would you overcome the failure cases?

Author Response

Please refer to the PDF documents for the revision of the manuscript and point-by-point response to the reviewer’s comments. The first part is a clean updated manuscript without marked. The second part is our point-by-point response to the reviewers’ comments. The last part is an updated manuscript with red marked indicating changes.

Author Response File: Author Response.pdf