An Adaptive Offset Activation Function for CNN Image Classification Tasks
Round 1
Reviewer 1 Report
The authors indicate that:
1) Their paper proposes a novel parametric activation function called AOAF.
2) A large number of experiments have demonstrated that AOAF achieves better average classification accuracy than other parametric and non-parametric activation functions in most image classification tasks.
However, it would better for the authors to add the following in their revise version:
1. Add geometric interpretation to the proposed model. How does the parameters are chosen? What criteria? What range of the free parameters? The authors also can discuss about the limit etc.
2. The authors made the conclusion that “Compared with ReLU, the average classification accuracy of our proposed activation function improved by 3.82%, 0.6%, 1.02% and 4.8% for the four datasets, respectively”. Best possible if the authors discuss, how does they compare the performance against any established methods with maximum 4.8% increment in accuracy? Does it significant contribution? Justify it with any supporting facts and evidence.
3. Why the proposed method is better? Does one hundred epochs are sufficient? The authors need to provide the justification on parameters setting. For instance, why Adam is used as optimizer? There must be a reason.
4. Maybe, the authors can provide the open-source codes to implement their methods. This will enrich the knowledge among our peers.
The rest looks nice.
Author Response
Dear reviewer, thank you for your patience and attention. According to your comments, we have made some answers and made corresponding modifications in the paper. Please check.
- Point 1: Add geometric interpretation to the proposed model. How does the parameters are chosen? What criteria? What range of the free parameters? The authors also can discuss about the limit etc.
Response 1:
- Add geometric interpretation to the proposed model.
Figure 5 (a) shows the image of the function of AOAF: the segment function with as the segment point, the left side of the segment point, consists of a horizontal straight line, at this time the input values less than 0.17α are uniformly transformed into 0.17α, the right side of the segment point consists of a slope line with slope 1, which also assumes that the input values greater than 0.17α will keep their original values. Figure 5(b) shows the image of the derivative function of AOAF: the same segment function with 0.17α as the segment point, the segment function consists of two horizontal straight lines, the value less than 0.17α will be converted to 0 after the input, and the value greater than 0.17α will be converted to 1
- How does the parameters are chosen? What criteria?
As described on page 4 of the article, we first proposed the underlying model of AOAF: , where represents the mean value of the input tensor and and are two adjustable parameters. In order to explore the appropriate values of adjustable parameters, we first make ==a, and choose different values between [0,1] and assign a to constitute the preliminary AOAF, and use the AOAF as the activation function of AlexNet at this time, and classify the CIFAR100 image dataset, and choose the value that makes AlexNet classify the image with the highest accuracy to assign a, at this time and , so we obtained the further form of AOAF: AOAF. Since represents the degree of upward bias of AOAF, we continued the parameter Again, we still choose different values between [0,1] to assign to after using the AOAF as the activation function of AlexNet for the image classification task at this time, and choose the value of when the model has the highest classification accuracy for the image, at this time . The above parameter selection process we simultaneously performed on The same classification experiments were conducted on ResNet at the same time, and the final results were the same as AlexNet.
- What range of the free parameters? The authors also can discuss about the limit etc
In the image classification model, the normalization operation is essential, the values of the input image tensor matrix fall between [0,1] after normalization, the adaptive parameters in AOAF are closely related to the degree of rightward shift of the function, so the adaptive parameters will also lie between [0,1].
In the base model stage of AOAF, we discuss the classification accuracies of 92.0% and 86.8% achieved by the AlexNet model equipped with AOAF as the activation function for the image classification task of CIFAR100 when ==0 and ==1, respectively, which are lower than the model achieved when==0.17. 92.5%. In the same configuration, ResNet achieved 93.4% and 93.0%, respectively, which are still lower than the 93.6% classification accuracy achieved by the model with ==0.17.
- Point 2: The authors made the conclusion that “Compared with ReLU, the average classification accuracy of our proposed activation function improved by 3.82%, 0.6%, 1.02% and 4.8% for the four datasets, respectively”. Best possible if the authors discuss, how does they compare the performance against any established methods with maximum 4.8% increment in accuracy? Does it significant contribution? Justify it with any supporting facts and evidence.
Response 2:
- Best possible if the authors discuss, how does they compare the performance against any established methods with maximum 4.8% increment in accuracy?
The average classification accuracy of four neural network models using ReLU as activation function on four image data sets is 74.33%, 85.33%, 83.98% and 75.78%, respectively. The four neural network models equipped with AOAF achieved average classification accuracy of 78.15%, 85.93%, 85.0% and 80.58%, respectively. It can be seen that when AOAF is used as the activation function of the model, compared with ReLU as the activation function, The average image classification accuracy of the model is improved by 3.82%, 0.6%, 1.02% and 4.8%, respectively.
- Does it significant contribution? Justify it with any supporting facts and evidence.
This result shows that the AOAF proposed in this paper has superior performance in image classification tasks compared to existing parametric and nonparametric activation functions, furthermore, the AOAF provides us with a new direction in the study of activation functions: adaptive offsets, by introducing the mean value of the input tensor into the activation function to achieve its flexibility, and by offsetting the function to introduce the idea of attention, so that the negative values can participate in the feature extraction process of convolutional neural networks as smaller positive values.
- Why the proposed method is better? Does one hundred epochs are sufficient? The authors need to provide the justification on parameters setting. For instance, why Adam is used as optimizer? There must be a reason.
Response 3:
- Why the proposed method is better?
On the basis of ReLU, we propose to introduce adaptive parameters and the idea of migration into the activation function, which changes ReLU's usual practice: If the negative input is directly converted to 0, the negative input cannot participate in the feature extraction process of the convolutional neural network, resulting in low classification accuracy of the model in image classification tasks. The AOAF proposed by us converts the negative input part into a smaller positive value and participates in feature extraction, making the best use of the existing information. In addition, we shift the function as a whole to the right, so that some of the positive numbers have the same properties as the negative input, and the degree of right shift is also determined by the mean of the input tensor. The above operations not only ensure the sparsity of input values, but also allow more input information to participate in the feature extraction of the model compared with ReLU, which makes the model have better performance.
- Does one hundred epochs are sufficient?
At the beginning of the experiment, we designed epochs as 40 in reference 15, and found that when epochs=40, the classification accuracy of any activation function for CIFAR100 data set was significantly lower than the expected value, so we increased epochs to 150. However, it was found that the highest classification accuracy in the experimental results usually appeared when the training epochs was around 70 to 80. In order to save the experiment time and extend the service life of the experimental equipment, we finally decided to set the epochs as 100.
- The authors need to provide the justification on parameters setting. For instance, why Adam is used as optimizer? There must be a reason.
In terms of optimizer, Adam is an excellent optimizer because it can dynamically and smoothly adjust the learning rate of each parameter and can quickly handle the sparse gradient problem of convex functions[参考文献]. In order to eliminate the influence of the optimizer performance on the experimental results of the activation function, we choose Adam as the optimizer for the model used in our experiments. Under the condition of ensuring the normal operation of the program, we pursue to make full use of the performance of the hardware by opening 4 threads at the same time when inputting images. Multi-threading represents that the model can be trained at a faster speed, and at the same time, each thread inputs 16 images at a time, i.e., num_workers=4, batch_size=16.
- Maybe, the authors can provide the open-source codes to implement their methods. This will enrich the knowledge among our peers.
Response 4:
It is a great honor for all the authors to hear you say so, but our next plan involves further improvement and application of this function. In order to avoid causing some unnecessary troubles, We are not able to provide the code for AOAF at the moment, When this work is completed, we will make it public at the appropriate time.
Thank you for your understanding.
Thank you again for your patience and attention.
Author Response File: 
Author Response.pdf
Reviewer 2 Report
Dear Authors,
I have reviewed the paper. AOAF Presented in this paper is quite interesting. The introduction and the literature section can be improved by adding and citing with more relevant research findings to Rectified linear Unit (ReLU) neural network learning algorithms. By adding more research works to the relevant sections, the readability will significantly improve.
Author Response
Dear reviewer, thank you for your patience and attention. According to your comments, we have made some answers and made corresponding modifications in the paper. Please check.
Point 1: I have reviewed the paper. AOAF Presented in this paper is quite interesting. The introduction and the literature section can be improved by adding and citing with more relevant research findings to Rectified linear Unit (ReLU) neural network learning algorithms. By adding more research works to the relevant sections, the readability will significantly improve.
Response 1:
When writing this paper, we did not give sufficient consideration to the references. After referring to your comments, we made certain supplements to the references about the improvement of ReLU and other activation functions in this paper.
Thank you again for your patience and attention.
Author Response File: Author Response.pdf
Round 2
Reviewer 1 Report
The revised version can be accepted for the publication. However, please go through the usage of English in the final version.
Author Response
Comments and Suggestions for Authors:
The revised version can be accepted for the publication. However, please go through the usage of English in the final version.
Response:
Dear reviewer:
After receiving your comments, we immediately started to check and revise the grammar of the article. We did find some words and phrases that needed to be corrected during the check. Thank you for your suggestions.
Thank you again for your time and patience.
Author Response File: Author Response.pdf
