A Method for Calculating the Derivative of Activation Functions Based on Piecewise Linear Approximation

Round 1

Reviewer 1 Report

There are no MAIN questions opened because the proposed work is, in my opinion, complete and solid. For a better understanding of the material, I would suggest the Authors address these MINOR points:

A.      Table 5: in the text it is not well defined the meaning of the “Backward time”. The Authors are invited to give a more clear explanation of this term in the context of the paper.

B.      Fig. 3: the numbers of the of the inset (zoom of data) of both main figures are not readable. The Authors are invited to improve the quality of this figure.

The main question addressed is the possibility of implementing a piecewise linear (PWL) approximation method to calculate the derivative of the activation function in a neural network.

There are no MAIN questions opened because the proposed work is, in my opinion, complete and solid.  

For a better understanding of the material, I would suggest the Authors would address the following MINOR points:

    Table 5: in the text it is not well defined the meaning of the “Backward time”. The Authors are invited to give a more clear explanation of this term in the context of the paper.
    Fig. 3: the numbers of the of the inset (zoom of data) of both main figures are not readable. The Authors are invited to improve the quality of this figure.

After what was mentioned above – in my opinion -  no further improvements or controls should be considered by the Authors.

The topic is relevant and exciting for the scientific community and adequate for the journal and the Special Issue.

The topic relevant to the scientific community has already been approached in the past, but, in my opinion, the present paper gives some original contributions – also validated by comparisons with solid and multiple reference results - and a different way to look at the proposed problem.

The originality of the present contribution is, in my opinion, in the use of a method based on a piecewise linear (PWL) approximation to evaluate the derivative of the activation function in conjunction with the least squares procedure not only to improve the PWL approximation calculation but also for controlling the absolute error and get less number of segments or smaller average error.

The paper is solid, and the results are well commented on. The figures and Tables are very explicative. The numerical experiments are good, well thoughts out, and done. All of them address very well the main questions posed.

 

Author Response

Thanks for the reviewer’s kind comments and precious suggestions.Our response is attached in word.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments for author File: Comments.docx

Author Response

Thanks for the reviewer’s kind comments and precious suggestions.Our response is attached in word.

Author Response File: Author Response.pdf

Reviewer 3 Report

I have reviewed the paper in detail. The authors focuse on the computation of the derivative values of the activation function for back-propagation in artificial neural networks training tasks and make the following  contributions.

1. They use least squares to improve a general, error-controlled piecewise linear approximation method to obtain fewer segments or smaller average errors, and then extend it to the calculation of various activation functions and their derivatives.

2. They evaluate the applicability of the method in neural networks in terms of convergence speed, generalization ability, and hardware overhead. 

3. They replace the derivative calculation in the neural network training task with proposed method and verify its effectiveness on a three-layer perceptron.

I have some remarks:

1. Line 125: "The core idea of the algorithm is to determine the subinterval straight line by least squares, then calculate the maximum absolute error between the line and the real curve, and find the maximum absolute error less than the predetermined error through continuous iteration, the steps of the algorithm are as follows approximation". So it is meaningful to mention some approximation methods (Mathematics, 9(16), (2021) 1895. Mathematics, 10(12) (2022) 2027. Mathematics 10(7) (2022) 1149. Numerical and theoretical approximation results for Schurer–Stancu operators with shape parameter $\lambda$, Comp. Appl. Math. 41 (2022)) that use the same idea to find maximum absolute errors.

2. Please make abbreviations when they first appear. Otherwise it is very difficult to follow them.

3. Eq 2 and Eq 3: What are the indices of the following sums?:

?? + ? ∑ ? = ∑ ?(?)

? ∑ ? + ? ∑ ?² = ∑ ? ?(?)

4. Line 143: Please put a dot.

5. Line 162: Please put a dot.

6. Line 168: Please put a dot.

7. Line 182: Please put a dot.

8. Please provide the codes that show how do you obtain Table 1 and Figure 5. Please explain Figure 5 more. How is it clear that the sigmoid, softplus functions implemented using the PWLMMAE algorithm converge faster than the sigmoid, softplus functions encapsulated in PyTorch.  

I want to see the revised version of the paper.

Author Response

Thanks for the reviewer’s kind comments and precious suggestions.Our response is attached in word.

Author Response File: Author Response.pdf

Round 2

Reviewer 3 Report

I thank the authors for considering my corrections. They corrected/revised the paper very well.  The quality of the paper is very good. The current version is ready for acceptance.