Exploring the Effects of Caputo Fractional Derivative in Spiking Neural Network Training

Round 1

Reviewer 1 Report

This manuscript electronics-1794615 proposed a fractional derivatives mehtod to Spiking Neural Network training using Caputo derivative-based gradient calculation. The authors focus on conducting extensive investigation of performance improvements via a case-study of small-scale networks using derivative orders in the unit interval. With Particle Swarm Optimization, the authors provide an example of handling the derivative order as an optimizable hyperparameter in order to find viable values of it. Using multiple benchmark datasets, the authors empirically show that there is no single generally optimal derivative order, rather this value is data-dependent. Improvements in convergence speed and training time are also examined and explained by the reformulation of the Caputo derivative-based training as an adaptive weight normalization technique. My overall impression of this paper is that it is in general well-organized. It was a pleasure reviewing this work and I can recommend it for publication in Electronics after a major revision. I respectfully refer the authors to my comments below.

1.       The English needs to be revised throughout. The authors should pay attention to the spelling and grammar throughout this work. I would only respectfully recommend that the authors perform this revision or seek the help of someone who can aid the authors.

2.       In the Introduction part, “main contributions” is best to list clearly by breaking it down into three points. Pleases adjust the major contributions.

3.       The original statement is revised as “Neural networks have been around in machine learning for decades by now [1-4]” ([1] https://doi.org/10.1016/j.infrared.2020.103594, [2] https://doi.org/10.1016/j.neucom.2020.05.081 [3] https://doi.org/10.1016/j.neucom.2020.09.068, [4] https://doi.org/10.1016/j.infrared.2019.103061)

4.       (Section 1. Introduction) The reviewer suggests authors don't list a lot of related tasks directly. It is better to select some representative and related literature or models to introduce with certain logic. For example, the latter model is an improvement on one aspect of the former model.

5.       The sentence is revised as “While Tempotron is not the latest architecture, its various applications [5-8] and enhancements are carried out by numerous research …”(DOI: 10.1109/TII.2021.3128240; https://doi.org/10.1016/j.neucom.2021.03.122; DOI: 10.1109/TII.2019.2934728; DOI: 10.1109/TMM.2021.3081873)

6.       (Section 3, Results) The reviewer suggests to add some compared methods to prove the performance of the proposed method.

7.       (Page 5, Section 3 Proposed Algorithm) In Section 3, the reviewer suggests authors add some formal descriptions of the proposed algorithm, for example add a new figure, so that the reader can better understand the process.

8.       (Page 17, Section 5 Conclusions) Please add some references. The original statement is revised as “… as well as using deep learning methods [9][10], like momentum or AdaDelta are to be addressed in future works.” ([9] https://doi.org/10.1016/j.infrared.2021.103823 [10] https://doi.org/10.1016/j.infrared.2022.104146)

9.       The authors are suggested to add some experiments with the methods proposed in other literatures, then compare these results with yours, rather than just comparing the methods proposed by yourself on different models, such as “Facial expression recognition method with multi-label distribution learning for non-verbal behavior understanding in the classroom”.

10.   (Section 4. Experimental Results) The reviewer suggest authors add more scenarios to compare with the state-of-the-art methods. 

My overall impression of this manuscript is that it is in general well-organized. The work seems interesting and the technical contributions are solid. I would like to check the revised manuscript again.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

In this work, the authors used Caputo derivative-based gradient calculation in Spiking Neural Network training, which had importance to researchers. And this manuscript was considered as a well organized and clearly exposited paper. Thus, I think that it should be accepted for publishing in the present form.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

In this paper, the authors have employed Caputo Derivative on the gradient descendence issue with a two-layer Tempotron SNN model. The cases under study were single-label classification with UCI and MNIST datasets. I would give my comments and suggestions at below,

1. The author pay a lot of efforts in explaining the details of Tempotron Spike Neural Network (SNN) model and the Caputo Derivative on the cost function and the tuning of weighting parameters. They have done a great job in understanding the theoretical background of this study.

2. As compared to studies by Artificial Neural Network (ANN), the authors chose a two-layer network which might be due to the limitation of the computing power of a CPU. In my first thought, this structure is too concise and would have less value for studies in pratical applications.  I would like to advise the authors to give more reasons on the suitability of the selected structure in this study. In particular, can the authors give expectations on possible practical applications based on the findings in this study?

3. Just like epilepsy, there might be disoders in human brains. Since the SNN is defined to behave like real nerons, are there possible worse situations for the input data or the network model that can inspire divergence or oscillation in tuning the parameters or optimizing the hyperparameters in the network under investigation? If yes, I would like to suggest the authors to give comments on the limitation or restriction for the use of Caputo Derivative, PSO or Tempotron network.

4. The caption of Fig. 2 is too long. Some description and explanation can be placed in the paragraph.

5. It is advised to revise the whole picture of this study in the two paragraphs in lines from 83 to 95 to make readers understand this study more smoothly. 

6. The expression in line 308 might be not very clear. Please revise it with some plain centences.

7. The accuracy used in the results is advised to give a definition. Or the authors can briefly describe how to estimate the acuuracy.

Author Response

Please see the attachment.

Author Response File: Author Response.pdf