The Cart-Pole Application as a Benchmark for Neuromorphic Computing

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The Cart-Pole Application as a Benchmark for Neuromorphic Computing

After reviewing the article, I identified several areas that require improvement. These shortcomings are outlined as follows:

 

1. The abstract's motivation section should be rewritten for greater clarity and conciseness.
2. Section 2 should be appended to the end of the Introduction to enhance logical flow.
3. The rationale for selecting this model and the methods used to validate its reliability should be elucidated.
4. The Discussion and Limitations sections are missing; these should be included to provide a comprehensive analysis of the study's findings and implications.

Author Response

Comment 1: The abstract's motivation section should be rewritten for greater clarity and conciseness.
Response 1: Thank you for this comment -- sometimes we forget that not everyone is as entrenched in neuromorphic computing as we are.  We have changed sentence 3 of the abstract to the following, to both motivate why we care about neuromorphic computing, and why we want to explore the cart-pole problem as a benchmark: "Spiking neural networks are the basis of brain-based, or neuromorphic computing.  They are attractive, especially as agents for control applications, because of their very low size, weight and power requirements.  We are motivated to help researchers in neuromorphic computing to be able to compare their work with common benchmarks, and in this paper we explore using the cart-pole application as a benchmark for spiking neural networks."

Comment 2: Section 2 should be appended to the end of the Introduction to enhance logical flow.  
Response 2: We have turned it from a section to a subsection, so that it is now part of the introduction, yet is still broken out for clarity.

Comment 3: The rationale for selecting this model and the methods used to validate its reliability should be elucidated.
Response 3: We have added material to three parts of the paper to address this comment.  First, we have added the following paragraph as the second paragraph of section 4 (Spiking Neural Networks) to 
specify our rationale for focusing on spiking neural networks: "We study spiking neural networks as they apply to control applications like the cart-pole problem for two main reasons. First, their temporal operation is a natural fit for the temporal application. Second, their hardware implementations can be very simple, resulting in very low size, weight and power requirements [19–22], which are ideal for control agents in the field."

Second, we have provided rationale for why we focus on the RISP model.  This is the last paragraph of section 4.3: "Our rationale for focusing on RISP in this work has several components. First, because of its limited nature, RISP may be viewed as a lingua franca for neuroprocessors — SNN’s developed for RISP may be ported easily to other more complicated neuroprocessors, such as Loihi [19], Nengo [32] NEST [33] or Brian [34]. Second, because of its simplicity, it is easier to reason about RISP networks and their functionalities than with more complicated neuron models. We do this in Section 8 below. Finally RISP has open-source support for both CPU simulation and FPGA implementation [17], making it an ideal neuroprocessor for embedded control applications like the cart-pole application."

Last, we have provided rationale for why we train with a genetic algorithm in Section 5: "For the training in this experiment, we use the EONS genetic algorithm [35] implemented in the TENNLab exploratory neuromorphic computing framework [12,13]. We chose this training methodology because of its previous success in training effective, but very small networks for the cart-pole problem [16,28]. Other training methodologies, particularly those that rely on backpropagation, result in vastly larger spiking neural networks [36–38]. Since independent EONS runs are massively parallel, we employ a large variety of processors for training, including Linux, Macintosh and Raspberry Pi computers of all vintages. Unless otherwise specified, for each test, we run 100 independent optimizations."

Comment 4: The Discussion and Limitations sections are missing; these should be included to provide a comprehensive analysis of the study's findings and implications.
Response 4: We have added both sections as recommended.  Thank you.  They are included below:

9. Discussion
For a problem to be an effective benchmark, it should first and foremost be widely available, either simple enough for researchers to implement on their own, or having an open-source implementation that is easy to port. The cart-pole application succeeds on this 
 front, particularly with its open-source implementation in OpenAI Gym [8], which makes it ubiquitously available. Second, it should provide a variety of challenge levels, from a simple level, such as the Easy setting described here, which serves as a first-level screen and proof-of-concept, to more challenging levels that stress both features of the AI agent and the training methodologies employed. The other three parameter settings provide this challenge. Last, it should be easy to understand, so that the results have concrete meaning to the observer. The cart-pole problem succeeds in this aspect as well, as visualizations of the application are widely available (as part of OpenAI gym and otherwise), and the fitness metrics of mission time and optionally a prevalence of “do-nothing” actions are quite intuitive. Even with its simplicity, performing experimentation with the cart-pole problem is not without challenges. Important parameter selections, such as encoding and decoding methodologies, must be made, and as shown in Section 6, they may impact performance drastically. Similarly, there are an enormous amount of hyperparameters that must be set, which impact both the success of training and how long it takes. Please see the appendix for examples. Regardless, the application can be successful in helping to assess the impact of features of a spiking neural network. In Section 7, we perform such an experiment which demonstrates the impact of inhibitory synapses, range of neuron potentials and synapse weights, and excessive synapse delay, on the effectiveness of training spiking neural networks for the various benchmark levels.
10. Limitations
The experimental work in this paper is inherently limited by the parameter and hyperparameter selections that we made. Although we have attempted to justify these selections with initial experiments and reference to previous work, it remains a fact that there could be other parameter settings that improve results, or that make them more applicable to other systems. Fortunately, our motivation is to demonstrate how the cart-pole problem may be employed to evaluate various neuroprocessors, and as such does not have to perform a complete assessment. It is our intent for this work to pave the way for other researchers to perform additional assessments.

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authors:

Thanks for your efforts.

The work proposes four parameter settings that scale the application in difficulty, in particular beyond the default parameter settings which do not pose a difficult test for AI agents.

Here are some questions:

Q1: Any flowchart or overview of methodology in the beginning of paper?

Q2: For Spiking Neural Networks (SNN), any structural graph for SNN with your proposed approach?

Q3: For Spike Encoders: Beside VAL encoding, others are all discrete spikes? VAL encoding is continuous spikes? Why is it so? Any justification?

Q4: In Figure 2: What are the parameters of each benchmark model respectively?

Q5: In Table 1: What are these parameters? Any definition?

Q6: In Figures of networks: What are the meanings of symbols, like dx-, dx+, theta-, theta+, dtheta-, dtheta+, x-, x+, L, R and H?

 

Thanks and Best Regards

Author Response

Comment 1: Any flowchart or overview of methodology in the beginning of paper?
Response 1: First, thank you for your comments to help improve this paper.  We have added a flowchart of the tasks of the paper as Figure 1.

Comment 2: For Spiking Neural Networks (SNN), any structural graph for SNN with your proposed approach?
Response 2: EONS trains unstructured networks.  We have added the following to Section 5 (Training) to address this point of confusion: "EONs trains unstructured networks, with the synaptic connections being part of the training process. This is what allows the networks to be much smaller than with other training techniques. As such, there are no structural models as there are with, for example, deep learning networks — the structure of the network is part of the training process."

Comment 3: For Spike Encoders: Beside VAL encoding, others are all discrete spikes? VAL encoding is continuous spikes? Why is it so? Any justification?
Response 3: We have changed the specification of "Values" to the following, to help answer this question: "Values: A single spike is applied to the input neuron.  The weight of the spike is scaled 261 by the value being encoded, potentially after rounding if the neuroprocessor only 262 accepts discrete values." The justification for these encodings is beyond the scope of this paper, but has been addressed in other works.  We have changed the sentence before section 4.1.1 to reflect this: "We refer the reader to [16,28] for more complete definitions of spike encoding, plus experimental evaluations of encoding techniques."

Comment 4: In Figure 2: What are the parameters of each benchmark model respectively?
Response 4: Those parameters have been explained in section 3: "The Benchmarks and Performance Goals".  In terms of the networks, EONS defines the structure.  We provide examples in section 8: "Example Networks".

Question 5: In Table 1: What are these parameters? Any definition?
Response 5: We've added extra description for the table labels (e.g. "Neuron Threshold Range"
instead of "Threshold Range").  We did not want to duplicate material from [17], to keep this
paper from being even longer than it is.   Hopefully those more detailed labels help to alleviate
confusion.

Question 6: Q6: In Figures of networks: What are the meanings of symbols, like dx-, dx+, theta-, theta+, dtheta-, dtheta+, x-, x+, L, R and H?
Response 6: Thank you for this comment as I'm sure others will be confused about it as well.  We have added the following paragraph just before section 8.1: "In the networks drawings, the labels x, dx, θ and dθ denote the four input parameters described in Section 2. Flip-flop encoders use two neurons for each value, so for example, x− is used for negative values of x and x+ is used for positive values. The neurons labeled L, R and − denote the actions “push-left,” “push-right,” and “do-nothing.” Neurons labeled H are hidden neurons."

Reviewer 3 Report

Comments and Suggestions for Authors

The paper presented a study of applying spike neural networks (SNNs) to benchmark the famous Cart-Pole application (normally solved with reinforcement learning) in four difficulty settings (Easy, Medium, Hard, and Hardest). Although there are no theoretical contributions to neuromorphic computing, this paper employed existing techniques to solve control application problems. Here are the strengths of the paper:

A. Sections 1 and 2 provide enough information about control application, neuromorphic computing, the cart-pole problem, and a brief summary of this research and the paper's contributions.   

B. Sections 3 and 4 provide more detail of the Cart-Pole and the four difficulty settings (Easy, Medium, Hard, and Hardest) to be formed as the benchmarks in the research.

C. Sections 5-8 presented the theoretical Spike Neural Network (SNNs), how the models can be trained for this research, and how the experiments can be performed on Neuroprocessor RISP for this research. The authors explained the encoding/decoding models and hyperparameters tuning process well.

D. In section 9, the authors provided good SNNs examples to solve each difficulty level with model diagrams and performance timelines.

 

E. The authors provided a detailed literature review in section 10 and a good summary of this research, including the methodology, experimental setup, results, and discussions.

Comments for author File: Comments.pdf

Comments on the Quality of English Language

However, there is one minor problem which should be addressed:

F. Many places in this paper should be improved in written English or formats. Here are some common problems:

- Misspelling or word choices: et al. (not et al) at lines 60, 240, 328, and 606; multiple (63); affect (line 99); neuromorphic (line 240); proportional (276); arithmetic (384), information (422); ensure (444);

- Missing articles, such as a/the, or punctuations between sentence phrases

- Verb tenses.

- Singular or plural words.

Please check the attached PDF file with the detailed highlighted comments to improve the writing of this article.

Author Response

Comment A: Sections 1 and 2 provide enough information about control application, neuromorphic computing, the cart-pole problem, and a brief summary of this research and the paper's contributions.
Response A: Thank you.  No response required.

Comment B. Sections 3 and 4 provide more detail of the Cart-Pole and the four difficulty settings (Easy, Medium, Hard, and Hardest) to be formed as the benchmarks in the research.
Response B: No response required.

Comment C. Sections 5-8 presented the theoretical Spike Neural Network (SNNs), how the models can be trained for this research, and how the experiments can be performed on Neuroprocessor RISP for this research. The authors explained the encoding/decoding models and hyperparameters tuning process well.
Response C: No response required.

Comment D. In section 9, the authors provided good SNNs examples to solve each difficulty level with model diagrams and performance timelines.
Response D: No response required.

Comment E. The authors provided a detailed literature review in section 10 and a good summary of this research, including the methodology, experimental setup, results, and discussions.
Response E: No response required.

Comment F. Many places in this paper should be improved in written English or formats.
Response F: Thank you for taking the time to pay attention in this level of detail.  I (Plank) strive to have every paper I write communicate clearly with very good English, so I do appreciate all of your detailed comments, and I apologize that you had to find typos and grammatical mistakes.  I address each comment from the marked up PDF below.

- P1 L6: s/trained on/trained in/ - Actually, I think "trained with" sounds better than either, so that's what I've done.
- P1 L21: s/application/application,/ - Done
- P1 L28: "low size, weight, and/or power solutions" -- sugggestion s/low/small/.  I use low, because usually one says "low SWaP", but I get it.  I think "reduced" sounds better than "low" or "small", so that's what I've done.
- P2 L39: s/for/for,/  -- Sorry, but I disagree and think it reads better without the comma.
- P2 L43: s/are/is/ - done
- P2 L49: s/pole angle velocity/and pole angle velocity/ - done
- P2 L53: s/mutiple/multiple/ - done.  Thank you.
- P2 L60: s/al/al./ - fixed everywhere.
- P3 L99: s/effect/affect/ - done.  Thank you.
- P3 L106: s/such as way/such a way/ - done.  Thank you.
- P4 L138: s/python/Python/ - done.
- P4 L142: s/Below/Below,/ - done.
- P4 L157: s/angle/angle,/ - you know, I had a co-author who advocated that in another paper -- I was taught that you don't include the comma here.  Grammarly calls it the "Oxford" comma and says "Some style guides call for using it, while others call for leaving it out."  I can't say I'm passionate about it, so I'll put it in.
- P4 L166: s/,/ and/ - done.
- P4 L180: s/Were the activity threshold 0.25, then the fitness would be 500/If the activity threshold is 0.25, then the fitness is 500/ - sure.  done.
- P5 L199: s/for which to train/for training/ - done.
- P5 L221: s/is performed/are performed/ - done.  Thank you.
- P6 L240: s/neuromophic/neuromorphic/ - done.  Thank you.
- P6 L242: s/can/could/ - sorry, I disagree.  The experiment was in the past, but the conclusion applies to the present.
- P6 L274: s/partitions/partition/ - done. Thank you.
- P6 L276: s/porportional/proportional/ - done.  Thank you.
- P6 L283: s/into/in/ - sorry, but I prefer "into".
- P7 L300: s/inputs/input/ - done.
- P7 L300: s/allowing neurons to be configured with all-or-nothing leak/allowing neurons to be configured with all-or-nothing leaks/ - Changing to "allowing each neuron to be configured to leak all of its potential at each timestep."  I deleted the subsequent sentence.
- P9 L373: s/section/section,/ - done.
- P9 L380: s/thresholds/thresholds,/ - here I think it reads better without the Oxford comma.
- P9 L382: s/therefore/therefore,/ - when I speak, the comma is not there, but I get it -- I've put it in.
- P9 L384: s/arithemtic/arithmetic/ - done.  Thank you.
- P10 L396: s/worst/worst,/ - done.
- P11 L416: s/ and/,/ - sorry, that should be "and", but I see it's convoluted.  Changed the sentence to: "In these graphs, RISP-127 fits naturally between RISP-7 and RISP-F.  Similarly, RISP-255+ fits naturally between RISP-15+ and RISP-F+."
- P11 L423: s/infomation/information/ - done.  Thank you.
- P12 L441: s/listing/listed/ - done.  Thank you.
- P12 L444: s/assure/ensure/ - nice.  Thanks.
- P13 L491: s/9/9,/ - done.  Thank you.
- P14 Caption to Figure 9: s/synapses/synapse/ - done.  Thank you.
- P15 L532: s/11/11,/ - done.  Thank you.
- P15 Caption to Figure 11: s/which/that/g - Done.
- P16 L566: s/13/13,/ - done.  Thank you.
- P17 L605: s/thie/this/ - done.  Thank you.
- P17 L619: s/to/of/ - done.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

Dear Authers:

Thanks for the efforts. 

The current version is fine by me.

Thanks and Best Regards