Alleviating the Communication Bottleneck in Neuromorphic Computing with Custom-Designed Spiking Neural Networks

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper is well written linguistically as well as scientific content wise. I just have some minor comments and suggestions which are listed below.

• Define abbreviation at their first appearance, eg. SNNs in abstract, FPGA in “Introduction, PCI in “2-Related work”. Some of these are defined elsewhere but it is important to define them right at their first-time appearance to salvage the reader from confusion. Also, once defined the authors need to stick to the use of the abbreviation and don’t interchange b/n the abbreviation and its full word from place to place.
• The authors should avoid redundant phrasing such as one in line 103-105 in which “where a network running on Loihi2 without Input/Output (I/O) is capable” is redundantly typed.
• I understand what the authors mean but I recommend them using a more appropriate terminology that can brilliantly represent their scientific work instead of using the term “hand-crafted” which, in my opinion, refers to more of crafting/ designing shapes and producing physical objects which is far less representative of the authors’ real work.
• Since the measure for how good a software or hardware solution in neuromorphic computing is its potential to leverage the time and energy bottle necks, it would be good if the authors can comment on to what extent are the SNNs they are proposing and their training process time and energy efficient compared to others already out there?

Comments for author File: Comments.pdf

Author Response

Comments 1: Define abbreviation at their first appearance, eg. SNNs in abstract, FPGA in “Introduction, PCI in “2-Related work”. Some of these are defined elsewhere but it is important to define them right at their first-time appearance to salvage the reader from confusion. Also, once defined the authors need to stick to the use of the abbreviation and don’t interchange b/n the abbreviation and its full word from place to place.
Response 1: Thank you for the suggestion.  Accordingly, we have: Fixed SNN in the abstract, related work (P3, l98, l115, l130), 
SNN Model (P4 l144), Figure 3 caption, 5.2 first sentence, caption to Figure 5, 
caption to Figure 7, Section 6 (P. 11, l389), Section 6.2.1 (p. 15 l492), section 6.2
(P. 19, l605), Section 7 (P21, l667, 669, and 681), Section 8 (p. 23, l711) and Section 10 (p23,
l756).

We have Fixed FPGA in the abstract, and introduced the abbreviation in section 1.

We have Fixed PCIe in related work.

We have Fixed UART in section 6.2.1.

We have Fixed RISP in section 6.2.1

Comments 2: The authors should avoid redundant phrasing such as one in line 103-105 in which “where a network running on Loihi2 without Input/Output (I/O) is capable” is redundantly typed.
Reponse 2: Thank you for catching that (had a duplicated line in the latex).  Fixed.

Comments 3: I understand what the authors mean but I recommend them using a more appropriate terminology that can brilliantly represent their scientific work instead of using the term “hand-crafted” which, in my opinion, refers to more of crafting/ designing shapes and producing physical objects which is far less representative of the authors’ real work.
Reponse 3: Thank you for paying attention to connotations.  We have changed "hand-crafted" to "custom-designed", so that it doesn't cheapen the work.  We do think that there needs to be a qualifier, because unless we do qualify it, many readers will think that the networks were a result of machine learning.

Comments 4: Since the measure for how good a software or hardware solution in neuromorphic computing is its potential to leverage the time and energy bottle necks, it would be good if the authors can comment on to what extent are the SNNs they are proposing and their training process time and energy efficient compared to others already out there?
Response 4: At the end of sections 6.2 and 6.3, we include power numbers, as produced by Vivado,
for the two applications.  The overall power consumption is increased by the larger SNNs, but
in absolute terms the power consumption is low.  To additionally address the reviewer's comment
we added a paragraph at the end of the related work section that highlights work 
that compares SNNs to ANNs in power:

       Finally, previous research has demonstrated the power efficiency of SNNs compared
       to ANNs, especially when dedicated hardware is employed. For example, Blouw, Choo,
       Hunsberger and Eliasmith demonstrate between 5.3x to 109.1x improvement in energy cost
       per inference when comparing SNNs implemented on Loihi, to ANNs implemented on
       CPU, GPU, Nvidia’s Jetson TX1 and the Movidius Neural Compute Stick [29]. Similarly,
       Vogginger et. al report that neuromorphic hardware is between 3 to 100 times more energy
       efficient than conventional hardware when performing inference [30]. Yan, Bai and
       Wong focus their research specifically of the energy savings of SNNs, demonstrating that
       with careful attention their operational regimes, SNNs outperform quantized ANNs with
       respect to their energy efficiency [31]. 

Reviewer 2 Report

Comments and Suggestions for Authors

This interesting paper focuses on strategies aimed at reducing communication bottlenecks in spiking neural networks. The authors evaluated several specific networks equipped with engineered interfaces, showcasing enhanced efficiency.

The writing is excellent, and the related codes are accessible online. In general, the paper leaves a favorable impression.

However, one area that needs further exploration is the scalability of the suggested methods. It would be beneficial if the authors could provide additional insights into scaling, such as demonstrating speedup relative to network size or other relevant parameters. Although I agree that the proposed methods enhance specific tasks, additional information regarding the general applicability of the approaches is necessary.

Overall, I recommend the publication contingent on addressing my request in the revision.

Author Response

Comments 1: However, one area that needs further exploration is the scalability of the suggested methods. It would be beneficial if the authors could provide additional insights into scaling, such as demonstrating speedup relative to network size or other relevant parameters. Although I agree that the proposed methods enhance specific tasks, additional information regarding the general applicability of the approaches is necessary.
Response 1: Thank you for this suggestion -- we have added three paragraphs on scalability to the end of the Discussion section.  We appreciate the suggestion to explicitly call out scalability to strengthen the work.  The paragraphs read as follows:

      A final discussion point is how the encoding techniques and the SNNs presented in
      section 5 scale. As summarized in Table 1, encoding using time and spike trains incurs
      an overhead of O(M) for values between 0 and M, while value encoding is much more
      efficient, at O(1). Thus, their overheads are functions of the values being encoded, and not,
      for example of the size of the SNN that processes them. Although efficient, the drawback of
      value encoding is that the encoding must be converted to another encoding to communicate
      it to other neurons or to the host.
      
      The conversion networks in Section 5 all run in O(M) time, meaning that from a
      scalability perspective, they are as efficient as they can be. As with the encodings, their
      scalability depends on the values themselves, and not on the SNNs that process them. The
      networks are all O(1) in size, employing fixed numbers of neurons. Therefore, from a size
      perspective, these networks scale very efficiently, without any dependence on the values
      being encoded or the SNNs to which they are attached.
      
      As a final remark involving scalability, there has been research on using a binary
      representation of numbers with spikes, meaning a maximum value of M may be encoded
      with O(logM) spikes. In separate works, Aimone et. al [42] and Wurm et al demonstrate
      SNNs that perform basic arithmetic operations on these binary-encoded spike trains. There
      is less work on leveraging this encoding to train SNNs for more complex tasks; however,
      these encodings may provide an efficient medium for communication information when
      composing SNNs

Reviewer 3 Report

Comments and Suggestions for Authors

The author purposed a noval, hand-craft spiking neural networks to alleviating the communication challenges in neuromorphic computing. This paper has a comprehensive introduction on the previous related works and the storing-communication models. Overall this paper is well-organized and the materials support the conclusion. I have a few comments and questions below.

1, The method discussed in this paper shows a faster speed compared to tranditional method, how about the power consumption? Could the author provide a quantitative calculation on power consunption? 

2, Does the system has a limitation on the number time steps, neuron and synapses used? If so, what's the potential solution to make it "stronger"?

3, For the conversion networks section, it may be helpful to put some of explanation in supplymental materials. In the main text, a conceptual and summarized explanation should be sufficient. 

Author Response

Comments 1: The method discussed in this paper shows a faster speed compared to tranditional method, how about the power consumption? Could the author provide a quantitative calculation on power consunption? 
Reponse 1: Thank you for bringing up this important point.  Please see "response 4" to Reviewer #1, which explains the modifications to the paper to address power consumption.

Comments 2: Does the system has a limitation on the number time steps, neuron and synapses used? If so, what's the potential solution to make it "stronger"?
Response 2: Thank you for this comment.  Table 2 summarizes the time steps, neurons and synapses for each of the conversion networks.  Please also see the three paragraphs added to the end of the discussion that address scalability.  The conversions are O(M) and the networks are O(1), which is optimially efficient because the encodings (time and spike train) are O(M), and value encodings need O(M) to "read" them.  Hopefully the new paragraphs address the reviewer's comments adequately.

Comments 3: For the conversion networks section, it may be helpful to put some of explanation in supplymental materials. In the main text, a conceptual and summarized explanation should be sufficient.
Response 3: This is a very good comment, and I (Plank) agree that a major difficulty in this paper is balancing the detail required to express the techniques precisely and having the amount of detail be overwhelming.  This is why we provide the supplementary material that implements/demonstrates each conversion using open-source software, and then provide video narration of how they work.  We hope that this structure and supplementary material makes it easier for the reader to digest the content without being overwhelmed with detail.