From Spontaneous Ignitions to Sensorimotor Cell Assemblies via Dopamine: A Spiking Neurocomputational Model of Infants’ Hand Action Acquisition

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper is devoted to modelling the process of memorizing successful motor actions. However, from the task formulation perspective, the task which the network solves is similar to associative memory: the network memorizes some patterns of spiking activity that are accompanied by reward, and is then capable of reproducing such patterns. The ability of reward-modulated Hebbian plasticity to achieve such memory is generally well known. The novelty of the paper can therefore only be in deriving hypotheses about the mechanisms of motor babbling from such a network.

The validity of the resulting biological hypotheses, however, is undermined by the lack of an ablation study. The network topology - the areas the network comprises, the inter-area connections, the intra-area inhibitory connections - are chosen so that to match existing body of biological knowledge as accurately as possible. Which of the components of this overcomplicated network are necessary for accomplishing the learning task, remains unexplored, while this is what would, in my opinion, constitute a valuable biological hypothesis.

The methodology is sound, except some details that remain unclear:

• At which particular time moment reward is applied, is not stated explicitly. This is important because one might expect the distal reward problem: if the reward is applied (which means, the plasticity amplitude is boosted) after observing the activity of the desired cell assembly, then that reward will facilitate learning the next cell assembly rather than the current, desired one. In the literature, the distal reward problem is usually addressed by employing eligibility trace in the plasticity model. How the proposed model achieves learning without eligibility trace, is interesting.
• The particular quantitative parameters of the network model (such as the number of neurons and the number of CAs) are not discussed. While these values must have been based on biological observations, the relation between them would be an interesting question, shedding light on the network capacity of the network.

Upon addressing the above research design issues, the paper can, in my opinion, be reconsidered.

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The authors study the neuronal correlations of learning and spontaneous acquisition of skilled hand-action. Through a first phase of random hand movements, the system discovers to generate several different actions. After this, an association period is used to associate hand-action related neuronal activation with rewards. Reactivation of neuronal populations becomes more frequent as their association with reward increases. The overall idea of the paper comes from theories of motor babbling. The authors provide a plausible neuronal instantiation of this skill discovery and further refinement and consolidation through reinforcement learning.

Main comments:

-I think that the idea is correct, but the modelling is very limited in that there is no actual hand-movement being modelled in any detail. At the very least, the paper should acknowledge from the very beginning, including in the abstract, that no modelling of movements is used, and that only an abstract representation of motor plans is considered without any reference to muscle contractions of hand trajectories. As it is written, it gives the impression that fine motor movements are considered.

-A basic question is how infants generate their own training data based on motor babbling. I think that random sampling might not be enough, as what is really needed is the coverage of action space; further the discretization has to be optimized to maximize the opportunity of discovering useful skills. A very related paper that learns discrete primitive tokens is

https://openreview.net/forum?id=HhbHw2yInZ

A further discussion between unsupervised skill discovery and random skill discovery proposed by the authors is very important.

-In addition, it seems important to mention in the discussion how the authors’ work relates to new theories of intrinsic motivation where the core idea is skilled coverage of action space, rather than random sampling.

-The assumption of i.i.d. gaussian noise needs to be clarified and further justified.

-I think that there must be an error in Eqs. 3,4 as the pulse should not be a 1, but rather a delta function. Only the integral of a delta gives a non-zero increase in the voltage. The reset mechanism of the IF neuron, if any, should be clarified.

-How is the clamping justified functionally? The clamping of neural activity seems to be important for learning, but it is not justified.

 

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

Please see the comments in the attached file.

Comments for author File: Comments.pdf

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have decided that an extensive ablation study of the role of particular network components in accomplishing hand-action acquisition is beyond the scope of the paper. This somewhat limits the significance of the results of the paper, but it still has undoubted merit.

The only remark remaining is that the authors should summarize more clearly and explicitly what biological hypothesis about the mechanisms of motor babbling can be formed thanks to the proposed model.

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have responded to my comments. However, they have not yet addressed a comparison with unsurpervised discovery of behaviors, as in

https://openreview.net/forum?id=HhbHw2yInZ

I think it is important to acknolwedge that there is some work already describing the posibility of discovering discrete behavior throught exploration and motor babling, as indicated in the quoted paper.

There is also an extended literature that needs to be addressed.

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

The comments have been well handled.

Author Response

Please see attachment.

Author Response File: Author Response.pdf

Round 3

Reviewer 2 Report

Comments and Suggestions for Authors

The authors have appropriatelly addressed all my comments.

Some of the new references do not indicate the journals or venue.

Author Response

Please see attachment.

Author Response File: Author Response.pdf