Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations

Round 1

Reviewer 1 Report

I have the following observations:

1.     This paper deal with Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations

2.     Please insert in References a more papers from last 5 years.

3.     Please make reference to [7--12] references in order.

4.     Please give references at beginning of Definitions and of each new Equations set.

Comments for author File: Comments.docx

Author Response

The authors would like to thank the reviewers for carefully reading this manuscript and providing useful comments to improve the quality of this manuscript. The authors are more than happy to answer all questions raised by the reviewers. The authors also hope that their replies can respond to the reviewers’ comments directly.

Reviewer 1:

This paper deal with Separable Gaussian Neural Networks: Structure, Analysis, and Function Approximations.

Comment 1: Please insert in References a more papers from last 5 years.

Answer: Thanks for the reviewer’s suggestion. The following three papers were added to references.

[1] Buvanesvari, R.M.; Joseph, K.S. RBFNN: a radial basis function neural network model for detecting and mitigating the cache pollution attacks in named data networking. IET Networks 2020, 9. 443

[2] Du, J.; Zhang, J.; Yang, L.; Li, X.; Guo, L.; Song, L. Mechanism Analysis and Self-Adaptive RBFNN Based Hybrid Soft Sensor Model in Energy Production Process: A Case Study. Sensors 2022, 22, 1333 (16 pages).

[3] Zheng, S.; Feng, R. A variable projection method for the general radial basis function neural network. Applied Mathematics and Computation 2023, 451, 128009 (17 pages).

Comment 2: Please make reference to [7--12] references in order.

Answer: Thanks for catching it. The reference indices have been fixed.

Comment 3: Please give references at beginning of Definitions and of each new Equations set.

Answer: Thanks for the reviewer’s comment. Definition 2 is new.  Definition 1 is a widely accepted concept in mathematics. We have not found a specific reference giving the definition of it.

Comment 4: Figures 2,3,4, and 5, which equations are precisely used in the derivation to plot these results in figures.

Answer: The equations for Figs. 2-5 are indicated in the captions. The explicit forms are listed in table 2.

Reviewer 2 Report

The paper presents SGNN as an alternative to the Gaussian-radial-basis function neural network (GRBFNN), in the hope of mitigating the computational intensity associated with high-dimensional input vectors in the GRBFNN. The originality is not major but within the scope of this publication.

The SGNN shows promising results in the computational intensity associated with high-dimensional input vectors, and it provides a viable alternative to the GRBFNN with improvements in both speed and accuracy. However, a more extensive comparative analysis with various neural network architectures would provide a more comprehensive evaluation of its capabilities.

It’s suggested to describe the difference between this work and existing works in the first figure.

The method employed certainly has trade-off, which should also be discussed. 

Author Response

Reviewer 2:

The paper presents SGNN as an alternative to the Gaussian-radial-basis function neural network (GRBFNN), in the hope of mitigating the computational intensity associated with high-dimensional input vectors in the GRBFNN. The originality is not major but within the scope of this publication.

The SGNN shows promising results in the computational intensity associated with high-dimensional input vectors, and it provides a viable alternative to the GRBFNN with improvements in both speed and accuracy.

Comment 1: However, a more extensive comparative analysis with various neural network architectures would provide a more comprehensive evaluation of its capabilities.

Answer: Thanks for the reviewer’s comments. Indeed, a more comprehensive evaluation should be performed. The aim of this paper is to develop a new NN to address the curse of dimensionality of GRBFNN. Therefore, the comparison has been performed mainly between SGNN and GRBFNN. In our future work, more comprehensive evaluations will be studied.

Comment 2: It’s suggested to describe the difference between this work and existing works in the first figure.

Answer: More description has been added to the first figure to describe the difference between SGNN and MLP.

The current caption is

The general structure of SGNNs. {\color{red}The distinct feature of the SGNN is that} input is divided and fed sequentially to {\color{red}hidden} layers.  Therefore, the depth (layers) of SGNNs is identical to the number of input dimensions. {\color{red}Each neuron in hidden layers is associated with an univariate Gaussian function. Each path in feedforward propagation will lead to a chain of multiplications of univariate Gaussian functions, equivalent to a $d$-dimensional Gaussian radial-basis function shown in Eq. (\ref{eq:product_chain}). In other words, each SGNN can be converted to a RGBFNN.} In this paper, the weights of the output layer are unity.

Comment 3: The method employed certainly has trade-off, which should also be discussed. 

Answer: Thanks for the reviewer’s insightful comment. Yes, the method certainly has trade-off, which is stated in section 3, SGNN VS GRBFNN.

In general, SGNN would have much fewer parameters than GRBFNN, which means for most possible GRFNN networks there is no equivalent SGNN network with an identical set of centers and widths. It is unclear at this time whether SGNNs can form a dense sub-set of GRBFNNs. The aim of this paper is to show that SGNN can yield substantially less computational effort than GRBFNN but provide comparable (occasionally even greater) accuracy through extensive numerical experiments in modeling ten very different functions. In this regard, even if SGNNs cannot lead to arbitrarily close approximation to GRBFFs, there is still value in using them for high-dimensional problems due to their computational efficiency.

In this paper, the comparison has been concentrated between SGNN and GRBFNN. In future studies, we shall focus more on the comparison between SGNN and other networks. The trade-off will be discussed in more detail.

Reviewer 3 Report

The authors introduce in the manuscript a factorization of Gaussian RBF neural networks as a product of independent scalar Gaussians, which has the effect of overcoming the curse of dimensionality in the vector-based RBF networks. While the factorization represents a subclass of the class of  GRBF neural networks, it uses far less neurons and is much easier to train without showing substantial performance loss. The work is original, valuable, and is clearly presented and organized.

1) Technically, the mathematics are fairly simple and easy to follow. Only a few minor typos have been detected (e.g., subscript \ell in (7))

2) The paper focuses on a particular type of decomposition (a product of scalar pdfs), equivalently, a completely separable Gaussian vector. It would be very interesting for future work to extend this analysis to other forms of factorization or marginalization, to capture more complex correlations, egg, f(x_1, x_2, f_3) = g(x_1, x_2) · h(x_3). Actually, any form of a factor graph as those used in Bayesian networks or in graph-based codes).

The paper is technically correct, interesting, and makes a good contribution. Aside from minor corrections, I find that it is ready for publication. 

Author Response

Reviewer 3:

The authors introduce in the manuscript a factorization of Gaussian RBF neural networks as a product of independent scalar Gaussians, which has the effect of overcoming the curse of dimensionality in the vector-based RBF networks. While the factorization represents a subclass of the class of GRBF neural networks, it uses far less neurons and is much easier to train without showing substantial performance loss. The work is original, valuable, and is clearly presented and organized.

Comment 1. Technically, the mathematics are fairly simple and easy to follow. Only a few minor typos have been detected (e.g., subscript \ell in (7))

Answer: We thank the reviewer for the positive comments. The typo in Eq. (7) has been fixed. We also checked other equations.

Comment 2. The paper focuses on a particular type of decomposition (a product of scalar pdfs), equivalently, a completely separable Gaussian vector. It would be very interesting for future work to extend this analysis to other forms of factorization or marginalization, to capture more complex correlations, eg, f(x_1, x_2, f_3) = g(x_1, x_2) · h(x_3). Actually, any form of a factor graph as those used in Bayesian networks or in graph-based codes).

Answer: We thank the reviewer for this insightful suggestion for our future work. It would be very interesting to investigate the networks formed by other forms of factorization. Indeed, such forms will lead to graph/tree networks. It is worthwhile to pay more attention to such structures as it may directly link to the mitigation of the curse of dimensionality.

Comment 3. The paper is technically correct, interesting, and makes a good contribution. Aside from minor corrections, I find that it is ready for publication. 

Answer: We thank the reviewer for the insightful comments again.

Reviewer 4 Report

The paper is a very interesting paper which presents a separable Gaussian neural network approach. The main purpose of the current approach is to reduce the complexity of this type of NN.

The introduction partmay be strengthen by adding some more comments about the application of GRBFNN.

The number of input layers in Figure 1 is just 3. It would be interesting if you provide a more general structure with ad hoc number of input layers. The number of neurons in each layer need to be more general as well. Probably adding some dots can be useful.

Is there a possibility to extend the current approach to run on a GPU rather than CPU? Please add some discussions.

Author Response

Reviewer 4:

The paper is a very interesting paper which presents a separable Gaussian neural network approach. The main purpose of the current approach is to reduce the complexity of this type of NN.

Comment 1: The introduction part may be strengthened by adding some more comments about the application of GRBFNN.

Answer: Thanks for the reviewer’s comment. We have added some short comments about the applications of GRBFNN in the introduction.

Comment 2: The number of input layers in Figure 1 is just 3. It would be interesting if you provide a more general structure with ad hoc number of input layers. The number of neurons in each layer need to be more general as well. Probably adding some dots can be useful.

Answer: A very good suggestion. We have modified Figure 1 to a more general structure.

Comment 3: Is there a possibility to extend the current approach to run on a GPU rather than CPU? Please add some discussions.

Answer: Yes. The network has been deployed in Tensorflow. It can be run seamlessly on a GPU.