Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-Based Generative Adversarial Network

Round 1

Reviewer 1 Report

1. Full forms of the keywords should be provided when they appear first in the abstract and body of the script.

2. In Fig. 11 test accuracy crossed training accuracy means over-fitting of the network, How you are justifying this?

3. ROC and precision-recall curves are missed

4. Spelling mistakes need to be rectified

5.  Compare the proposed method with standard deep learning architectures

6. Write contributions in a separate section

7. Cite the following recent works

Liang, Zhen, et al. "Characterization of electroencephalography signals for estimating saliency features in videos." Neural Networks 105 (2018): 52-64.

Rundo, Francesco, Roberto Leotta, and Sebastiano Battiato. "Real-Time Deep Neuro-Vision Embedded Processing System for Saliency-based Car Driving Safety Monitoring." 2021 4th International Conference on Circuits, Systems and Simulation (ICCSS). IEEE, 2021.

Allam, Jaya Prakash, et al. "Customized deep learning algorithm for drowsiness detection using single-channel EEG signal." Artificial Intelligence-Based Brain-Computer Interface. Academic Press, 2022. 189-201.

Author Response

Reviewer#1:

• ⎫ While thanking the esteemed reviewer for a thorough review of the manuscript, we believe that your suggestions have been very useful and effective in improving the scientific version of the manuscript. We carefully answered all the questions and suggestions of the esteemed reviewer and added them to the manuscript.

Comment 1: Full forms of the keywords should be provided when they appear first in the abstract and body of the script.

Answer: The manuscript is revised based on this comment. According to the opinion of the respected reviewer, the full form of abbreviated words is presented for the first time in the abstract. Also, the keyword form was modified as requested which is highlighted in page 1 and lines 6 & 7.

• ⎫ “In this paper, a deep model is proposed to reconstruct the image stimuli from electroencephalogram (EEG) recordings via visual saliency. To this end, the proposed geometric deep network-based generative adversarial network (GDN-GAN) is trained to map the EEG signals to the visual saliency maps corresponding to each image.”

Comment 2: In Fig. 11 test accuracy crossed training accuracy means over-fitting of the network, How you are justifying this?

Answer: The iterated 10-fold cross-validation has been used to train the model. The test fold is not a fixed part of the data and in every iteration, the test fold changes to cover testing among the whole dataset. The overall accuracy and the reported accuracy is a mean average over all dataset after training, not a fixed part of the dataset. Also, the confusion matrix that illustrates the predicted class versus true class is obtained among the whole 11965 128-channel EEG recordings of the EEG-ImageNet dataset. This illustration in Fig.14 is a good confirmation of the performance of the proposed GDN method. The explanations related to this figure is added to the manuscript which is highlighted in page 21 and lines 455-459.

Comment 2: ROC and precision-recall curves are missed.

Answer: Yes, the reviewer's opinion is absolutely correct. Accordingly, in accordance with the opinion of the respected reviewer, ROC curves and the precision-recall scores of different methods have been added to the manuscript for further analysis of the proposed network which are highlighted in pages 19-20 and lines 435-441.

• ⎫ “ Fig. 12 shows the receiver operating characteristic (ROC) plot for GDN part of the proposed GDN-GAN and other state-of-the-art methods for classification of the EEG-ImageNet dataset including region-level stacked bi-directional LSTMs [47], stacked LSTMs [48] and siamese network [39]. The superiority of the GDN in terms of the area under the ROC can be seen in this figure compared to the other existing methods. Furthermore, performance of the GDN against the above-mentioned state-of-the-art methods in terms of precision, F1-score and recall metrics is shown in Table 5.”

Furthermore, confusion matrix of the proposed method for classification of 40 different categories of EEG-ImageNet dataset is added to the manuscript. The confusion is obtained among the whole dataset containing 11965 128-channel EEG recordings after training the proposed GDN network.

The explanation related to figure of confusion matrix is highlighted in lines 455-459.

• ⎫ “A good confirmation to the performance of the proposed method is the confusion matrix shown in Fig.14. Confusion matrix is an appropriate illustration of the performance of a network on test splits in case of multi-class classification. Fig. 14 shows the confusion matrix of the GDN part of the proposed method. This Figure confirms the good performance of the classification part of the GDN-GAN.” 

Comment 3: Spelling mistakes need to be rectified.

Answer: The manuscript is revised based on this comment. According to the reviewer's opinion, the manuscript was read again and the spelling and grammar errors were corrected and highlighted in the manuscript. Fixed spelling errors include the following:

• • Convolutoion = Convolution in page 6  , line 216.
• • kernal size = The kernel size in page 13 , line 364 .
• • saliency = saliency in page 1, lines 4&8.
• • brian = brain in page 1, line 2.

Comment 4: Compare the proposed method with standard deep learning architectures

Answer: The manuscript is revised based on this comment. According to the reviewer's opinion, the proposed network architecture was compared with the standard deep network architecture in the discussion section which is highlighted in pages 19-20 and lines 442-454.

• ⎫ “To demonstrate the efficiency of the GDN, we compare the performance of our method with traditional feature-based CNN and MLP. For this purpose, three hidden layers for MLP and CNN with a learning rate of 0.001 have been considered. Maximum, skewness, variance, minimum, mean, and kurtosis have been used as feature vectors for every single channel. According to Fig. 13, feature-based traditional deep networks such as MLP, CNN and feaure-based GDN have a poor performance in the case of classification of the EEG-ImageNet dataset with 40 different categories. This figure shows the obtained accuracy of feature-based MLP, CNN , GDN in 50 iterations. This figure illustrates that feature-based GDN, MLP and CNN have shown relatively similar performance. This figure confirms the good efficiency of the proposed GDN against the traditional feature-based deep networks. According to this figure, the proposed network has high convergence speed and high classification accuracy compared to the other networks.”

Comment 5: Write contributions in a separate section.

Answer: The manuscript is revised based on this comment. According to the opinion of the respected reviewer, the contribution of the manuscript is as follows:

1. (i) It provides an efficient deep network to extract saliency map of visual stimulifrom visually provoked EEG signals.
2. (ii) Reconstruction of the visual stimuli is possible through the proposed deep network.
3. (iii) It provides a geometric visual decoding network for extracting features from the EEG recordings to identify 40 different patterns of EEG signals corresponding to 40 image categories.
4. (iv) Graph representation of EEG channels is imposed as input to the proposed GDN-GAN in which functional connectivity between 128 EEG channels is employed to construct the graph.
5. (v) In the proposed method, the time samples of EEG channels are used directly as the graph nodes to remove the feature extraction phase and decrease the computational burden.
6. (vi) For the first time, it presents a model to connect the EEG recordings, visual saliency and visual stimuli together.
7. (vii) For the first time, it proposes a fine-tuning process to realize image reconstruction from EEG signals via visual saliency reconstruction.

which has been added separately to the introduction section (page 2 and lines 59-76) at the request of the respected reviewer.

Comment 6: Cite the following recent works:

• • Liang, Zhen, et al. "Characterization of electroencephalography signals for estimating saliency features in videos." Neural Networks 105 (2018): 52-64.
• • Rundo, Francesco, Roberto Leotta, and Sebastiano Battiato. "Real-Time Deep Neuro-Vision Embedded Processing System for Saliency-based Car Driving Safety Monitoring." 2021 4th International Conference on Circuits, Systems and Simulation (ICCSS). IEEE, 2021.
• • Allam, Jaya Prakash, et al. "Customized deep learning algorithm for drowsiness detection using single-channel EEG signal." Artificial Intelligence-Based Brain-Computer Interface. Academic Press, 2022. 189-201.

Answer: The manuscript is revised based on this comment. The first study by Liang et al. is cited in the manuscript in reference 44. The other recent studies recommended by the esteemed reviewer have been added to the manuscript which are highlighted in references 5 and 4.

Author Response File: Author Response.pdf

Reviewer 2 Report

This paper reports the results of an exciting study. Overall, the manuscript is well written. The background, the related work, and the methods are well presented. However, regarding the results section, I would suggest elaborating more on the results of the proposed method. Several tables (e.g. Tables 5 and 6) provide validation and performance metrics data. However, the results were not discussed adequately. The authors must explain the meaning of these results and how the proposed method outperforms the state-of-the-art methods. Although the numbers in the tables provide facts, these should be explained in the text. Moreover, a section discussing limitations and threats to validity and reliability is missing.

In line 345, the authors refer to table 6. Should it probably be a reference to Table 2? In the first paragraph of Section 5, the number of the figure the authors are referring to is missing - see Line 412. 

Although the manuscript is well written, there are some typos, which is why I recommend proofreading the text. For example:

• In the abstract, for the word "saliency", we can find "saliecny" (in Line 4) and "slaiency" (in Line 7). In Line 44, the word "slaient" can be found. 
• The statement in Line 143 should be checked: "Each of the layers with ReLU and a pooling layer." 
• In Line 453, "The kernal size..." should probably be "The kernel size..." 
• In the title of section 3.2. the authors wrote "Chebyshev Graph Convolutoion" Convolutoion -> Convolution
• Next, for example, the statement that starts in Line 79 should be checked: "Visual attention is a selective procedure occurs for understanding the visual area input."

Author Response

Reviewer#2:

Comment 1: This paper reports the results of an exciting study. Overall, the manuscript is well written. The background, the related work, and the methods are well presented. However, regarding the results section, I would suggest elaborating more on the results of the proposed method. Several tables (e.g. Tables 5 and 6) provide validation and performance metrics data. However, the results were not discussed adequately. The authors must explain the meaning of these results and how the proposed method outperforms the state-of-the-art methods. Although the numbers in the tables provide facts, these should be explained in the text. Moreover, a section discussing limitations and threats to validity and reliability is missing.

• ⎫ We thank you for your thorough review of this manuscript and we appreciate your valuable suggestions and precise point of view that have led to improvement in our manuscript. We have considered your comments one by one and revised the manuscript accordingly. The added sentences and the revised parts are highlighted in the revised manuscript.

Answer: The manuscript is revised based on this comment. Yes, the opinion of the honorable reviewer is absolutely correct.

• • According to the opinion of the respected reviewer, the explanation of the results of Table 5 which is Table 6 in the revised version  has been added to the manuscript which are highlighted in lines 466-486.

• ⎫ “This table illustrates saliency evaluation metrics according to the proposed method. EEG signals are categorized in first part of the GDN-GAN. According to the extracted label, image stimuli is determined and this image with the extracted feature of the first phase of the proposed method are imposed to the GAN part of the network to map the EEG signal to the saliency map of the image stimuli. After training, to test the GAN part, the EEG signals are imposed to the GDN-GAN, and the extracted images are compared to the original ground truth data through different saliency evaluation metrics and the average of these metrics are reported in this table according to each category. Furthermore, the overall SSIM, CC, NSS and s-AUC are represented through computing the average of the saliency evaluation metrics among all categories.

According to this table, the proposed category-level performance of the visual saliency reconstruction method is over 90% except for six categories including Revolver, Running shoe, Lantern, Cellular phone, Golf ball and Mountain tent in terms of SSIM and s-AUC. SSIM interpretes the structural similarity index using the mean and standard deviation of pixels of a selected window with fixed size in reconstructed image and the ground truth data and it would bring a reliable measure of similarity. The s-AUC uses true positives and false positives according to the pixels of the reconstructed image in locations of fixations in ground truth data and is a confident metric of similarity between two images. Considering these details, SSIM and s-AUC illustrates the limitations of the proposed GDN-GAN. However, considering the detailed values of four saliency metrics, this table shows that the proposed GDN-GAN is a reliable and efficient method to map the EEG signals to saliency map of the visual stimuli.”

Also, in order to more accurately evaluate, the ROC curve and precision-recall values according to the proposed GDN and other state-of-the-art method along with their complete explanation have been added to the manuscript which are highlighted in pages 19-20 and lines 435-441. Table 5 and Fig.12 are added to cover these missing points.

• ⎫ “ Fig. 12 shows the receiver operating characteristic (ROC) plot for GDN part of the proposed GDN-GAN and other state-of-the-art methods for classification of the EEG-ImageNet dataset including region-level stacked bi-directional LSTMs [47], stacked LSTMs [48] and siamese network [39]. The superiority of the GDN in terms of the area under the ROC can be seen in this figure compared to the other existing methods. Furthermore, performance of the GDN against the above-mentioned state-of-the-art methods in terms of precision, F1-score and recall metrics is shown in Table 5.”

• • Furthermore, confusion matrix of the proposed method for classification of 40 different categories of EEG-ImageNet dataset is added to the manuscript. The confusion is obtained among the whole dataset containing 11965 128-channel EEG recordings after training the proposed GDN network. The explanation related to figure of confusion matrix is highlighted in lines 455-459.

• ⎫ “A good confirmation to the performance of the proposed method is the confusion matrix shown in Fig.14. Confusion matrix is an appropriate illustration of the performance of a network on test splits in case of multi-class classification. Fig. 14 shows the confusion matrix of the GDN part of the proposed method. This Figure confirms the good performance of the classification part of the GDN-GAN.” 

• • In addition, according to the opinion of the respected reviewer, research limitations have also been added to the discussion section which are highlighted in lines 516-528.

• ⎫ “ In spite of that the proposed GDN-GAN have a good performance in reconstruction process, the limitations of the approach could not be ignored. The first is that the ground truth data is generated using the pre-trained Open-Salicon using the image samples corresponding to the EEG-ImageNet database. This point should be considered in future works and the solution is to use a good eye-tracker device and record the eye fixation maps at the same time with EEG recordings. These recorded eye fixation maps should be used as the ground truth data in future works. Another limitation of the proposed GDN-GAN is the two-phase process of saliency reconstruction and three-phase of image reconstruction considering the functional connectivity-based graph representation of the EEG signals imposed as the input to the network. An end-to-end process should be considered as the target deep network to decrease the training phases and eventually reducing the computational complexity and hence increasing the speed of the network.”

Comment 2: In line 345, the authors refer to table 6. Should it probably be a reference to Table 2? In the first paragraph of Section 5, the number of the figure the authors are referring to is missing - see Line 412.

Answer: According to the opinion of the respected reviewer, reference to Table is corrected in the revised manuscript and changed to Table 2 which is highlighted in line 355.

The missing Fig. number is considered and highlighted in line 419 as Fig.10.

Comment 3: Although the manuscript is well written, there are some typos, which is why I recommend proofreading the text.

In the abstract, for the word "saliency", we can find "saliecny" (in Line 4) and "slaiency" (in Line 7). In Line 44, the word "slaient" can be found.

The statement in Line 143 should be checked: "Each of the layers with ReLU and a pooling layer."

In Line 453, "The kernal size..." should probably be "The kernel size..."

In the title of section 3.2. the authors wrote "Chebyshev Graph Convolutoion" Convolutoion -> Convolution

Next, for example, the statement that starts in Line 79 should be checked: "Visual attention is a selective procedure occurs for understanding the visual area input."?

Answer: The manuscript is revised based on this comment. Thanks for the careful opinion of the honorable reviewer, the manuscript was read again and the spelling and grammar errors were corrected and highlighted in the manuscript.

Fixed spelling errors include the following:

• • saliency = saliency in page 1, lines 4&8.
• • salient = salient in page 2, line 46.
• • “Each of the layers with ReLU and a pooling layer." is corrected which is highlighted in line 163-164 of the revised manuscript.

      “ReLU is used as the activation function of each layers of shallow net.”

• • kernal size = The kernel size in page 13 , line 364.
• • Convolutoion = Convolution in page 6  , line 216.
• • "Visual attention is a selective procedure occurs for understanding the visual area input."

is corrected which is highlighted in line 100 of the revised manuscript.

“Visual attention is a selective procedure occurs for understanding the visual input to the brain from the surrounding environment.”

Author Response File: Author Response.pdf

Reviewer 3 Report

The manuscript entitled “Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-based Generative Adversarial Network” proposes a deep learning model for reconstructing the image stimuli from recorded EEG signals via visual saliency based on the geometric deep network-based generative adversarial network (GAN) (GDN-GAN). The proposed method’s efficiency and viability are validated experimentally.

The manuscript is well-written and easy to follow. The presented study is interesting and has potential practical application. The utilized techniques are described in detail. The results are well presented (with room for some visual improvements) and appropriately discussed.

However, here are some comments I would like the authors to address before the manuscript is considered for publication:

1.      In the Abstract, please define the abbreviations EEG and GAN when first mentioned. Although well known, this would slightly improve Abstract’s clarity and consistency.

2.      The word “saliency” is misspelled several times in the Abstract and the rest of the paper. Please correct.

3.      Please increase the text size in Tables 2 and 3, as it is difficult to read.

4.      Please try improving the resolution of Figure 9 and the text size, as it is hardly visible and can not be appropriately interpreted.

5.      Please increase the size of the graphs in Figure 12.

6.      In the Conclusion section, please provide some limitations of the proposed method.

7.      In the Conclusion section, please also provide some directions for future research.

Author Response

Reviewer#3:

Comments:

The manuscript entitled “Visual Saliency and Image Reconstruction from EEG Signals via an Effective Geometric Deep Network-based Generative Adversarial Network” proposes a deep learning model for reconstructing the image stimuli from recorded EEG signals via visual saliency based on the geometric deep network-based generative adversarial network (GAN) (GDN-GAN). The proposed method’s efficiency and viability are validated experimentally.

The manuscript is well-written and easy to follow. The presented study is interesting and has potential practical application. The utilized techniques are described in detail. The results are well presented (with room for some visual improvements) and appropriately discussed.

However, here are some comments I would like the authors to address before the manuscript is considered for publication:

• ⎫ We appreciate your detailed review of our manuscript and thank for your valuable suggestions. We believe that your comments have improved the quality of this manuscript. We have considered your comments one by one and revised our manuscript accordingly. The added sentences and the revised parts are highlighted in the revised manuscript.

Comment 1: In Abstract, please define the abbreviations EEG and GAN when first mentioned. Although well known, this would slightly improve Abstract’s clarity and consistency.

Answer: The manuscript is revised based on this comment. With respect to the opinion of the respected reviewer, the two abbreviations "EEG" and "GAN" are fully defined in the abstract which are highlighted in page 1 and lines 6-7.

Comment 2: The word “saliency” is misspelled several times in the Abstract and the rest of the paper. Please correct.

Answer: The manuscript is revised based on this comment. Thanks to the accuracy of the honorable reviewer's opinion, yes, the honorable reviewer's opinion is completely correct. The correct form of the desired word in the entire text of the manuscript was corrected and highlighted.

• • saliency = saliency in page 1, lines 4&8.
• • salient = salient in page 2, line 45.

Comment 3: Please increase the text size in Tables 2 and 3, as it is difficult to read.

Answer: The manuscript is revised based on this comment. According to the reviewer's opinion, the size of the text in Tables 2 and 3 was increased for better readability which are highlighted in pages 14 & 15.

Comment 4: Please try improving the resolution of Figure 9 and the text size, as it is hardly visible and cannot be appropriately interpreted.

Answer: The manuscript is revised based on this comment. This Figure is Figure 10 in the revised manuscript. According to the reviewer's opinion, the resolution size of Figure 10 was improved for better readability which is highlighted in page 19.

Comment 5: Please increase the size of the graphs in Figure 12.

Answer: The manuscript is revised based on this comment. This Figure is Figure 15 in the revised manuscript. According to the reviewer's opinion, the size of the graphs in Figure 15 was improved for better readability which is highlighted in page 24.

Comment 6: In the Conclusion section, please provide some limitations of the proposed method.

Answer: The manuscript is revised based on this comment. The limitations of the present study are highlighted in lines 516-528 of results and discussion section:

• ⎫ “ In spite of that the proposed GDN-GAN have a good performance in reconstruction process, the limitations of the approach could not be ignored. The first is that the ground truth data is generated using the pre-trained Open-Salicon using the image samples corresponding to the EEG-ImageNet database. This point should be considered in future works and the solution is to use a good eye-tracker device and record the eye fixation maps at the same time with EEG recordings. These recorded eye fixation maps should be used as the ground truth data in future works.

Another limitation of the proposed GDN-GAN is the two-phase process of saliency reconstruction and three-phase of image reconstruction considering the functional connectivity-based graph representation of the EEG signals imposed as the input to the network. An end-to-end process should be considered as the target deep network to decrease the training phases and eventually reducing the computational complexity and hence increasing the speed of the network.”

Also, the limitations are added to the conclusion section and highlighted in lines 548-554 according to the opinion of the respected reviewer.

• ⎫ “The limitation concerning to the ground truth data would be considered in future works to have a deep network that acts more similar to the real-world circumstances. The ground truth data in the proposed GDN-GAN is generated using the Open-Salicon pre-trained weights. This data should be recorded using a good eye tracker device in the same time with the EEG recordings. Considering the eye fixation maps of the subjects as the ground truth data would increase the efficiency of the proposed GDN-GAN in BCI applications.”

Comment 7 : In the Conclusion section, please also provide some directions for future research.

Answer:  The manuscript is revised based on this comment. Suggestions for future research are added to the conclusion section and highlighted in lines 539-554.

• ⎫ “ This research would be applicable to BCI projects for helping disable people to communicate with their surrounding world. Neural decoding of the visually provoked EEG signals in BCI will interpret the brain activity of the subject and realize the automatic detection of the stimuli. It will pave the way to the mind reading and writing via EEG recordings and is a preliminary step to help blind people with producing a module to realize vision through generating EEG signals corresponding to the visual surrounding environment.

The limitation concerning to the ground truth data would be considered in future works to have a deep network that acts more similar to the real-world circumstances. The ground truth data in the proposed GDN-GAN is generated using the Open-Salicon pre-trained weights. This data should be recorded using a good eye tracker device in the same time with the EEG recordings. Considering the eye fixation maps of the subjects as the ground truth data would increase the efficiency of the proposed GDN-GAN in BCI applications. ”

Author Response File: Author Response.pdf