Classification of Problem and Solution Strings in Scientific Texts: Evaluation of the Effectiveness of Machine Learning Classifiers and Deep Neural Networks

Round 1

Reviewer 1 Report

I have reviewed the manuscript entitled Classification of Problem and Solution Statements in Scientific Texts: Evaluation of the Effectiveness of Machine Learning Classifiers and Deep Neural Networks. The paper is, in general, well-written and structured. The authors obtain positive results and they derive nice conclusions from them. However, some points should be addressed before I can recommend the editor to accept the manuscript for publication:

1) The first thing that I have noticed is that the authors perform two different classification setups, one for problem/no-problem and the other solution/no-solution. I don't see why it cannot be merged in a single classification setup in which problem/solution is studied. In the end, it will just be a binary classification problem that can be addressed with the same tools the authors use in their paper. Can the authors elaborate a bit more on that?

2) I don't really understand method 1. You perform 5-10 iterations on the same training + test set (67% + 33%). Of course, you have better results there ... you are relearning all the time the same set and learning the features of the test set ... that is dangerous because you may have an overfitting phenomenon. A good way of double-checking this is, after these 5-10 iterations, use a different test set. I am pretty convinced that the accuracy will drop considerably there. Can you explain better method 1 and relate why this method can be reliable?

3) There are some typos in the text. A careful reading will highlight them..

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

In their paper, the authors give an analysis that demonstrates that deep learning networks outperform machine learning classifiers when it comes to identifying statements as either a problem or a solution statement.

The highlight your contribution in bullet format, it is always helpful for the reader.

The English writing is sloppy. Authors are advised to improve the writing in the revised manuscript.

A corpus which contains articles from multi-domains is essential which will identify different characteristic to problem-solution patters. How have you incorporated this issue in the paper?

Discuss about available public domain datasets. Zonal context is also an important factor is such type of research. How can you remove this bias?

I do not see details result related to the parameter tunning. Can you please include it?

Please include challenges in the discuss section.

What is the rationale behind considering these related algorithm and the proposed method?

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

Summary:

Authors propose using various machine learning classifiers and deep learning models on a set of features to intelligently classify a statement as a problem statement or a solution statement. Their analysis shows that deep learning networks outperform machine learning classifiers in classifying a statement as a problem statement or a solution statement. Their best model was able to classify a problem statement from a non-problem statement with an accuracy of 90.0% and a solution statement
from a non-solution statement with an accuracy of 86.0%.

General comments:

1) paper, in general, is well wirtten. Still there are some section where is difficult to follow the lecture. A deep revision is strongly recommended.

2) paper, presents and interesting comparisson among different techniques reported in literature.

3) A comparisson with other non-standard techniques such us those based in morphological processing with dendritic processing and spiking proceesing is recommended.

Some activities to undartake to improve the quaity of the paper:

1) In section 1, clearly state the contributions to the state of the art.

2) Initiate section 3 with paragraph that introduces the reader with the content of this section.

3) At the end of tables 1 to 9, add a line with the averages and put in bold the best performances.

4) Try to compare with other non-standard classification thecniques such us morphological neural networks with dendritic processing and spiking neural networks.

5) Some related wroks that could be added to your reference section are next listed:

Comparative analysis of the classification performance of machine learning classifiers and deep neural network classifier for prediction of Parkinson disease. AU Haq, J Li, MH Memon, J Khan… - … on Wavelet Active …, 2018 - ieeexplore.ieee.org

On the effectiveness of machine and deep learning for cyber security. G Apruzzese, M Colajanni, L Ferretti… - … conference on cyber …, 2018 - ieeexplore.ieee.org

Identifying problem statements in scientific text. K Heffernan, S Teufel - 2016 - repository.cam.ac.uk   Efficient training for dendrite morphological neural networks. H Sossa, E Guevara - Neurocomputing, 2014 - Elsevier   Evaluating shallow and deep neural networks for network intrusion detection systems in cyber security. RK Vigneswaran, R Vinayakumar… - 2018 9th …, 2018 - ieeexplore.ieee.org   Dendrite morphological neurons trained by stochastic gradient descent E Zamora, H Sossa - Neurocomputing, 2017 - Elsevier   Analysis and comparison of machine learning classifiers and deep neural networks techniques for recognition of Farsi handwritten digits YA Nanehkaran, D Zhang, S Salimi, J Chen… - The Journal of …, 2021 - Springer   Differential evolution training algorithm for dendrite morphological neural networks. F Arce, E Zamora, H Sossa, R Barrón - Applied Soft Computing, 2018 - Elsevier   Smooth dendrite morphological neurons. W Gómez-Flores, H Sossa - Neural Networks, 2021 - Elsevier   Extreme learning machine for a new hybrid morphological/linear perceptron. P Sussner, I Campiotti - Neural Networks, 2020 - Elsevier   Deep morphological networks G Franchi, A Fehri, A Yao - Pattern Recognition, 2020 - Elsevier

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 1 Report

This reviewer is satisfied with the changes made by the authors and the answers provided. I recommend to accept the paper.

Reviewer 2 Report

The authors have included all of my concerns. Thus this can be considered for publication

Reviewer 3 Report

Authors have taken into account my comments.