Understanding Natural Disaster Scenes from Mobile Images Using Deep Learning

Round 1

Reviewer 1 Report

Paper summary

The authors used three deep learning models (i.e., Faster R-CNN with FZNet as feature extractor, Faster R-CNN with ResNet50 as feature extractor, Single Shot Multibox Detector with ResNet50 as feature extractor) to train a natural disaster dataset collected and annotated by the authors and their team, which contains hazard types and  damage levels. The authors conclude that hazard types are more identifiable than the damage levels.

Strengths of the paper:

1 Any methods and work that contribute to disasters management are timely and have wide social impacts.

2 The authors and their team collected and labelled the disaster-scene data used in this work.

Weaknesses of the paper:

1. The dataset  in the paper is very imbalanced, so the DL methods used in the manuscript are not proper.  The focal loss in RetinaNet proposed in the ICCV paper below don't require classes to be balanced (since foreground features will always be outweighed by background classes).

Lin, T. Y., Goyal, P., Girshick, R., He, K., & Dollár, P. (2017). Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision (pp. 2980-2988). https://openaccess.thecvf.com/content_ICCV_2017/papers/Lin_Focal_Loss_for_ICCV_2017_paper.pdf 

2. Ablation studies to show the methods and results for the importance and impact of different training parameters are missing.

3. Section 3.2  should be one of the core subsections, but it is too thin in the current version,  please extend it and explain the figure 2 in much more detail.

4. ZFNet is a very old CNN model, why you used it as feature extractor, instead of more state-of-art CNN architecture? Also, what is the main reason you choose the three DL models (Faster R-CNN with FZNet as feature extractor, Faster R-CNN with ResNet50 as feature extractor, Single Shot Multibox Detector with ResNet50 as feature extractor)? Instead of Single Shot Multibox Detector with FZNet as feature extractor or other more state-of-art CNN architecture as feature extractor?

General comments:

The paper has very good motivation. The methods used in the paper are not new, as it mainly used existing deep learning and computer vision models and methods for object detection and classification, but it seems ok to publish in the journal [applied science].   

Specific comments and suggestions:

1. Title is very important. It can affect your citations etc. The general rule of a good paper title is: The more concise and better reflecting the contribution of your paper the better. I would suggest something as the following: Identifying natural disaster scenes from mobile images using deep learning. your current title “Machine Understanding of Disaster Scenes from Mobile Images” is too vague and broad and is not able to  not actually reflect the content of the manuscript.  
2. Beyond title, Keywords are very important for a paper as well. It would be nice to include some high level keywords, such as computer vision you used, but it would be better to include some specific keywords that can make your paper stand out from other papers that use the same high level keywords. The current set of keywords used are “Aerial-ground structural health monitoring; Disaster scenes; Classification; Deep learning.”, I would suggest changing “disaster scenes” to “natural disaster scenes”, as that is what your paper focuses on. Also, the deep learning models you used are actually for object detection, not classification, so replace your keyword “classification” with “object detection”. I would suggest adding 2-3 specific keywords that are more representative for your paper, for example, hazard types, damage level, mobile images.  It is ok to have redundancy with your abstract and title, the redundancy is actually good because it will help your paper rank high when your potential readers search for papers related to your paper topic.

3. Please provide a figure that contains your training and test curves for each DL model (for space limitation, I suggest  you put it in an appendix) , otherwise it is not possible to see whether the DL model has encountered overfitting (simply provides a F1 socore cannot say your model performed well, especially when your dataset is very imbalanced)

4. Be sure to keep terms used in the manuscript consistent, for example, in the abstract “deep-learning (DL)”, one of the keywords “deep learning”, also, once you defined deep learning (DL), be sure to use DL later, instead of sometimes use full term, sometimes DL. Also, some places used “hazard-type”, where other places used hazard types, similarly for “damage-level” and “damage level”.

5. In your paper, you use DL abbr for two things, deep learning, and damage level, it is confusing, please make sure the abbr is unique throughout the manuscript. You never give the full name of ZF term, I know you mean ZFNet, but do not expect all of your readers know this!

6. Also , please ask a english native speaker to help you proofread the manuscript, the current manuscript is too wordy.

7. Please provide hyperparameters used in each of your DL models in an appendix, otherwise, it is not possible to allow other researchers to reproduce your experiments. For now, I only see that the authors briefly mentioned about epoch in section 3.4.3. But epoch is not the only hyperparameter in the DL models.

Author Response

Please see our letter of response to the reviewer. 

Author Response File: Author Response.docx

Reviewer 2 Report

This paper uses deep learning image detection methods to detect and classify the disaster scene.

The application of this paper is important and the proposed solution is reasonable and promising. In general, this paper is clearly written, with comprehensive experiments and result analysis. 

The originality of this paper is relatively limited. The image detection model is basically from an existing mature model from the computer vision community, and a fine-tuning process as the transfer learning paradigm is applied to adapt the model for disaster scene classification tasks.

For real-world application, the hazard type (Tornado, Tsunami and Earthquake) is not so important because it can be easily identified because the type of disaster information can be easily obtained and the location and timestamp of the images can be easily matched with the disaster. However, more fine-grained categories, such as building collapse, road damage, people injury, are more expected in the emergency management scenarios. If the authors can conduct more experiments including more semantic labeling of the images rather than simply categorize it by types and severity would make this paper more interesting by the emergency management community.

Author Response

Please see our response letter to the reviewer. 

Author Response File: Author Response.docx

Reviewer 3 Report

The paper proposes an advanced deep learning based framework for disaster-type classification and damage-level prediction from mobile images. The paper is interesting and technically sound. It just needs to address the following revision:

There is no comparison results presented in the paper. The authors perhaps show some self comparison results, i.e. compare the achieved results with the ones obtained using the original Faster R-CNN models  and the original SSD network.

Author Response

Please see our response letter to the reviwer. 

Author Response File: Author Response.docx