Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps

Round 1

Reviewer 1 Report

The article reports a cohesive and very clear way of implementing a semantic segmentation deep learning using modified UNet for extracting surface mine extents from historic topographic Maps. The authors showed the segmentation performance in terms of precision, recall, F1 score, and accuracy. However, I have some comments on the article and suggested some optional improvements (see comments).

1. Many acronyms were used in the manuscript; a list of acronyms can be provided before the ‘references’ section for easy readership. 

2. Some recent semantic segmentation architectures can be mentioned in the related        works like “HRNets” or “Gated-SCNN”

3. Related semantic segmentation architectures such as HRNets or Gated-SCNN can be used to compare the results of the proposed method (if time permits). The authors can also discuss the probable performance of the other method on the dataset that is used in this paper.

Overall, I think it is a good research contribution to extract surface mine extents from historic topographic maps using deep learning techniques. 

Author Response

Please see attachment.

Author Response File: Author Response.docx

Reviewer 2 Report

Dear authors

I would like to congratulate the authors on the subject chosen, historical cartography represents an important source of information that must be properly reevaluated. The article is well structured and fluent and English language check is not necessary. In detail: the references are proper and strictly relevant to the study; the introduction clearly outlines the research problem that authors intend to address. The methodology is clearly described, well structured, the effectiveness of the proposed method is evaluated using a total of 170 historic topographic maps and the validation is performed through accuracy assessment metrics (precision, recall, specificity, the F1 score, and overall accuracy).

In my opinion, the paper is ready to be published. 

Best regards.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 3 Report

The Authors present an interesting paper on Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps.  The Authors investigated specific mapping problem using UNet semantic segmentation deep learning (DL) and a large example dataset. However, this manuscript needs numbers of improvement before publication remote sensing journal. There is no clarification of CNNs model in Abstract section but authors simply mentioned in keywords. Furthermore, DL and CNNs function, data extraction and process of study need to discuss in the manuscript. The methods are well however methodological novelties are still not visible in the manuscript. Conclusion section is not clear and not clearly mentioned about the way forward. There are multiple algorithms are practices to extract the feature value using moderate to high resolution satellite images however authors missed to discuss in introduction and discussion section.

Others comments

 Please address the CNNs in abstract section and remove CNNs from keywords and put only convolutional neural networks.

Line 42: authors wrote LULC change is the complex due to is temporal divers. Please review more about the LULC change monitoring in topographically complex region and its accuracy.

 Line 78 don’t give the example of figure 1 in introduction part (Rf-19-22).

Figure 1: use degree decimal.

 Line 106:  No need to mentioned about the paper-based map 1: 100000 and 1: 250,000. Please write scale 1: 100000 and 1: 250,000. Please write

 Figure 2: use small font for F: and S:

 Line 256: Please write ArcGIS pro-software environment.

 Please Check font of figure 7 as well as other figures.

 Rewrite your conclusion section.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 4 Report

The topic of this paper is interesting and the paper is described well. However, there is no technical nor methodological innovation.

The authors can reorganize the paper from the perspective of constructing a new dataset about surface mine extents. Besides, a variety of deep learning models should be used to evaluate the performance of the new dataset. Unless the authors modify the paper from the perspective of constructing a new dataset or provide a new approach for extracting surface mine extents, I would suggest that this paper not be published.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Reviewer 5 Report

Title of the paper looks very exciting. However, overall this paper would be suitable for technical note other than an article.

Please rewrite the introduction as a research article, and established your argument of the study, prove the novelty of your research. I think you do not need to say limitation of Landsat images as you are not using Landsat images. Write about the importance of topographic maps and how that can use a machine learning algorithm. I did not recognize the main benefit of applying deep learning on topographic maps that are already digitized. Please do not mix up analysis methods in the materials section. 

Dice Coefficient = 0.763 is very poor, you may have such level of accuracy applying an ordinary classification algorithm.

Author Response

Please see the attachment.

Author Response File: Author Response.docx

Round 2

Reviewer 3 Report

Dear Authors 

  Thank you very much for this revised version. Please verify all table number presented in main text, figure number ; figure caption and references.

Author Response

Response to Referees (Round 2)

Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps

We would like to thank the two anonymous referees for their very thoughtful comments, which helped us to further strengthen the paper.

Below we have pasted the complete comments of each of the referees, along with our responses.  The original is in black; our responses are in red.

Academic Editor

Your ms should be acceptable as a research paper once you address minor issues raised by reviewer 4. Please make sure to highlight the novelty and main contribution of this ms as outlined in your response to the editor. This should clarify any remaining confusion regarding the nature of your research work (research vs technical note) and why "investigation of deep learning applied to non-image geospatial datasets is an important contribution".

We have made additional edits to the Introduction and Discussion sections to more explicitly highlight the novelty, main contributions, and limitations/complexities of the work. We hope that this addresses the comments of Reviewer 4 and are willing to make additional edits if necessary. Thanks.

Introduction (Final Paragraph):

Given the complexity of extracting such features, in this study we investigate the use of deep learning (DL), modified UNet semantic segmentation using convolutional neural networks (CNNs) as a technique for extracting surface mine features from historic topographic maps. We make use of digitized and georeferenced historic topographic maps made publicly available by the USGS [26] along with manually digitized extents produced by the USGS Geology, Geophysics, and Geochemistry Science Center (GGGSC) [23]. Our primary objectives are to (1) quantify how well models trained on a subset of topographic maps in KY generalize to different maps in KY, VA, and Ohio (OH) and (2) assess the impact of training sample size, or the number of manually digitized topographic maps available, on model performance. This study does not attempt to develop a new DL semantic segmentation algorithm or compare a variety of existing algorithms for this specific mapping task. Our primary focus in this study is the use of DL-based semantic segmentation for extracting historic information from cartographic maps, which provide a wealth of information to extend land change studies and further quantify anthropogenic landscape alterations. Such methods are needed to take full advantage of current and historic data.

Discussion (Final Paragraph):

Future research should investigate the mapping of additional features from historic topographic maps, such as the extent of forests and wetlands. DL could also be applied to other cartographic representations that characterize historic landscapes, represent the cumulative efforts of many professionals over many decades, and are not readily available for use in spatial analysis and change studies. For example, features could be extracted from historic reference and geologic maps. As in the lead author’s prior study [44], we argue that there is a need to develop multiple and varied benchmark datasets to support DL semantic segmentation research, including those derived from image datasets and other geospatial data, such as digital terrain data, historic topographic maps, and other cartographic presentations. Such datasets will be of great value in comparative studies and for further development and refinement of algorithms. Comparisons of algorithms were not a focus of this study. Since DL semantic and instance segmentation algorithm development and refinement are still actively being studied, there will be a continued need to investigate these new and refined methods for a wide variety of mapping tasks. For example, high-resolution networks (HRNets) [94–97] have recently been shown to be of value for dealing with issues of intra-class heterogeneity and inter-class homogeneity. Further, gated shape CNNs have been shown to be useful for differentiating features based on unique shape characteristics [98,99]. This further highlights the need for the development of a wide variety of benchmark datasets. Comparison of DL algorithms is complicated by processing time and computational costs, which makes it difficult to consistently and systematically compare algorithms, assess the impact of algorithm settings and architecture, experiment with reductions in sample size and generalization to new data and/or geographic extents, and incorporate multiple datasets into studies [44,54,55]. Since this study relied on a modified UNet algorithm and explored a single mapping task and input dataset, our findings associated with sample size and model generalization may not translate to other algorithms and/or classification problems, which further highlights the need for additional research.

We also added the URLs for the code on GitHub and the data on our West Virginia View website. The GitHub repo is now active and publicly available. The data should be available by the middle of next week. We are waiting on our web administrator to make the link active. Thanks.

The data used in this study are made available through the West Virginia View website (http://www.wvview.org/data_services.html) while the code can be obtained from the associated GitHub repository (https://github.com/maxwell-geospatial/topoDL). We hope that these data and resources will aid in future DL research.

Reviewer 3

Thank you very much for this revised version. Please verify all table number presented in main text, figure number ; figure caption and references.

We have reviewed our tables, figures, and associated captions. The numbering aligns with the references in the text. Our reference numbers also align. Thanks for checking.

Reviewer 4

Thanks for your response. I approve of your contribution on investigation of deep learning applied to non-image geospatial datasets. The topic is really interesting and I still think the novel dataset in your paper is more valuable. Indeed, your study has comparable merit to those published works you listed and would make an equally valuable contribution.

However, another contribution (or conclusion) on several key research questions (e.g., training data size and how well models generalize to new data ) is obtained only from the analysis of one deep learning model, are these findings consistent with other models, such as SegNet、PSPNet、DeepLab v1/v2/v3/v3+? I think more experiments are needed to outline these issues. Besides, there is still no technical nor methodological innovation. In the future, look forward to your improvements and new models of surface mine extents extraction from historic topographic maps.

We have made additional edits to the Introduction and Discussion sections to more explicitly highlight the novelty, main contributions, and limitations/complexities of the work. We hope that this addresses the comments of Reviewer 4 and are willing to make additional edits if necessary. Thanks.

Introduction (Final Paragraph):

Given the complexity of extracting such features, in this study we investigate the use of deep learning (DL), modified UNet semantic segmentation using convolutional neural networks (CNNs) as a technique for extracting surface mine features from historic topographic maps. We make use of digitized and georeferenced historic topographic maps made publicly available by the USGS [26] along with manually digitized extents produced by the USGS Geology, Geophysics, and Geochemistry Science Center (GGGSC) [23]. Our primary objectives are to (1) quantify how well models trained on a subset of topographic maps in KY generalize to different maps in KY, VA, and Ohio (OH) and (2) assess the impact of training sample size, or the number of manually digitized topographic maps available, on model performance. This study does not attempt to develop a new DL semantic segmentation algorithm or compare a variety of existing algorithms for this specific mapping task. Our primary focus in this study is the use of DL-based semantic segmentation for extracting historic information from cartographic maps, which provide a wealth of information to extend land change studies and further quantify anthropogenic landscape alterations. Such methods are needed to take full advantage of current and historic data.

Discussion (Final Paragraph):

Future research should investigate the mapping of additional features from historic topographic maps, such as the extent of forests and wetlands. DL could also be applied to other cartographic representations that characterize historic landscapes, represent the cumulative efforts of many professionals over many decades, and are not readily available for use in spatial analysis and change studies. For example, features could be extracted from historic reference and geologic maps. As in the lead author’s prior study [44], we argue that there is a need to develop multiple and varied benchmark datasets to support DL semantic segmentation research, including those derived from image datasets and other geospatial data, such as digital terrain data, historic topographic maps, and other cartographic presentations. Such datasets will be of great value in comparative studies and for further development and refinement of algorithms. Comparisons of algorithms were not a focus of this study. Since DL semantic and instance segmentation algorithm development and refinement are still actively being studied, there will be a continued need to investigate these new and refined methods for a wide variety of mapping tasks. For example, high-resolution networks (HRNets) [94–97] have recently been shown to be of value for dealing with issues of intra-class heterogeneity and inter-class homogeneity. Further, gated shape CNNs have been shown to be useful for differentiating features based on unique shape characteristics [98,99]. This further highlights the need for the development of a wide variety of benchmark datasets. Comparison of DL algorithms is complicated by processing time and computational costs, which makes it difficult to consistently and systematically compare algorithms, assess the impact of algorithm settings and architecture, experiment with reductions in sample size and generalization to new data and/or geographic extents, and incorporate multiple datasets into studies [44,54,55]. Since this study relied on a modified UNet algorithm and explored a single mapping task and input dataset, our findings associated with sample size and model generalization may not translate to other algorithms and/or classification problems, which further highlights the need for additional research.

Reviewer 4 Report

Thanks for your response. I approve of your contribution on investigation of deep learning applied to non-image geospatial datasets. The topic is really interesting and I still think the novel dataset in your paper is more valuable. Indeed, your study has comparable merit to those published works you listed and would make an equally valuable contribution.

However, another contribution (or conclusion) on several key research questions (e.g., training data size and how well models generalize to new data ) is obtained only from the analysis of one deep learning model, are these findings consistent with other models, such as SegNet、PSPNet、DeepLab v1/v2/v3/v3+? I think more experiments are needed to outline these issues. Besides, there is still no technical nor methodological innovation. In the future, look forward to your improvements and new models of surface mine extents extraction from historic topographic maps.

Author Response

Response to Referees (Round 2)

Semantic Segmentation Deep Learning for Extracting Surface Mine Extents from Historic Topographic Maps

We would like to thank the two anonymous referees for their very thoughtful comments, which helped us to further strengthen the paper.

Below we have pasted the complete comments of each of the referees, along with our responses.  The original is in black; our responses are in red.

Academic Editor

Your ms should be acceptable as a research paper once you address minor issues raised by reviewer 4. Please make sure to highlight the novelty and main contribution of this ms as outlined in your response to the editor. This should clarify any remaining confusion regarding the nature of your research work (research vs technical note) and why "investigation of deep learning applied to non-image geospatial datasets is an important contribution".

We have made additional edits to the Introduction and Discussion sections to more explicitly highlight the novelty, main contributions, and limitations/complexities of the work. We hope that this addresses the comments of Reviewer 4 and are willing to make additional edits if necessary. Thanks.

Introduction (Final Paragraph):

Given the complexity of extracting such features, in this study we investigate the use of deep learning (DL), modified UNet semantic segmentation using convolutional neural networks (CNNs) as a technique for extracting surface mine features from historic topographic maps. We make use of digitized and georeferenced historic topographic maps made publicly available by the USGS [26] along with manually digitized extents produced by the USGS Geology, Geophysics, and Geochemistry Science Center (GGGSC) [23]. Our primary objectives are to (1) quantify how well models trained on a subset of topographic maps in KY generalize to different maps in KY, VA, and Ohio (OH) and (2) assess the impact of training sample size, or the number of manually digitized topographic maps available, on model performance. This study does not attempt to develop a new DL semantic segmentation algorithm or compare a variety of existing algorithms for this specific mapping task. Our primary focus in this study is the use of DL-based semantic segmentation for extracting historic information from cartographic maps, which provide a wealth of information to extend land change studies and further quantify anthropogenic landscape alterations. Such methods are needed to take full advantage of current and historic data.

Discussion (Final Paragraph):

Future research should investigate the mapping of additional features from historic topographic maps, such as the extent of forests and wetlands. DL could also be applied to other cartographic representations that characterize historic landscapes, represent the cumulative efforts of many professionals over many decades, and are not readily available for use in spatial analysis and change studies. For example, features could be extracted from historic reference and geologic maps. As in the lead author’s prior study [44], we argue that there is a need to develop multiple and varied benchmark datasets to support DL semantic segmentation research, including those derived from image datasets and other geospatial data, such as digital terrain data, historic topographic maps, and other cartographic presentations. Such datasets will be of great value in comparative studies and for further development and refinement of algorithms. Comparisons of algorithms were not a focus of this study. Since DL semantic and instance segmentation algorithm development and refinement are still actively being studied, there will be a continued need to investigate these new and refined methods for a wide variety of mapping tasks. For example, high-resolution networks (HRNets) [94–97] have recently been shown to be of value for dealing with issues of intra-class heterogeneity and inter-class homogeneity. Further, gated shape CNNs have been shown to be useful for differentiating features based on unique shape characteristics [98,99]. This further highlights the need for the development of a wide variety of benchmark datasets. Comparison of DL algorithms is complicated by processing time and computational costs, which makes it difficult to consistently and systematically compare algorithms, assess the impact of algorithm settings and architecture, experiment with reductions in sample size and generalization to new data and/or geographic extents, and incorporate multiple datasets into studies [44,54,55]. Since this study relied on a modified UNet algorithm and explored a single mapping task and input dataset, our findings associated with sample size and model generalization may not translate to other algorithms and/or classification problems, which further highlights the need for additional research.

We also added the URLs for the code on GitHub and the data on our West Virginia View website. The GitHub repo is now active and publicly available. The data should be available by the middle of next week. We are waiting on our web administrator to make the link active. Thanks.

The data used in this study are made available through the West Virginia View website (http://www.wvview.org/data_services.html) while the code can be obtained from the associated GitHub repository (https://github.com/maxwell-geospatial/topoDL). We hope that these data and resources will aid in future DL research.

Reviewer 3

Thank you very much for this revised version. Please verify all table number presented in main text, figure number ; figure caption and references.

We have reviewed our tables, figures, and associated captions. The numbering aligns with the references in the text. Our reference numbers also align. Thanks for checking.

Reviewer 4

Thanks for your response. I approve of your contribution on investigation of deep learning applied to non-image geospatial datasets. The topic is really interesting and I still think the novel dataset in your paper is more valuable. Indeed, your study has comparable merit to those published works you listed and would make an equally valuable contribution.

However, another contribution (or conclusion) on several key research questions (e.g., training data size and how well models generalize to new data ) is obtained only from the analysis of one deep learning model, are these findings consistent with other models, such as SegNet、PSPNet、DeepLab v1/v2/v3/v3+? I think more experiments are needed to outline these issues. Besides, there is still no technical nor methodological innovation. In the future, look forward to your improvements and new models of surface mine extents extraction from historic topographic maps.

We have made additional edits to the Introduction and Discussion sections to more explicitly highlight the novelty, main contributions, and limitations/complexities of the work. We hope that this addresses the comments of Reviewer 4 and are willing to make additional edits if necessary. Thanks.

Introduction (Final Paragraph):

Given the complexity of extracting such features, in this study we investigate the use of deep learning (DL), modified UNet semantic segmentation using convolutional neural networks (CNNs) as a technique for extracting surface mine features from historic topographic maps. We make use of digitized and georeferenced historic topographic maps made publicly available by the USGS [26] along with manually digitized extents produced by the USGS Geology, Geophysics, and Geochemistry Science Center (GGGSC) [23]. Our primary objectives are to (1) quantify how well models trained on a subset of topographic maps in KY generalize to different maps in KY, VA, and Ohio (OH) and (2) assess the impact of training sample size, or the number of manually digitized topographic maps available, on model performance. This study does not attempt to develop a new DL semantic segmentation algorithm or compare a variety of existing algorithms for this specific mapping task. Our primary focus in this study is the use of DL-based semantic segmentation for extracting historic information from cartographic maps, which provide a wealth of information to extend land change studies and further quantify anthropogenic landscape alterations. Such methods are needed to take full advantage of current and historic data.

Discussion (Final Paragraph):

Future research should investigate the mapping of additional features from historic topographic maps, such as the extent of forests and wetlands. DL could also be applied to other cartographic representations that characterize historic landscapes, represent the cumulative efforts of many professionals over many decades, and are not readily available for use in spatial analysis and change studies. For example, features could be extracted from historic reference and geologic maps. As in the lead author’s prior study [44], we argue that there is a need to develop multiple and varied benchmark datasets to support DL semantic segmentation research, including those derived from image datasets and other geospatial data, such as digital terrain data, historic topographic maps, and other cartographic presentations. Such datasets will be of great value in comparative studies and for further development and refinement of algorithms. Comparisons of algorithms were not a focus of this study. Since DL semantic and instance segmentation algorithm development and refinement are still actively being studied, there will be a continued need to investigate these new and refined methods for a wide variety of mapping tasks. For example, high-resolution networks (HRNets) [94–97] have recently been shown to be of value for dealing with issues of intra-class heterogeneity and inter-class homogeneity. Further, gated shape CNNs have been shown to be useful for differentiating features based on unique shape characteristics [98,99]. This further highlights the need for the development of a wide variety of benchmark datasets. Comparison of DL algorithms is complicated by processing time and computational costs, which makes it difficult to consistently and systematically compare algorithms, assess the impact of algorithm settings and architecture, experiment with reductions in sample size and generalization to new data and/or geographic extents, and incorporate multiple datasets into studies [44,54,55]. Since this study relied on a modified UNet algorithm and explored a single mapping task and input dataset, our findings associated with sample size and model generalization may not translate to other algorithms and/or classification problems, which further highlights the need for additional research.