Accurate Feature Extraction from Historical Geologic Maps Using Open-Set Segmentation and Detection

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

Aaron Saxton et al. have developed a novel artificial intelligence approach to extract polygon and point features from historical geological maps for assessing mineral resources required for source transformation. The innovation lies in using map units within the legend as prompts for segmentation and detection during geological feature extraction. The model is combined with a human-in-the-loop system, allowing geologists to refine the results effectively, merging the power of artificial intelligence with expert oversight. Tested on geological maps labeled for the AI4CMA DARPA Challenge 6 by the U.S. Geological Survey and DARPA, our method achieved a median F1 score of 0.91 for polygon feature segmentation and 0.73 for point feature detection with rich annotation data, outperforming current benchmarks. By digitizing historical geological data efficiently and accurately, it provides significant insights for responsible policy-making and effective resource management in the global energy transition.However, some critical issues must be resolved.

 

1.     To improve model performance, it is recommended to expand the dataset size, and ensure that the maps are fully and correctly annotated, to achieve a more balanced class distribution and accurate point and line labels.

2.     Point detection models have limitations: for rare point symbols, the point detection model is constrained by the amount of training data. Due to the low frequency of certain symbols appearing on some maps, there is insufficient data representation. It is recommended to develop tools that can effectively combine human and machine intelligence, allowing for rapid review of tool outputs, adjustment of legend annotations for precise feature extraction, and correction of imperfections in feature extraction.

3.     How does the model perform with blurry, faded, or damaged maps?

4.     Figure 4 and Figure 5 are labeled incorrectly.

5.     Hope the instructions below the image can be more concise.

6.     Figures 8 and 9 need to be split into subfigures a and b.

7.     Please provide some additional references.

 

 

All in all, this is an article about a new method based on artificial intelligence to improve geological maps, which greatly promotes the combination of AI and geological maps. The method is very novel, and the model establishment and selection are appropriate. It is hoped that more data will be improved in the future to enhance the accuracy of the model.

Comments on the Quality of English Language

Minor editing of English language required.

Author Response

Comment 1: To improve model performance, it is recommended to expand the dataset size, and ensure that the maps are fully and correctly annotated, to achieve a more balanced class distribution and accurate point and line labels.

Response: Thank you for your thoughtful comment. Expanding the dataset and ensuring accurate annotations are indeed vital for improving model performance. However, human annotation for these maps is quite labor-intensive, taking about a week to fully annotate a single map. The USGS has already invested significant effort into annotating 283 maps (169 for training, 82 for validation, and 32 for testing) for this initial phase. Our AI model is performing well on most maps, and for those that need improvement, a human-in-the-loop system (a demo video is attached) has been implemented that allows humans to correct the model's outputs, reducing the annotation time from one week to a few hours. This is an ongoing effort to build a larger dataset that includes more edge cases, and we plan to retrain the model with this expanded dataset in the future. We have added this to the paper discussion.

Comment 2: Point detection models have limitations: for rare point symbols, the point detection model is constrained by the amount of training data. Due to the low frequency of certain symbols appearing on some maps, there is insufficient data representation. It is recommended to develop tools that can effectively combine human and machine intelligence, allowing for rapid review of tool outputs, adjustment of legend annotations for precise feature extraction, and correction of imperfections in feature extraction.

Response: Thank you for your insightful comment about the limitations of point detection models and suggestions for tools that can effectively combine human and machine intelligence. We have two strategies to address these data constraints: 1) As mentioned in the previous comment, we are developing a human-in-the-loop system to facilitate quicker annotations for model training. 2) We also plan to explore alternative models that require less data, including non-learning-based methods like template matching, and class-agnostic learning using foundation models as a prior. We have highlighted this discussion in the revised manuscript.

Comment 3: How does the model perform with blurry, faded, or damaged maps?

Response: While we have not encountered many cases involving faded or damaged maps, we appreciate the importance of addressing these edge-case scenarios. The model has been designed with generalizability and robustness in mind. To study these challenging edge cases, we have carefully examined scenarios such as: 1) instances where multiple points are closely lumped together (in Figure a); 2) symbols overlapping with other linear features, making them less discernible (in Figure b); and 3) color mismatches between the symbols on the map and those in the legend (in Figure c)). These situations present difficulties for point detection models to perform well. We have added more discussion in the model performance section.

Comment 4: Figure 4 and Figure 5 are labeled incorrectly.

Response: We adjust Figure 4 and Figure 5 in the correct order.

Comment 5: Hope the instructions below the image can be more concise.

Response: We have simplified them to make the information clearer and easier to follow.

Comment 6: Figures 8 and 9 need to be split into subfigures a and b.

Response: We have modified all figures to ensure consistency in labeling by using labels like "a," "b," "c," and so on.

Comment 7: Please provide some additional references.

Response: We have reviewed more literature and included additional relevant references in the Related work section and the Discussion section to strengthen the work.

Author Response File: Author Response.pdf

Reviewer 2 Report

Comments and Suggestions for Authors

Comments

This manuscript proposes a method for the automatic digitization of geological maps, effectively addressing the closed set problem faced by traditional methods. However, there are still some issues that need to be revised:

1.    Firstly, in line 68 of the introduction, the deep learning method employed in this study is mentioned. Please provide a brief introduction to this method when it is first referenced.

2.    In lines 92 to 95, traditional digital methods and machine learning methods were mentioned. Please provide examples for each and compare the two approaches.

3.    The layout of Figure 1 appears slightly cluttered. Please adjust the sizes of the images and the positions of the text to enhance its overall neatness.

4.    Please check whether the order of Figures 4 and 5 is incorrect. Additionally, the layout of Figures 4 and 5 is quite disorganized. In Figure 5b (as indicated in your text), the font is too small.

5.    The network model (YOLO) mentioned in the section on feature extraction lacks corresponding architectural diagrams. Please provide the relevant model architecture diagrams and explain the differences among the various feature extraction components.

6.  Section 4.2.1 investigates the impact of patch size on experimental results. However, the settings of these hyperparameters should be considered part of model tuning. Please explain the significance of this aspect.

7. It would be clearer to distinguish the three images in Figure 9 by labeling them as a, b, and c.

 

Comments for author File: Comments.pdf

Comments on the Quality of English Language

The language in the manuscript needs polishing.

Author Response

Comment 1: Firstly, in line 68 of the introduction, the deep learning method employed in this study is mentioned. Please provide a brief introduction to this method when it is first referenced.

Response: Thank you for your valuable feedback. We have included a concise overview of the method at its first mention to enhance clarity and ensure readers have the necessary context.

Comment 2: In lines 92 to 95, traditional digital methods and machine learning methods were mentioned. Please provide examples for each and compare the two approaches.

Response: Thank you for your helpful comment. We have included specific examples of traditional digital methods, alongside examples of machine learning methods in the section. Additionally, we have incorporated a comparison of the two approaches, highlighting their strengths and weaknesses in the context of our study.

Comment 3: The layout of Figure 1 appears slightly cluttered. Please adjust the sizes of the images and the positions of the text to enhance its overall neatness.

Response: Thank you for your constructive feedback regarding Figure 1. We have adjusted the sizes of the images and repositioned the text to create a more organized and visually appealing layout.

Comment 4: Please check whether the order of Figures 4 and 5 is incorrect. Additionally, the layout of Figures 4 and 5 is quite disorganized. In Figure 5b (as indicated in your text), the font is too small.

Response: We adjust Figure 4 and Figure 5 in the correct order. We also re-draw Figure 5 to make it more readable.

Comment 5: The network model (YOLO) mentioned in the section on feature extraction lacks corresponding architectural diagrams. Please provide the relevant model architecture diagrams and explain the differences among the various feature extraction components.

Response: We appreciate your suggestion for improving the manuscript. To clarify, we utilized the default YOLO v8 architecture, modifying the input to facilitate the concatenation of map patches and prompt patches. Additionally, we adjusted the output for binary classification to suit the needs of our study. There is no specific adaptation made to the standard YOLO architecture.

Comment 6: Section 4.2.1 investigates the impact of patch size on experimental results. However, the settings of these hyperparameters should be considered part of model tuning. Please explain the significance of this aspect.

Response: Thank you for your insightful comment. We have clarified the significance of hyperparameter settings in Section 4.2.1, particularly regarding their role in model tuning. Patch size is a crucial hyperparameter that can significantly influence the performance of our model, as it affects the context available for learning and can impact both accuracy and computational efficiency. By discussing this aspect, we aim to emphasize how careful selection and tuning of hyperparameters like patch size are integral to optimizing model performance and achieving reliable experimental results.

Comment 7: It would be clearer to distinguish the three images in Figure 9 by labeling them as a, b, and c.

Response: We make this adjustment in the revised version to ensure that each image is easily distinguishable.

Author Response File: Author Response.pdf

Reviewer 3 Report

Comments and Suggestions for Authors

This study introduces a novel AI method for extracting polygon and point features from historical geological maps, facilitating their quick and efficient digitization. The proposed method involves using map units in the legends as prompts for one-shot segmentation and detection in geological feature extraction. The manuscript is well-organized and clearly written, presenting the challenges, workflow, and advantages and disadvantages of the proposed method in detail. Accompanying figures generally provide helpful explanations that assist the reader in following the manuscript. I recommend publication of this work after minor revisions focused on improving certain aspects of the text and figures. My comments and suggestions are outlined below.

The current Introduction section primarily emphasizes the identification and assessment of mineral resources, which may give the impression that the proposed method is tailored exclusively for this purpose. However, the method presented in this work can be utilized to digitize maps applicable to various fields within geosciences. I recommend first introducing readers to the broader issue of digitizing historical geological maps, which is of a broad interest, and then narrowing the focus to the specific challenge of assessing mineral resources essential for the energy transition.

What is the final product of the proposed method? It is unclear to me what the output of the entire process is. Is it an exact digital copy of the original scanned geological map, with all the rock types and features displayed? Additionally, can this method recognize structures such as faults on the map and include them in the final digitized output? Please discuss this aspect. Structures like faults are crucial for understanding the geological history and evolution of rocks, as well as the distribution of mineral resources.

After presenting the method and model performance, I would expect to see a specific example demonstrating the application of the proposed method. If possible, please include an original scanned map alongside the results after applying the method, showcasing all vectorized polygons and point features.

Figures: Please ensure consistency in labeling across all figures. I recommend using labels such as "a," "b," "c," etc. for all figures. For example, in Figure 6, it would be clearer to label each column (i.e., a, b, c, and d) to help readers follow the figure and its caption more easily. Currently, it is difficult to identify which column corresponds to the true segmentation mask; according to the caption, it should be the third column. Is that correct? Additionally, Figure 4 should be placed in the appropriate position, before Figure 5.

Lines 140, 156, 158, and 418: Use "et al." for citations with multiple authors. For example, "Guo et al. [21] used a graph…".

Author Response

Comment 1: The current Introduction section primarily emphasizes the identification and assessment of mineral resources, which may give the impression that the proposed method is tailored exclusively for this purpose. However, the method presented in this work can be utilized to digitize maps applicable to various fields within geosciences. I recommend first introducing readers to the broader issue of digitizing historical geological maps, which is of a broad interest, and then narrowing the focus to the specific challenge of assessing mineral resources essential for the energy transition.

Response: Thank you for your valuable feedback. We have revised it to first highlight the overarching importance of digitizing historical geological maps across various geoscience fields in the Introduction section.

Comment 2: What is the final product of the proposed method? It is unclear to me what the output of the entire process is. Is it an exact digital copy of the original scanned geological map, with all the rock types and features displayed? Additionally, can this method recognize structures such as faults on the map and include them in the final digitized output? Please discuss this aspect. Structures like faults are crucial for understanding the geological history and evolution of rocks, as well as the distribution of mineral resources.

Response: Thank you for your thoughtful comment. As shown in Figure 7, the final product of the proposed method is indeed a digitized version of the original scanned geological map, which includes detailed representations of the polygon and point features. Besides this, our pipeline also automatically extracts the legend information and its associated text information, as shown in Figure 5 (a).

In a geologic map, each rock unit is assigned a unique color and symbol, which we describe in our paper as a "polygon feature"—a type of map unit found in the legend. Strike and dip symbols are referred to as "point features" in our work. Faults are indicated by bold lines on geologic maps; however, addressing this feature falls outside the scope of our current study. We acknowledge the importance of faults and are actively working on collecting additional training data for line features, with plans to develop a model for these features in the future.

Comment 3: After presenting the method and model performance, I would expect to see a specific example demonstrating the application of the proposed method. If possible, please include an original scanned map alongside the results after applying the method, showcasing all vectorized polygons and point features.

Response: Thank you for your valuable suggestion. Including a specific example to demonstrate the application of our proposed method would indeed enhance clarity. We have incorporated an original scanned map alongside the results, clearly showcasing all vectorized polygons and point features, as shown in figure 7.

Comment 4: Figures: Please ensure consistency in labeling across all figures. I recommend using labels such as "a," "b," "c," etc. for all figures. For example, in Figure 6, it would be clearer to label each column (i.e., a, b, c, and d) to help readers follow the figure and its caption more easily. Currently, it is difficult to identify which column corresponds to the true segmentation mask; according to the caption, it should be the third column. Is that correct? Additionally, Figure 4 should be placed in the appropriate position, before Figure 5.

Response: Thank you for your valuable feedback regarding the figures. We have modified all figures to ensure consistency in labeling by using labels like "a," "b," "c," and so on. We also adjust Figure 4 and Figure 5 in the correct order.

Comment 5: Lines 140, 156, 158, and 418: Use "et al." for citations with multiple authors. For example, "Guo et al. [21] used a graph…".

Response: Thank you for pointing this out. We have revised the citations at lines 140, 156, 158, and 418 to use "et al." for works with multiple authors. We have also checked other citations to ensure consistency.

Author Response File: Author Response.pdf

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

The manuscript has been revised as required.