Mapping of Potential Fuel Regions Using Uncrewed Aerial Vehicles for Wildfire Prevention

Round 1

Reviewer 1 Report

The manuscript “Mapping of Fuel Regions using Unmanned Aerial Vehicles for Wildfire Mitigation” proposes a workflow for a sensor package, SLAM, LiDAR processing, and semantic segmentation. The manuscript is well written and timely. My main concerns center around being clear about (1) the goal of the proposed workflow and (2) the limitations of the proposed workflow. I am not sure how this workflow could be used to mitigate wildfires, it seems like a mapping workflow. The goal and how it relates to fire needs to be made clear. The authors quickly reduce fuel classes to landcover classes and do very little with fuel types. Clarify the goals of the workflow, then clarify the limitations of the workflow, then explain how it relates to fire, and I believe this could be an interesting paper. Be clear about what the contribution of this work is.

Limitations: Workflow Limitations and major decisions are not well justified/explained. See the following points:

-          Line 250 – what is the point of your work? It is not ‘fuel’ mapping as you aggregate fuel types into 3 landcover classes. I could see the value in that, but you do not explain it well in your work. There are many major decisions that are lightly justified/explained in this work. If you aggregate 15 classes into 3, you have not “used the Anderson fuel model” line 249. I do not see what the use of the classes described in table 2 would be. You are distinguishing between ‘green stuff’ and ‘other’, for what purpose? Lines 367-369 must be expanded upon. Many of these classes are critical fuel classes and the purpose of this segmentation is not clear if it cannot differentiate these fuel classes. There seems to be no point in bringing up fuel classes since they are never used.

-          Line 168 – flight time is a major concern. 10-20 minutes is very limiting, especially in a fire environment. I do not see how this could be useful given current technology. Expand on this limitation, this is the main drawback to ‘let’s just put all the sensors on the drone’. I appreciate that it is modular, but that already exists. RGB/LIDAR/multispectral cameras could all be mounted on drones as needed. If the package is too heavy to fly, what would it be used for?  

-          Error in the result (Figure 5) is high. How could this be used if the error is so large?

-          I am unsure how the multispectral, LIDAR and RGB data are used/combined. Seems like RGB was used for segmentation, and LIDAR for elevation mapping, but how was multispectral used? There was no mention of how multispectral fits into this workflow. Why not combine results/sensors, at least in the semantic segmentation? What is the overall purpose of this work/mapping? Honestly, I am a bit confused as to why SLAM work is being included with semantic segmentation work and the development of a sensor suite. The overall ‘story’ of this work confuses me; this is highlighted in the discussion where SLAM is followed by semantic segmentation followed by LiDAR, without any overlap or continuity.

Other Comments:

1.       Title: ‘Fuel Regions’ are never described in the manuscript and I cannot see how this could be used for ‘wildfire mitigation’. Do you propose to do this mapping before wildfires (everywhere?) and use targeted fuel removal? How could this be done with 10 minute flights? The proposed workflow is not related to mitigation as far as I can tell. Reword the title. Similarly, modify lines 8-9 as you do not predict fuel classes, you amalgamate into 3 landcover classes.

2.       Abstract: “We integrated advanced technologies and methods to address this need.” Avoid vague statements like this, especially in the abstract. Abstract should be very clear and concise, while summarizing the main findings.

3.       Avoid undefined acronyms, especially in the abstract/first sentence.

4.       Line 10 – no run times are given in the work, how do we know all of this is processing in real time? Give run times for each step or remove this claim as it is not discussed in the manuscript.

5.       Line 14 – why report one metric for one class? Report a support weighted recall, precision and IOU for the method.

6.       Line 15 “potential to improve forest management and conservation efforts”, this is not shown in the manuscript. How could this improve things?

7.       Line 28 – Unclear to me why it would be irreversible. Some forests require fire. Maybe you are talking about a specific region.

8.       Line 33 – There is a push to use gender neutral terms. You might consider remotely piloted aircraft (RPA) or uncrewed aerial vehicle (UAV).

9.       Line 40 – Do you mean ‘digital twin’?

10.   Lines 123-133 – this paragraph seems out of place. Put it with the other mapping/fuels literature review. Swap it with the SLAM literature review paragraph.

11. Preferably you would share your trained model and training data. Please make a statement regarding data/model sharing.

12.   Line 292 – do you mean ‘experienced’?

13.   Study site 1 flight path (figure 3) seems strange. Why this pattern? Seems like it would not work well and adds an additional variable to correct for in the comparison to study site 2. Discuss.

14.   Table 3 – add units of measurement for these.

15.   Table 3 vs. Figure 3 – is site 2 in’ Gascola’ or ‘Pennsylvania’, or both? Be consistent throughout when referring to the sites to help the reader.

16.   Lines 310-311 does not make sense to me, please reword.

17.   Table 3 – it is hard to know if these errors are ‘low enough’, but Figure 4 is much clearer. In Figure 4 it seems like the errors from FASTLIO-SC are clearly too much. Comment on these errors and implications. Yes, one method has less error, but is it low enough to be useful? Also, what are the yellow arrows in the cutouts?

18.   Figure 4 – no scale on figure axes.

19.   Line 332 – capitalize all acronyms and define them on first use.

20.   Line 333 – it could be quantitative if you plotted a histogram of the Hausdorff distance.

21.   Figure 5c – is this error too much? Can this map still be used? What is it to be used for?

22.   Line 338 – saying that google earth resolution is 0.15m and ‘accurate enough’ is not justification as the resolution does not relate to the accuracy of the map. What is the error or accuracy of google earth mapping? This must be stated to enable you to use it as ground truth.

23.   Line 344-345 – this is a logical fallacy. Just because LIOSAM is ‘closer’ to the ground truth than other methods, does not mean it is close enough and is ‘suitable for generating accurate 3D models’. That has not been shown. 

24.   Line 348 – what is meant by ‘overestimated’ in this context? Consider rewording.

25.   Table 4 – add errors for each.

26.   Figure 7 – having very different training and testing sets can cause issues (i.e. dataset shift), please discuss and clarify the impacts.

27.   Table 5 – show the error for the aggregated classes. Also provide a support weighted overall precision, recall, and IOU. Line 374-375, this is not shown in the manuscript/results.

28.   Figure 8 – what is “GT”? ground truth?

29.   Lines 377-383 – I do not understand. Why not just compare the same metrics, i.e. precision, recall, IOU? Why can you only qualitatively compare? You’ve already done this on the training and test data.

30.   Section 4.3 does not add anything to the manuscript. Remove or expand the analysis. This is not ‘fuel mapping’.

31.   Table 2 has the class ‘canopy’ while Figure 9 uses ‘vegetation’. What are the 3 aggregate classes? Be consistent throughout.

32.   Line 391 – You did not propose LIOSAM.

33.   Lines 397-398 – The flights are very different, you should comment on that.  

34.   Section 5 – hard to draw definitive conclusions based on 2 flights.

35.   Lines 443-444 – where is this shown in the results?

36.   Line 445 – how does any of this preserve forests?

37.   Line 454 – how is it lightweight if it drops the flight time of an M600 to 10 minutes?? Reword.

Well writing with some grammatical edits required, see above comments. 

Author Response

Comments to the Author:

The manuscript “Mapping of Fuel Regions using Unmanned Aerial Vehicles for Wildfire Mitigation'' proposes a workflow for a sensor package, SLAM, LiDAR processing, and semantic segmentation. The manuscript is well written and timely. My main concerns center around being clear about (1) the goal of the proposed workflow and (2) the limitations of the proposed workflow. I am not sure how this workflow could be used to mitigate wildfires, it seems like a mapping workflow. The goal and how it relates to fire needs to be made clear. The authors quickly reduce fuel classes to landcover classes and do very little with fuel types. Clarify the goals of the workflow, then clarify the limitations of the workflow, then explain how it relates to fire, and I believe this could be an interesting paper. Be clear about what the contribution of this work is.

Limitations: Workflow Limitations and major decisions are not well justified/explained. See the following points:

R1.1      Line 250 – what is the point of your work? It is not ‘fuel’ mapping as you aggregate fuel types into 3 landcover classes. I could see the value in that, but you do not explain it well in your work. There are many major decisions that are lightly justified/explained in this work. If you aggregate 15 classes into 3, you have not “used the Anderson fuel model” line 249. I do not see what the use of the classes described in table 2 would be. You are distinguishing between ‘green stuff’ and ‘other’, for what purpose? Lines 367-369 must be expanded upon. Many of these classes are critical fuel classes and the purpose of this segmentation is not clear if it cannot differentiate these fuel classes. There seems to be no point in bringing up fuel classes since they are never used.

A1.1 We appreciate the comment. We have cleared up our and improved the explanation about the use of fuel mapping and the Anderson fuel model in our work. In the new version of the manuscript we rewrote this paragraph to avoid confusion; see Lines 262-269 and Lines 271-273

R1.2     Line 168 – flight time is a major concern. 10-20 minutes is very limiting, especially in a fire environment. I do not see how this could be useful given current technology. Expand on this limitation, this is the main drawback to ‘let’s just put all the sensors on the drone’. I appreciate that it is modular, but that already exists. RGB/LIDAR/multispectral cameras could all be mounted on drones as needed. If the package is too heavy to fly, what would it be used for?  

A1.2 The flying time was previously estimated taking into account the battery life where we stop flight around 30% battery. We have rectified in the text to the closest estimated time based on our payload weight for each drone. See line 242. We have also cleared up our meaning of fuel mapping and wildfire to address the confusion regarding when the drone would be flown as Lines 271-273

R1.3    Error in the result (Figure 5) is high. How could this be used if the error is so large?

A1.3 We use the Hausdorff metric and its representation in Figure 5 as a qualitative comparison. As noted in the text, the higher distances correspond to treetops (red regions), which contain less denser and noisy points in each of both of the point clouds. Additionally, given the photogrammetry mesh/pointcloud was produced with monocular cameras, the elevation of the treetops can be subject to errors [1] (quantifying them was out of the scope of this work), which makes the treetops not a good baseline to compare height.  For this reason, using this approach as a qualitative metric was not suitable, and hence we also evaluated the maps measuring their geometric variables. To avoid confusion, we have clarified this in the manuscript's current form. See line 357-363.

R1.4      I am unsure how the multispectral, LIDAR and RGB data are used/combined. Seems like RGB was used for segmentation, and LIDAR for elevation mapping, but how was multispectral used? There was no mention of how multispectral fits into this workflow. Why not combine results/sensors, at least in the semantic segmentation? What is the overall purpose of this work/mapping? Honestly, I am a bit confused as to why SLAM work is being included with semantic segmentation work and the development of a sensor suite. The overall ‘story’ of this work confuses me; this is highlighted in the discussion where SLAM is followed by semantic segmentation followed by LiDAR, without any overlap or continuity.

A1.4 Thank you for such an insightful comment. We have addressed this concern throughout the whole paper in this new version. Namely, we emphasized that the goal of the system and this work is wildfire prevention and not fighting current fire. (Lines 272-274). Additionally, we added information about quality of sensors and payload allows for ablation study regarding different cameras/slam/mapping techniques to ensure best method is used in the end as perception research in forestry is still very limited (Lines 151-155; 256-260; 469-472; 528-529); and add some continuity between subsections for clearer ‘flow’ (Lines 183-184)

Other Comments:

R1.5 Title: ‘Fuel Regions’ are never described in the manuscript and I cannot see how this could be used for ‘wildfire mitigation’. Do you propose to do this mapping before wildfires (everywhere?) and use targeted fuel removal? How could this be done with 10 minute flights? The proposed workflow is not related to mitigation as far as I can tell. Reword the title. Similarly, modify lines 8-9 as you do not predict fuel classes, you amalgamate into 3 landcover classes.

A1.5 We have modified the title accordingly, clarified our meaning of wildfire prevention and added some details regarding our aggregate classes in Lines 410-417.

R1.6 Abstract: “We integrated advanced technologies and methods to address this need.” Avoid vague statements like this, especially in the abstract. Abstract should be very clear and concise, while summarizing the main findings.

A1.6  We have improved the abstract, following the Reviewer’s concern.

R1.7 Avoid undefined acronyms, especially in the abstract/first sentence.

A1.7 As requested by the Reviewer, we have added all the definitions to the abstract/first sentences.

R1.8   Line 10 – no run times are given in the work, how do we know all of this is processing in real time? Give run times for each step or remove this claim as it is not discussed in the manuscript.

A1.8 We have added the proper frequency for the mapping with respect to our real time claim.

R1.9 Line 14 – why report one metric for one class? Report a support weighted recall, precision and IOU for the method.

A1.9 Attending the Reviewer's comment, we have substantiated the main metric chosen in the paper in Line 403-406.

R1.10 Line 15 “potential to improve forest management and conservation efforts”, this is not shown in the manuscript. How could this improve things?

A1.10 Attending the Reviewer’s concern, we have rephrased this sentence.

R1.11 Line 28 – Unclear to me why it would be irreversible. Some forests require fire. Maybe you are talking about a specific region.

A1.11 Following the Reviewer’s comment, we have improved the explanation of our point, expanding the sentence in Line 28-30

R1.12 Line 33 – There is a push to use gender neutral terms. You might consider remotely piloted aircraft (RPA) or uncrewed aerial vehicle (UAV).

A1.12 Following the Reviewer’s comment, we have changed the word unmanned to uncrewed.

R1.13  Line 40 – Do you mean ‘digital twin’?

A1.13 Yes. The manuscript current form now contains the change suggested by the Reviewer.

R1.14   Lines 123-133 – this paragraph seems out of place. Put it with the other mapping/fuels literature review. Swap it with the SLAM literature review paragraph.

A1.14 Following the Reviewer’s guidelines, we have re-organized the paragraph.

R1.15 Preferably you would share your trained model and training data. Please make a statement regarding data/model sharing.

A1.15 As suggested, we have added the url for the model when we have it available to the public. Releasing the training data requires additional permissions, which are under review.

R1.16   Line 292 – do you mean ‘experienced’?

A1.16 This grammar error has been corrected in the manuscript's current form.

R1.17   Study site 1 flight path (figure 3) seems strange. Why this pattern? Seems like it would not work well and adds an additional variable to correct for in the comparison to study site 2. Discuss.

A1.17 The UAV piloted by human experts carried out study site 1 surveys; thus, to avoid loss of visual track of the UAV, we decided to perform a flight path within a traversable area. These surveys help us to set up an initial database for testing our SLAM algorithms. However, on study site 2 we decided to exploit the UAV autopilot to perform the aerial survey. In the manuscript's new version, we have included these details in Fig. 3 caption.

R1.18   Table 3 – add units of measurement for these.

A1.18 We thank the reviewer for pointing out this issue. In the new version of the manuscript, we added the corresponding units. Furthermore, we included the total distance of the trajectory for each survey and the RMSE as a percentage of this distance; see Table 3.

R1.19   Table 3 vs. Figure 3 – is site 2 in’ Gascola’ or ‘Pennsylvania’, or both? Be consistent throughout when referring to the sites to help the reader.

A1.19 As suggested by the Reviewer, in the new version of the manuscript, we keep a consistent reference for study sites; see Table 3 and Fig. 3.

R1.20   Lines 310-311 does not make sense to me, please reword.

A1.20 In the new version of the manuscript, we have rewritten these lines. See lines 318-324.

R1.21   Table 3 – it is hard to know if these errors are ‘low enough’, but Figure 4 is much clearer. In Figure 4 it seems like the errors from FASTLIO-SC are clearly too much. Comment on these errors and implications. Yes, one method has less error, but is it low enough to be useful? Also, what are the yellow arrows in the cutouts?

A1.21 We sincerely appreciate the insightful comments. As the reviewer mentions, low RMSE might not be enough to determine whether or not a SLAM algorithm is suitable for mapping a forested region. To better support the selection of the SLAM algorithm, we included a new reference [1] , which reports that in forested regions, one could expect an RMSE between 0.32-2.90 m (depending on the forested area). Moreover, the RMSE was further expressed as a percentage of the overall trajectory for each survey. Thus, in the new version of the manuscript, we included a broader description of the selection of the SLAM algorithm. See lines 307-324.

[1] Pierzchała, M., Giguère, P., & Astrup, R. (2018). Mapping forests using an unmanned ground vehicle with 3D LiDAR and graph-SLAM. Computers and Electronics in Agriculture, 145, 217-225.

R1.22   Figure 4 – no scale on figure axes.

A1.22  The figure has been updated. Furthermore the caption has been edited to include the description of yellow arrows. 

R1.23   Line 332 – capitalize all acronyms and define them on first use.

A1.23 Attending the Reviewer’s comment, we have corrected this issue.

R1.24 Line 333 – it could be quantitative if you plotted a histogram of the Hausdorff distance.

A1.24 This is certainly a good point. However, we consider the results based on the Hausdorff distance are mainly qualitative since some of the vegetation areas are not dense and can be noisy, which would not provide a clear assessment of the mapping results. Nevertheless, we have included a visualization of the distances in the form of a boxplot, as suggested by the Reviewer. We opted for a box plot since it provided a better visual representation of the outliers and percentiles of the data.

Furthermore, we have also clarified the reasons of considering the Hausdorff distance as qualitative in the manuscript current form in Line 357-363

R1.25   Figure 5c – is this error too much? Can this map still be used? What is it to be used for?

A1.25 Please refer to the response A1.3.

R1.26   Line 338 – saying that google earth resolution is 0.15m and ‘accurate enough’ is not justification as the resolution does not relate to the accuracy of the map. What is the error or accuracy of google earth mapping? This must be stated to enable you to use it as ground truth.

The resolution of 0.15m is accurate enough considering we are evaluating an area of 3000m2, as stated in the manuscript. We would not consider that resolution to be accurate enough if we were evaluating a small region. We chose google earth since it provides a helpful open source tool to measure the geometric variables at that scale. Our further work will consist in measuring the same geometric variables in situ, which because of operational issues was not possible in our experiments. 

To avoid confusions, in the current version of the manuscript we are renaming google earth to “main baseline”. We have also included the discussion of the accuracy of the google earth map and the collection of  real ground truth in Section 6 as can be seen in Line 526-529.

R1.27   Line 344-345 – this is a logical fallacy. Just because LIOSAM is ‘closer’ to the ground truth than other methods, does not mean it is close enough and is ‘suitable for generating accurate 3D models’. That has not been shown. 

A1.27 As stated in the manuscript: “in order to have a strong indication of the quality of the generated map regarding its morphological, visual, and geometry aspects, two approaches were used: i) using a cloud-to-cloud comparison with the photogrammetry output and ii) measuring geometric variables (i.e., distances and areas) in different maps.”. 

In the qualitative evaluation we showed that the baseline and the generated point cloud have a similar appearance. In the second part, we showed that the geometry of the map, measured by different lengths and areas and compared with the Google Earth baseline, has a low error. The results of these two evaluations allow us to infer that the map generated is an accurate 3D representation of the environment (with respect to the baselines). Furthermore, our conclusion is not solely drawn based on the second evaluation, but also in the first one and the quantification of the error in the drone’s pose estimation. 

We have included this remark in the manuscript's current form in Line 28-30.

Finally, it is worth mentioning that to the best of our knowledge, the number of works reporting such thorough evaluation of online localization and mapping results in forestry applications (which are carried out in highly unstructured environments) is rare in the related state of the art.

R1.28   Line 348 – what is meant by ‘overestimated’ in this context? Consider rewording.

A1.28 Following the Reviewer’s comment, we have reworded the sentence.

R1.29   Table 4 – add errors for each.

A1.29 Following the Reviewer’s comment, we have updated Table 4 to include errors.

R1.30   Figure 7 – having very different training and testing sets can cause issues (i.e. dataset shift), please discuss and clarify the impacts.

A1.30 Attending  the Reviewer’s concern, we have added how we  addressed the problem of dataset shift in Line 412-413.

R1.31   Table 5 – show the error for the aggregated classes. Also provide a support weighted overall precision, recall, and IOU. Line 374-375, this is not shown in the manuscript/results.

A1.31 In our study, we aggregated the Anderson model classes in post-processing due to the absence of certain classes at the testing sites (e.g., litterfall, wood pieces, people). Additionally, training the model on fine-grained classes allowed for a more accurate representation of the environment, capturing its nuances effectively and thus providing a more general way to model fuel in new environments. For this reason, we consider that evaluating the system based solely on the aggregated classes might introduce subjectivity and deviate from the original standardization idea of using the Anderson model. To explain this point in the manuscript, we have included a detailed discussion, providing transparency about our aggregation approach and its potential impact on the interpretation of results in Line 263-270. By doing so, we aim to prevent any potential misinterpretations by readers.

R1.32   Figure 8 – what is “GT”? ground truth?

A1.32. This issue has been corrected in the latest version of the manuscript.

R1.33   Lines 377-383 – I do not understand. Why not just compare the same metrics, i.e. precision, recall, IOU? Why can you only qualitatively compare? You’ve already done this on the training and test data.

A1.33 Following the Reviewer’s comment, we have addressed this issue and improved the writing to clarify this point. Line 421-428 was added to clear it up.

R1.34   Section 4.3 does not add anything to the manuscript. Remove or expand the analysis. This is not ‘fuel mapping’.

A1.34 This section has been merged with the previous section.

R1.35   Table 2 has the class ‘canopy’ while Figure 9 uses ‘vegetation’. What are the 3 aggregate classes? Be consistent throughout.

A1.35 We appreciate this comment and we added consistency by only using the canopy as a class.

R1.36   Line 391 – You did not propose LIOSAM.

A1.36 This is certainly a good point. In the new version of the manuscript we replaced “proposed” by “implemented”. See Line 404. 

R1.37   Lines 397-398 – The flights are very different, you should comment on that.  

A1.37 Following the Reviewer's comment, we have updated the sentence ton include this point in Line 446-450.

R1.38   Section 5 – hard to draw definitive conclusions based on 2 flights.

A1.38 This section has been modified to avoid making definitive conclusions about the behavior of the SLAM algorithms in any forested area. Instead, we commented on the performance of these methods (which required some parameter tuning) on the study sites. Although one could expect similar outcomes in forest environments, more experiments should be performed to make judgments about the generalization of SLAM algorithms in forested areas, especially those where GPS signals may be occluded. Please see lines 438-439,  446-449, 457-458. 

R1.39   Lines 443-444 – where is this shown in the results?

A1.39 We addressed the issue pointed out by the Reviewer  in the new version of the manuscript. See line 397-399.

R1.40   Line 445 – how does any of this preserve forests?

A1.40 We hope to have clarified in A1.39

R1.42  Line 454 – how is it lightweight if it drops the flight time of an M600 to 10 minutes?? Reword.

A1.42 Thank you again for pointing out this concern, we addressed it as mentioned in A1.2.

Reviewer 2 Report

In my opinion, the paper is well structured and worked.

Just two objections:

1. Perhaps the paragraph titled Discussion should be part of Conclusions.

2. In some biobibliographic references the doi is missing.

For example, reference 39 is https://doi.org/10.1080/02827581.2017.1418421

Author Response

Comments to the Author

In my opinion, the paper is well structured and worked.

Just two objections:

R2.1 Perhaps the paragraph titled Discussion should be part of Conclusions.

We sincerely acknowledge the Reviewer’s suggestion, however, we believe that having a Section for discussion only helps to highlight the main findings of our work and stress the analysis of the results. 

R2.2 In some biobibliographic references the doi is missing. For example, reference 39 is https://doi.org/10.1080/02827581.2017.1418421

A2.2 Thank you for pointing out that issue, we have resolved it in the new version.

Round 2

Reviewer 1 Report

I am happy with the changes the authors made. There could be some improvement in the grammar of the manuscript, otherwise, I am happy with the submitted version. 

I am happy with the changes the authors made. There could be some improvement in the grammar of the manuscript, otherwise, I am happy with the submitted version.