Automated Plantation Mapping in Southeast Asia Using MODIS Data and Imperfect Visual Annotations

Round 1

Reviewer 1 Report

The work is very complex and gives the idea of a well planned and intense work.

the only perplexity or curiosity concerns the years investigated. why do the authors stop at 2014 and don't use a more recent date?

The authors should write a comment on this

Author Response

We thank the reviewer for the comments. We stopped at 2014 since we only had access to DigitalGlobe data on 2014, which we used as ground-truth data to validate our results.

Reviewer 2 Report

This study developed an automatic way to map the large-scale tree plantation at the annual pace. The workflow design is reasonable, which combined deep learning, HMM, and spatial clustering. The results are well presented. Below are my comments:

Line 96-109: This paragraph seems to belong to the method section.
Line 262-266: How did you integrate the three DBN models to the final class map?
Line 310: How did you tune the transition matrix T? My understanding is Baum–Welch algorithm only gives you better T estimation between 2001, 2005, and 2010, but it still can not represent the yearly transition matrix. For example, some trees could be cut and grow back or replanted in 5 years.
Line 326-328: It sounds like you are using the validation data to tune the algorithm? It will skew the accuracy assessment.

Author Response

We thank the reviewer for the detailed comments.

Review: Line 96-109: This paragraph seems to belong to the method section.

Response: We have moved the technical details (originally in Line 96-109) to the method section (before 3.1.1).

Review: Line 262-266: How did you integrate the three DBN models to the final class map?

Response: We have added the following description about how to use majority voting to aggregate three DBNs at the end of Section 3.1.1.

“Since each binary classifier focuses on differentiating between a specific pair of classes, eight possible combinations represent potential outcomes from the three classifiers. Based on the separate predictions from each classifier, we assign the aggregated prediction result as the majority class label. For instance, if both P-F and O-P classifiers predict a test location as “plantation”, then we will label this test location as “plantation” regardless of the prediction of F-O classifier. When the three binary classifiers generate mutually different labels the test sample will be assigned to the “Unknown” (U) class. This "Unknown" class is handled through our post-processing step as discussed in Section 3.1.5.”

Review: Line 310: How did you tune the transition matrix T? My understanding is Baum–Welch algorithm only gives you better T estimation between 2001, 2005, and 2010, but it still can not represent the yearly transition matrix. For example, some trees could be cut and grow back or replanted in 5 years

Response: We also added more details about HMM method used for post-processing in Section 3.1.5. Basically, we first interpolate yearly data and then used it for initializing the transition matrix and the emission matrix. Our initialization does not consider the cases where trees can be cut and grow back after 5 years and thus is not accurate. We use fine-tuning by Baum-Welch algorithm to fix these errors.

Review: Line 326-328: It sounds like you are using the validation data to tune the algorithm? It will skew the accuracy assessment.

Response: We have added clarification at the end of the Introduction. We trained and validated our proposed model in the same region but we did an unbiased assessment by ensuring that the labels used in validation (taken from DigitalGlobe) were never touched during training. The goal of this work is to generate high-quality maps using remote sensing data and imperfect visual annotations (TP and RSPO).  The training process only uses labels from TP and RSPO. 

Reviewer 3 Report

This is a very well written, interesting paper on automated plantation detection using MODIS. At a time when most approaches are using Landsat/Sentinel 1 and 2, this paper deviates to a coarser resolution sensor with the argument that this could be generated annually. I think we are not far from a situation where higher resolution imagery can feasibly be used to produce annual maps on an operational basis, but this paper, nevertheless, has some interesting approaches and results. 

Some minor comments below:

Although the paper is well written, there are a few typos/mistakes in the paper, e.g., line 61, interpret should be interpreted; line 388, remove 'of' after 'understand, etc. I'm not going to list them all - I'm sure these will be spotted in another read of the paper.  

Are there any plans to produce the PALM product annually and provide it to users, e.g., on the Global Forest Watch website?

I would like to have seen some comparisons with official statistics published by, e.g., the Indonesian Ministry of Agriculture. Although I realize you are underestimating smallholder palm oil plantations, some comparison with official statistics would have been useful.

It would be good to see a bit more clarification around the overall accuracy and producer's/user's accuracy. Did you use the same validation data set for overall accuracy as you did for producer's/user's accuracy? Did you do this for only the year 2014 (to match the date of the Digital Globe imagery)? In other words, does the overall accuracy apply to 2014 only? As you have produced a time series, it is not so clear which year you are actually validating. To be fair, it would be good to have a validation for 2010 (if the very high resolution imagery was available).

Regarding Figure 7e, did you use Google Earth to see whether you could find older imagery (around 2010) to confirm that the RSPO is really incorrect? 

Author Response

We thank for the detailed comments by the reviewer. We have read through the paper and polished the writing.

Review: Are there any plans to produce the PALM product annually and provide it to users, e.g., on the Global Forest Watch website?

Response: We have been in contact with Global Forest Watch and they have shown great interests in making these results accessible to public. Once this paper is accepted, we will be ready to release the generated maps to public.

Review: I would like to have seen some comparisons with official statistics published by, e.g., the Indonesian Ministry of Agriculture. Although I realize you are underestimating smallholder palm oil plantations, some comparison with official statistics would have been useful. 

Response: Since our study is based on MODIS tiles, which does not precisely capture country boundaries, we do not conduct country-wise comparison to known statistics. However, official results Indonesian Department of Statistics for Oil Palm plantations reported in Fig.1 of Jafari et al. (see [1] below) are consistent with  our reported results for plantations for the 3 MODIS tiles (Fig. 6), as in both cases the growth  increased greatly during 2006-2011.

[1] Jafari, Yaghoob, Jamal Othman, Peter Witzke, and Sufian Jusoh. "Risks and opportunities from key importers pushing for sustainability: the case of Indonesian palm oil." Agricultural and Food Economics 5, no. 1 (2017): 13.

Review: It would be good to see a bit more clarification around the overall accuracy and producer's/user's accuracy. Did you use the same validation data set for overall accuracy as you did for producer's/user's accuracy? Did you do this for only the year 2014 (to match the date of the Digital Globe imagery)? In other words, does the overall accuracy apply to 2014 only? As you have produced a time series, it is not so clear which year you are actually validating. To be fair, it would be good to have a validation for 2010 (if the very high resolution imagery was available).

Response: We are using the same data for evaluating overall accuracy and user’s/producer’s accuracy. The overall accuracy is computed as a result of estimated user’s and producer’s accuracy as well as the number of pixels for the entire study region and the detected plantations by each method (clarified in Section 4.2.3). Currently the validation is only focused on 2014 since we only had access to the high-resolution DigitalGlobe data in 2014.

Review: Regarding Figure 7e, did you use Google Earth to see whether you could find older imagery (around 2010) to confirm that the RSPO is really incorrect? 

Response: Unfortunately, Google Earth has very limited imagery for older dates, making it difficult to use it to assess if a location might have been plantation in 2010 and got converted to some other land cover by 2014.  We did study the Enhanced Vegetation Index (EVI) time series (2001-2014) for all the selected locations in Fig. 7-e and found the EVI values to be quite high and stable (as expected for a forested location), which makes it unlikely that these locations were plantations before and got converted to some other land type (e.g., forest)  later. (This is also consistent with the fact that each location that has been classified as plantation in any of the RSPO maps is also labeled as a plantation in each of the future RSPO maps.)

However, this EVI-based analysis may fail to detect some conversions from plantations to forests and other land overs, and we have added the following discussion about this limitation in Section 5.4.

“Another assumption that underlies our evaluation relative to RSPO (as shown in Figure 7 (e)) is that a location that is a plantation in 2010 is not converted back to other land cover (e.g., forests) in 2014. This assumption is consistent with the fact that each location that has been classified as plantation in any of the RSPO maps is also labeled as a plantation in each of the future RSPO maps.  However, this assumption, if incorrect for some locations, could inflate the errors reported for RSPO.”