Reply to Comment on “Comparison of Cloud Cover Detection Algorithms on Sentinel–2 Images of the Amazon Tropical Forest”

Abstract

In their comments about our paper, the authors remark on two issues regarding our results relating to the MACCS-ATCOR Joint Algorithm (MAJA). The first relates to the sub-optimal performance of this algorithm under the conditions of our tests, while the second corresponds to an error in our interpretation of MAJA’s bit mask. To answer the first issue, we acknowledge MAJA’s capacity to improve its performance as the number of images increases with time. However, in our paper, we used the images we had available at the time we wrote our paper. Regarding the second issue, we misread the MAJA’s bit mask and mistakenly labelled shadows as clouds. We regret our error and here we present the updated tables and images. We corrected our estimation and, consequently, there is an increment in MAJA’s accuracy in the detection of clouds and cloud shadows. However, these increments are not enough to change the conclusion of our original paper.

In our paper [1], we compare four cloud detection algorithms for the Amazon forest using Sentinel images from October 2016 to November 2018. Three of the algorithms (Fmask4, s2cloudless, and Sen2Cor) identify clouds using only spacial features. On the other hand, MACCS-ATCOR Joint Algorithm (MAJA) uses time series of images for this purpose. We compared the results of the four algorithms using points collected and classified by experts on remote sensing.

In their comments, Hagolle and Colin [2] pointed to two issues in the method reported in our paper [1], both related to MAJA. The first issue is related to the sub-optimal performance due to the interval between successive observations. The second issue indicates a misinterpretation of MAJA’s bit mask.

Concerning the first issue, it is noteworthy that in the experimental design, we delimited the evaluation period between October 2016 and November 2018. This period was the same used in the comparison of the evaluated cloud detection algorithms. We recognize that this temporal delimitation may have impacted the accuracy of MAJA, since its detection improves with the inclusion of more images observed over time, as demonstrated in the results presented in [2].

Regarding the second issue, we misread the MAJA’s bit mask. We mistakenly took shadows for clouds. We apologise for our error. To fix this error, the MAJA column in Table 3 of our paper must be ignored and instead one should use

Table 1. Updated version of Table 3 [1]. Label recoding of the detection algorithms.

Expert Label	Fmask4	s2cloudless	Sen2Cor
Clear	0 Clear land	0 Clear	4 Vegetation
	1 Clear water		5 Non vegetated
	3 Snow		6 Water
			11 Snow
Cloud	4 Cloud	1 Cloud	8 Cloud medium probability
			9 Cloud high probability
			10 Thins cirrus
Cloud shadow	2 Cloud shadow		2 Dark area pixels
			3 Cloud shadows
Other			0 No data
			1 Saturated or defective
			7 Unclassified

. To correctly interpret MAJA’s cloud mask, please use

Table 2. Bit interpretation of the MACCS-ATCOR Joint Algorithm’s (MAJA) cloud mask. Source [3].

Bit	Description
0	All clouds except the thinnest and all shadows
1	All clouds (except the thinnest)
2	Cloud shadows cast by a detected cloud
3	Cloud shadows cast by a cloud outside image
4	Clouds detected via mono-temporal thresholds
5	Clouds detected via multi-temporal thresholds
6	Thinnest clouds
7	High clouds detected by 1.38 µm

. This is because of the bit structure used by MAJA.

Figure 1b updates Figure 3d in [1]. This new version clearly identifies cloud shadows for MAJA. On the other hand, Figure 1b in [2] shows larger clear areas than Figure 1b. This is probably due to the differences in the length of the time series used. Our experiment used almost three years of images, most of them from Sentinel–2A because Sentinel–2B came into operation in July 2017, while [2] used images of five years (from 2016 to 2020). We added Figure 1a only as a reference for the readers.

Because of the misinterpretation of MAJA’s bit mask, MAJA’s accuracy in identifying clouds and cloud shadows needs to be updated, while the accuracy of clear observation was not affected. We updated the accuracy reported on Tables 4 and 5 of our paper and now they are presented here in

Table 3. Updated version of Table 5 [1] with updated values of MAJA (in bold face). User and producer accuracies for each tile and cloud-detection algorithm.

	Fmask4			MAJA			s2cloudless			Sen2Cor
Label	F1	User	prod	F1	User	prod	F1	User	Prod	F1	User	prod
Clear	0.90	0.90	0.89	0.73	0.82	0.66	0.44	0.42	0.46	0.77	0.67	0.89
Cloud	0.94	0.91	0.96	0.80	0.69	0.96	0.66	0.59	0.75	0.89	0.90	0.88
C. Shadow	0.79	0.84	0.75	0.20	0.42	0.13			0.00	0.50	0.95	0.34
Overall	0.90			0.71			0.52			0.79

and

Table 4. Updated version of Table 4 [1] with updated values of MAJA (in bold face). User and producer accuracies for each tile and cloud-detection algorithm.

		Fmask4			MAJA			s2cloudless			Sen2Cor
Tile	Label	F1	User	prod	F1	User	prod	F1	User	prod	F1	User	prod
T19LFK	Clear	0.83	0.81	0.86	0.66	0.69	0.63	0.47	0.31	0.94	0.66	0.52	0.92
	Cloud	0.96	0.96	0.96	0.92	0.87	0.97	0.77	0.94	0.66	0.94	0.96	0.92
	C. Shadow	0.68	0.71	0.66	0.15	0.40	0.10			0.00			0.00
	Overall	0.92			0.83			0.64			0.84
T20NPH	Clear	0.91	0.95	0.88	0.78	0.89	0.70	0.53	0.47	0.62	0.84	0.73	1.00
	Cloud	0.95	0.90	1.00	0.78	0.64	0.98	0.57	0.54	0.60	0.93	0.99	0.88
	C. Shadow	0.80	0.82	0.78	0.34	0.55	0.24			0.00	0.59	1.00	0.42
	Overall	0.91			0.72			0.50			0.84
T21LXH	Clear	0.88	0.82	0.95	0.80	0.89	0.72	0.35	0.33	0.36	0.78	0.64	0.99
	Cloud	0.94	0.96	0.92	0.80	0.67	0.98	0.60	0.52	0.70	0.91	0.99	0.83
	C. Shadow	0.81	0.89	0.75	0.17	0.49	0.10			0.00	0.58	0.98	0.41
	Overall	0.90			0.73			0.45			0.81
T22MCA	Clear	0.94	0.94	0.94	0.88	0.83	0.93	0.58	0.62	0.54	0.85	0.74	0.98
	Cloud	0.94	0.90	0.98	0.84	0.74	0.97	0.70	0.56	0.93	0.95	1.00	0.90
	C. Shadow	0.81	0.89	0.74	0.08	0.52	0.46			0.00	0.49	0.87	0.34
	Overall	0.92			0.78			0.58			0.83
T22NCG	Clear	0.87	0.95	0.80	0.42	0.70	0.30	0.23	0.34	0.18	0.63	0.64	0.63
	Cloud	0.87	0.79	0.96	0.63	0.48	0.91	0.61	0.46	0.94	0.71	0.62	0.83
	C. Shadow	0.80	0.82	0.77	0.22	0.30	0.17			0.00	0.53	0.97	0.36
	Overall	0.86			0.51			0.43			0.65

, respectively.

Table 3 shows the updated accuracy of MAJA according to the cloud detection algorithms. After the correction, there is an increment of 0.02 in the overall accuracy of MAJA. Furthermore, Table 4 shows the accuracy of each Sentinel–2 tile. The increments range from 0.01 to 0.05 in overall accuracy by tile.

However, these updates do not change the conclusion in our original paper.