Water Body Extraction from Sentinel-3 Image with Multiscale Spatiotemporal Super-Resolution Mapping

Abstract

Water body mapping is significant for water resource management. In the view of 21 spectral bands and a short revisit time of no more than two days, a Sentinel-3 OLCI (Ocean and Land Colour Instrument) image could be the optimum data source in the near-real-time mapping of water bodies. However, the image is often limited by its low spatial resolution in practice. Super-resolution mapping (SRM) is a good solution to generate finer spatial resolution maps than the input data allows. In this paper, a multiscale spatiotemporal super-resolution mapping (MSST_SRM) method for water bodies is proposed, particularly for Sentinel-3 OLCI images. The proposed MSST_SRM method employs the integrated Normalized Difference Water Index (NDWI) images calculated from four near-infrared (NIR) bands and Green Band 6 of the Sentinel-3 OLCI image as input data and combined the spectral, multispatial, and temporal terms into one objective function to generate a fine water body map. Two experiments in the Tibet Plate and Daye lakes were employed to test the effectiveness of the MSST_SRM method. Results revealed that by using multiscale spatial dependence under the framework of spatiotemporal super-resolution Mapping, MSST_SRM could generate finer water body maps than the hard classification method and the other three SRM-based methods. Therefore, the proposed MSST_SRM method shows marked efficiency and potential in water body mapping using Sentinel-3 OLCI images.

1. Introduction

Water resources are important natural resources for the development of human beings and the ecological sustainability of ecosystems. However, many countries suffer from water scarcity because of several reasons, such as land-use change, natural calamities, and water pollution [1,2]. Water body extraction and mapping are essential and useful in many water applications, including water resource management [3,4], wetland protection [5,6], lake change detection [7,8], flood detection and assessment [9,10,11], and cyanobacteria blooms [12].

Advances in remote sensing technology provide access to water body monitoring and mapping [13,14]. Thus far, many papers have proposed water body mapping approaches from satellite images. Water indices are often employed as a popular and valid way to extract water because they enhance the distinction between water and nonwater through band calculations [15,16]. Existing water extraction methods are mostly based on single-source remotely sensed images such as Moderate resolution Imaging Spectroradiometer (MODIS), Sentinel-3, Landsat, or Sentinel-2. However, images such as Landsat have a relatively high spatial resolution but have a repeat frequency of 16 days, and, sometimes, the data is not available because of bad weather or cloud cover. These characteristics are insufficient to capture short-term water body changes, especially for the daily monitoring of water. On the other hand, images such as Sentinel-3 have high temporal resolution and can provide near-real-time water monitoring but are often limited by their low spatial resolution. In fact, a single source sensor cannot obtain both high spatial resolution and high temporal resolution images at the same time because of the limitations of satellite sensor hardware technology and satellite launch costs. Moreover, most of the single-image-based methods are only calculated on the pixel scale. Water body pixels, especially those at water boundaries, often have mixed class composition [17,18]. Mixed pixels cannot be classified correctly through hard classification because it treats each pixel as a single water/nonwater class. Even if spectral unmixing technologies can determine the ratio of water/nonwater types in a pixel and generate a more accurate water body map, the specific spatial distribution of each mixed water pixel is still indistinguishable [19,20].

Super-resolution mapping (SRM) was first proposed by Atkinson in 1997 to increase the spatial resolution of spectral unmixing results [21]. It aims at producing a higher spatial resolution map by maximizing the spatial correlation or spatial dependence among neighboring subpixels according to the original pixel proportions [22,23,24,25]. SRM has achieved success in producing water maps. Ling et al. employed the digital elevation model (DEM) as an additional tool to extract high-resolution information at the subpixel scale for waterline subpixel mapping [26]. Foody G.M. et al. proposed a subpixel-based method for mapping waterlines from a soft image classification [27]. Li et al. used a locally adaptive SRM method to extract waterlines [28]. Niroumand-Jadidi et al. combined the linear spectral mixture model (LSMM) and pixel swapping (PS) algorithm for the mapping and monitoring of water boundaries [29]. However, existing methods consider only the spatial dependence at the subpixel level and assume the center subpixel is just as affected by its neighboring subpixels. Therefore, the center subpixel is often labeled according to the neighboring subpixels. Nevertheless, without coarse pixel information, maintaining the overall information and spatial continuity in the result of water body maps may be difficult. Spatial dependence at the pixel level also has a significant effect on the center subpixel. By adopting the coarse-pixel-based spatial dependence, the whole water body information for the resultant water body map can be maintained. Therefore, the multiscale spatial dependence at the pixel and subpixel levels should be considered rather than only one of them [30].

Compared with other satellites such as Landsat, SPOT, and HJ-1A, Sentinel-3 has unique advantages, with its 21 spectral bands ranging from visible to near-infrared and almost-two-day revisit cycle providing a new data source for water body monitoring. With such unique advantages, Sentinel-3 OLCI images have been used to draw water quality parameter maps [31], calculate the down-welling diffuse attenuation coefficient of lakes [32], estimate the composition of suspended particulate matter in lakes [33], classify natural colors of water [34], and monitor water levels in rivers [35]. Most of these applications have been performed for quantitative inversion from Sentinel-3 OLCI images because they can produce more continuous spectral curves for the water. However, on account of its sensitivity to water reflection and high temporal resolution, using Sentinel-3 for water body mapping is more interesting. Unfortunately, Sentinel-3 OLCI images have not been widely used in mapping water bodies [36]. One possible reason is the relatively low spatial resolution of 300 m, which results in a large number of mixed pixels, especially those around the boundaries of two classes.

To address this problem, researchers have employed the SRM method to extract the water body. SRM is intended to generate a land-cover map with higher spatial resolution than the input data [37,38]. From this perspective, SRM can be utilized to generate water body maps of finer spatial resolution from the Sentinel-3 images. Wang et al. [36] proposed an unsupervised sub-pixel water body mapping (USWBM) method to extract water from Sentinel-3 images to draw 30-m water body maps. Compared with traditional methods, USWBM obtained results with improved accuracy. However, the resultant water body maps have limitations in terms of spatial details of some water features, particularly for small patches and linear features. The accuracy needs to be improved because it only uses one coarse spatial resolution Sentinel-3 image as the input. The successful launch of numerous high-resolution satellites has produced abundant high spatial resolution remotely sensed images, with which finer water body maps can be generated. These archived finer water body maps can provide prior spatial information for the processing of subpixel locating. Spatiotemporal super-resolution mapping (STSRM), which is developed based on SRM, can realize the fusion of multisource remote sensing data with different spatial and temporal resolutions. Finally, by combining the advantages of a short period, the many spectral bands of Sentinel-3 images, and the abundant archived high spatial resolution images, high spatial–temporal resolution water body maps can be generated.

In this study, a water body extraction method under the multiscale spatial–temporal super-resolution mapping (MSST_SRM) method is put forward for Sentinel-3 OLCI images to overcome the above problem of water body extraction. First, an integrated NDWI image calculated directly from the original Sentinel-3 image is used as model input for the processing of water body mapping. Then, MSST_SRM is implemented based on four terms: the spectral water index term, the subpixel/pixel spatial dependence term, and the temporal term. In particular, the spectral water index term is grounded on the Markov random field (MRF) approach; spatial dependence is calculated at the multiscales, and a transfer matrix is employed in the temporal term to calculate the change in the water body.

The rest of the paper is organized as follows. Detailed introductions of the MSST_SRM method is presented in Section 2. Results and analysis of the implementations are outlined in Section 3. Discussions are shown in Section 4. Finally, Section 5 concludes.

2. Methods

Figure 1 shows the flowchart of the proposed MSST_SRM approach. After preprocessing the Sentinel-3 OLCI image, five bands, four NIR bands, and one green band were employed to calculate the NDWI images. Normally, in Sentinel-3, Band 17 is regarded as the NIR band in the NDWI calculation. However, using one NIR band of Sentinel-3 OLCI would be too narrow compared with Landsat sensors because the wavelength range of NIR in Landsat covers almost four NIR bands (Bands 17–20) in Sentinel-3. Hence, to ensure the same bandwidth of NIR and a comprehensive description of the water features, NIR bands from 17 to 20 of Sentinel-3 were used to calculate the NDWI images, and all precalculated NDWI images were combined into one image as input for the MSST_SRM model. MSST_SRM is implemented by the following four terms. Detailed information on these four terms is discussed in Section 2.1, Section 2.2, Section 2.3 and Section 2.4. In addition, Section 2.5 discusses the implementation of MSST_SRM and some validation methods.

2.1. MSST_SRM

The target of MSST_SRM is to generate a finer water body map, X, from the combined NDWI image Y with the assistance of the previous water body map $X_{former}$. Suppose the spatial resolution of the Sentinel-3 image is R and the scale factor is $z$. Then, the spatial resolution of NDWI images Y = [$y_1,y_2,y_3\dots y_i\dots y_n$] is also R because it is calculated from the Sentinel-3 image. Here, $y_i$ is a single NDWI image, and n is equal to 4. Four NDWI images comprise the integrated input NDWI image. The spatial resolution of $X_{former}$ and X are both $R∕z$. Each pixel in image Y includes $z\times z$ pixels in $X_{former}$ and X. In general, the optimal MSST_SRM results, X, is a minimized optimization problem. The objective function of MSST_SRM is composed of four terms and expressed as follows:

$$U_{opt}=\arg \left\{\min \left(U_{index\_spectral}+\alpha \left[\delta U_{spatial\_sp}+\left(1-\delta \right)U_{spatial\_cp}\right]+\beta U_{temporal}\right)\right\}$$

(1)

where $U_{index\_spectral}$, $U_{spaial\_sp}$, $U_{spatial\_cp}$, and $U_{temporal}$ are the energy functions of the four parts: a spectral term for the water index, a spatial term at the subpixel scale, a spatial term at the pixel scale, and a temporal term, respectively. The weighting coefficients α, $\delta$, and β control the weights of the four terms.

2.2. Spectral Water Index Term

The goal of this term in MSST_SRM is to make the class proportions in the resultant water body map consistent with the input NDWI images. The Markov random field (MRF) method [39,40] has been successfully applied as a spectral term in SRM to mitigate the effects of classification errors on change detection. The energy function $U_{index\_spectral}$ is expressed as the following:

$$U_{index\_spectral}={\displaystyle \sum _{j=1}^{z^2}{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{\omega =1}^C\left[\frac{1}{2}\left(y\left(x_i\right)-V_i\right){}^{\prime }{M_i}^{-1}\left(y\left(x_i\right)-V_i\right)+\frac{1}{2}\ln \left|M_i\right|\right]}}}$$

(2)

where z is the scale factor, equal to 10, and C is the number of classes, equal to 2. $D$ represents the number of pixels in NDWI image Y. $x_i$ is a coarse pixel in NDWI image at the current time, y($x_i$) is the observed pixel’s spectral value of $x_i$, assuming that $x_i$ is allocated with the normal distribution of the mean vector $V_i$ and the covariance matrix $M_i$, according to the category proportions.

$$V_i={\displaystyle \sum _{\omega =1}^C{f}_{\omega i}{V}_{\omega }}$$

(3)

$$M_i=\sum _{\omega =1}^C{f}_{\omega i}{M}_{\omega }$$

(4)

where $f_{{\omega }_i}$ is the category proportion of class $\omega$ in pixel $x_i$. The value of $f_{{\omega }_i}$ is obtained from the fraction images of Y.

2.3. Multispatial Term

The multispatial term includes two parts, namely, the subpixel spatial dependence term and the pixel spatial dependence term. In MSST_SRM, each subpixel is surrounded by two neighborhood systems. According to the maximum spatial dependence rule, the center subpixel is affected by the neighboring pixels and subpixels. Suppose the pixel neighborhood window size is 5, and the subpixel neighborhood window size is 7. Figure 2 shows an example to illustrate the multispatial neighborhood system. The center subpixel is affected by the surrounding 48 neighboring subpixels and 24 neighboring pixels. Detailed subpixel and pixel spatial dependence and calculation methods are described in Section 2.3.1 and Section 2.3.2.

2.3.1. Subpixel-Based Spatial Dependence Term

This term intends to locally smoothen the resultant water body map. In SRM, the maximum spatial dependence rule is a popular method for expressing the spatial correlation and spatial pattern among subpixels. Under this rule, subpixels that are nearby may have similar class labels than those with longer distances [19,30]. MSST_SRM adopts the maximum spatial dependence rule to express the spatial relationships between the center subpixel and its neighbors. The subpixel spatial energy function $U_{spatial\_sp}$ is formulated as follows:

$$U_{spatial\_sp}=-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^{z^2}{\displaystyle \sum _{l\in N_{sp}(x_{j,i})}\theta \left(c\left(x_{j,i}\right),c\left(x_l\right)\right)·\eta \left(x_{j,i},x_l\right)}}}$$

(5)

where $x_{j,i}$ is the $j\mathrm{th}$ subpixel in $i\mathrm{th}$ coarse pixel. $x_l$ is the neighboring subpixel of $x_{j,i}$. $N_{sp}\left(x_{j,i}\right)$ represents the subpixel neighborhood system for $x_{j,i}$. Suppose the neighborhood window size is $w$, the neighborhood system of $x_{j,i}$ is a $w\times w$ square window. $x_{j,i}$ is located in the center of the window, with $\left(w\times w-1\right)$ subpixel neighbors. $c\left(x_{j,i}\right)$ is the class label of $x_{j,i}$ and $c\left(x_l\right)$ is the class label of $x_l$. $\theta \left(c\left(x_{j,i}\right),c\left(x_l\right)\right)$ aims to show the land cover spatial dependence and is calculated as 1 if $c\left(x_{j,i}\right)$ equals $c\left(x_l\right)$. Otherwise, it is 0. The spatial weighting function $\eta \left(x_{j,i},x_l\right)$ weighs the contribution of $x_l$ to $x_{j,i}$, which is calculated as follows:

$$\eta \left(x_{j,i},x_l\right)=\frac{1}{\Omega }{\left(d\left(x_{j,i},x_l\right)\right)}^{-1}$$

(6)

where $\Omega$ is the selected normalization constant to ensure ${\sum }_{lϵN_{sp}\left(x_{j,i}\right)}\eta \left(x_{j,i},x_l\right)=1$. $d\left(x_{j,i},x_l\right)$ is the distance between $x_{j,i}$ and $x_l$.

2.3.2. Pixel-Based Spatial Dependence Term

The goal of this term is to provide holistic water information for the results. Under the principle of this term, the class label of each center subpixel is decided by its adjacent coarse pixels. The objective function of this term is to maximize the spatial correlation of neighboring coarse pixels and it is expressed as

$$U_{spatial\_cp}=-{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{j=1}^{z^2}{\displaystyle \sum _{t\in N_{cp}(x_{j,i})}\varphi \left(c\left(x_{j,i}\right),c\left(x_t\right)\right)}}}·\eta \left(x_{j,i},x_t\right),$$

(7)

where $x_{j,i}$ is the $j\mathrm{th}$ subpixel in $i\mathrm{th}$ coarse pixel, $x_t$ is the $t\mathrm{th}$ neighboring coarse pixel around subpixel $x_{j,i}$. $N_{cp}\left(x_{j,i}\right)$ is the pixel neighborhood system for $x_{j,i}$. Suppose $W$ is the pixel neighborhood window size, which has the same meaning as $w$ in Section 2.3.1. $\varphi \left(c\left(x_{j,i}\right),c\left(x_t\right)\right)$ is the spatial coefficient of the neighboring coarse pixel $x_t$ at the class of $c\left(x_{j,i}\right)$ indicating the contribution of $x_t$. The spatial weighting function $\eta \left(x_{j,i},x_t\right)$ is formulated as follows:

$$\eta \left(x_{j,i},x_t\right)=e^{-d^2\left(x_{j,i},x_t\right)/{\epsilon }^2}$$

(8)

where $\epsilon$ is the distance–decay parameter, and $d\left(x_{j,i},x_t\right)$ represents the Euclidean distance of $x_{j,i}$ and its neighboring coarse pixel $x_t$.

The spatial coefficient $\left(c\left(x_{j,i}\right),c\left(x_t\right)\right)$ is calculated according to the category proportion of the neighboring coarse pixels. The category proportion can be obtained from the fraction image through spectral unmixing. Suppose $x_q$ is another neighboring coarse pixel of subpixel $x_{j,i}$, then $\varphi \left(c\left(x_{j,i}\right),c\left(x_t\right)\right)$ is formulated as

$$f_{c\left(x_{j,i}\right)}\left(x_q\right)={\displaystyle \sum _{t=1}^{N_{cp}}\varphi }\left(c\left(x_{j,i}\right),c\left(x_t\right)\right)·\eta \left(x_q,x_t\right)$$

(9)

where $f_{c\left(x_{j,i}\right)}\left(x_q\right)$ represents the category proportion of the pixel $x_q$ when the class label is $c\left(x_{j,i}\right)$. The spatial correlation function $\eta \left(x_q,x_t\right)$ is used to calculate the spatial correlation between $x_q$ and $x_t$. The expression of $\eta \left(x_q,x_t\right)$ is similar to Equation (8). After $f_{c\left(x_{j,i}\right)}\left(x_q\right)$ and $\eta \left(x_q,x_t\right)$ are obtained, the solution of $\varphi \left(c\left(x_{j,i}\right),c\left(x_t\right)\right)$ can be resolved by the matrix in Equation (10):

$$\left[\begin{array}{ccc}\eta \left(x_1,x_1\right)& \cdots & \eta \left(x_1,x_{N_{cp}}\right)\\ \eta \left(x_2,x_1\right)& \cdots & \eta \left(x_2,x_{N_{cp}}\right)\\ ⋮& ⋮& ⋮\\ \eta \left(x_{N_{cp}},x_1\right)& \cdots & \eta \left(x_{N_{cp}},x_{N_{cp}}\right)\end{array}\right]\times \left[\begin{array}{c}\varphi \left(c\left(x_{j,i}\right),c\left(x_1\right)\right)\\ \varphi \left(c\left(x_{j,i}\right),c\left(x_2\right)\right)\\ ⋮\\ \varphi \left(c\left(x_{j,i}\right),c\left(x_{N_{cp}}\right)\right)\end{array}\right]=\left[\begin{array}{c}{f}_{c\left(x_{j,i}\right)}\left(x_1\right)\\ f_{c\left(x_{j,i}\right)}\left(x_2\right)\\ ⋮\\ f_{c\left(x_{j,i}\right)}\left(x_{N_{cp}}\right)\end{array}\right]$$

(10)

The spatial energy function $U_{spaial}$ combining subpixel and coarse pixel spatial dependence terms are expressed as follows:

$$U_{spatial}=\delta Uspatial\_sp+\left(1-\delta \right)Uspatial\_cp$$

(11)

where $\delta$ controls the weight of the subpixel spatial term.

2.4. Temporal Term

Earlier archived images with high spatial resolution can provide prior information and spatial patterns for the processing of subpixel location. However, the water body changes over time. In this paper, transition probability $P$, which represents the change and transfer laws of the land-cover class, is employed in the temporal term to maintain the correlation between the previous water body map $X_{former}$ and the current NDWI image Y. In the subpixel temporal neighborhood system $N_t$, the subpixel at the current time is affected by its temporal neighbors at the former time. The degree of influence is usually expressed as the transition probability. In general, the higher the transfer probability for a certain class of subpixel, the lower the temporal energy function of this class should be. Therefore, the objective function $U_{temporal}$ can be modeled as follows:

$$U_{temporal}={\displaystyle \sum _{j=1}^{z^2}{\displaystyle \sum _{i=1}^D{\displaystyle \sum _{k\in N_t(x_{j,i})}-P\left(c_2\left(x_{j,i}\right)|c_1\left(x_k\right)\right)}}}$$

(12)

where $c_2\left(x_{j,i}\right)$ is the class label of subpixel $x_{j,i}$ at the current time. $c_1\left(x_k\right)$ is the class label of the fine pixel $x_k$ at the previous time. $P\left(c_2\left(x_{j,i}\right)|c_1\left(x_k\right)\right)$ is the transition probability from $x_k$ at the previous time to $x_{j,i}$ at the current time. In MSST_SRM, $P$ is calculated based on the global transition model by comparing the previous water body map and current SR map pixel by pixel. Suppose the land cover class is C. In this study, there are only two classes, so C equals to 2. Transition probability matrix T is composed of $C\times C$ transition probabilities $P$.

$$T=\left[\begin{array}{cc}P\left(c_2(•)={\omega }_1|c_1(•)={\omega }_1\right)& P\left(c_2(•)={\omega }_2|c_1(•)={\omega }_1\right)\\ P\left(c_2(•)={\omega }_1|c_1(•)={\omega }_2\right)& P\left(c_2(•)={\omega }_2|c_1(•)={\omega }_2\right)\end{array}\right]$$

(13)

where $c_1(•)$ is the class label of the arbitrary fine pixel at the previous time, and $c_2(•)$ represents the class labels of an arbitrary subpixel at the current time. $P$ is calculated as [40].

$$P\left(c_2(•)={\omega }_b|c_1(•)={\omega }_a\right)=\frac{{\displaystyle {\sum }_{m=1}^{n\times z^2}I\left(c_1\left(x_m\right)={\omega }_a\begin{array}{c}\end{array}AND\begin{array}{c}\end{array}{c}_2\left(x_m\right)={\omega }_b\right)}}{{\displaystyle {\sum }_{m=1}^{n\times z^2}I\left(c_1\left(x_m\right)={\omega }_a\right)}}$$

(14)

where $I\left(c_1\left(x_m\right)={\omega }_a\right)$ is an indicator function equaled to 1 when $c_1\left(x_m\right)={\omega }_a$ and 0 otherwise.

2.5. Implementation of MSST_SRM and Accuracy Validation

Generally, the process of obtaining the optimal solution of the MSST_SRM algorithm is to minimize the objective function. In this paper, the iterative conditional model (ICM) is adopted as a fast convergence algorithm to minimize global energy iteratively. The main steps of the MSST_SRM are as follows:

(1)

Initialize the input data and parameters and generate NDWI images by calculating Band 6 and Bands 17–20 from Sentinel-3 OLCI; set the proper values for scale factor z, key parameters α, $\mathsf{\delta }$, and β, and window sizes $w$ and $W$.

(2)

Extract the endmember and then obtain fraction images by using the linear unmixing method. Then, initialize the SR map based on it.

(3)

Calculate the transition probability matrix.

(4)

Update the subpixel labels iteratively all over the whole image. The class label for each subpixel that contributes to the last iteration process is regarded as the final class label for this subpixel.

(5)

Stop the iteration after the algorithm converges or the iteration exceeds the setting number; otherwise, step 4 is repeated.

Kappa coefficient, overall accuracy (OA), PULC, and PCLC are employed to assess the accuracy of the proposed method. PULC and PCLC represent the correct classification ratio of the unchanged and changed pixels, respectively.

3. Experiments and Analysis

3.1. Experiment 1

3.1.1. Study Area

The Tibetan Plateau is located in central Asia, and it is the Earth’s broadest and highest highland. With its rich water resources, the Tibetan Plateau is also called the “Water Tower of Asia”. The Tibetan Plateau is not only the source of many large rivers in East Asia and Southeast Asia, but it also has a great number of alpine lakes and is the most concentrated area of lakes in China. The study area (Figure 3) is located in the Tibetan Plateau.

3.1.2. Data Sources and Preprocessing

Three kinds of data were used in Experiment 1 (Figure 4) as inputs, namely, the current Sentinel-3 image, previous Landsat image providing additional prior information for the processing of subpixel locating, and current Landsat image used for model validation. Specifically, the Sentinel-3A OLCI image was acquired on 25 July 2019 and downloaded from the European Space Agency (https://scihub.copernicus.eu). The Sentinel Toolboxes of Sentinel Application Platform (SNAP) is used to preprocess the Sentinel-3A image for cloud detection, atmospheric correction, and coordinate transformation. Image correction for atmospheric effects (iCOR) is used for OLCI Sentinel-3 atmospheric correction. The coverage area of the Sentinel-3A image is 12 × 12 km, containing 40 × 40 pixels and 21 bands. The two Landsat images were obtained from the United States Geological Survey (USGS) Earth Explorer (https://earthexplorer.usgs.gov). The previous image of Landsat-5 was obtained for 28 August 2009, and the reference image of Landsat-8 was acquired on 25 September 2019. The Landsat acquisition period was the same as the Sentinel-3A image in the summer season to ensure the same water level in the Sentinel-3A image. The two Landsat images were divided into water and nonwater classes to extract water body maps and they contained 400 × 400 pixels and 9 bands. The changed pixels of water accounted for 11.31% of all water pixels.

Model parameters were initialized after data preprocessing. The pixel- and subpixel-based neighborhood window sizes were both set to 7, which indicates that the center subpixel has 48 neighboring pixels and subpixels. The value of scale factor $z$ was 10. Weighting coefficients of α, $\mathsf{\delta }$, and β were gained by trying and testing. Kappa coefficient, overall accuracy (OA), PULC, and PCLC were employed to calculate the accuracy of the result.

3.1.3. Results and Analysis

Four additional approaches, namely, hard classification (HC), MSS_SRM, MCT_SRM, and MST_SRM were used for comparison. HC is a pixel-based method that labels the principal component class with the pixel class label according to the fraction images. MSS_SRM is an SRM-based method with a multiscale spatial term, but it uses only one single remotely sensed image regardless of the temporal term. MCT_SRM and MST_SRM are methods based on spatiotemporal super-resolution mapping (STSRM), but they use only single scale spatial dependence. Specifically, MCT_SRM adopts a pixel-based spatial dependence without considering its subpixel spatial neighbors, while MST_SRM utilizes subpixel-based spatial dependence. The resultant water body maps, generated through the different methods, are shown in Figure 5.

Table 1. Confusion matrices of the different methods for Tibet Plate Lake.

Methods		Water	Nowater
HC	water	64,019	10,681
	nonwater	5740	79,560
MSS_SRM	water	53,781	2164
	nonwater	15,978	88,077
MCT_SRM	water	57,607	118
	nonwater	12,152	90,123
MST_SRM	water	61,503	393
	nonwater	8256	89,848
MSST_SRM	water	62,296	439
	nonwater	7463	89,802

presents the confusion matrices of the five different methods. Based on these confusion matrices, the corresponding quantitative results of these five different methods are obtained, as shown in

Table 2. Accuracy statistics of the different methods for Tibet Plate Lake.

Methods	Kappa	OA(%)	PULC(%)	PCLC(%)
HC	0.7930	89.74	91.03	64.83
MSS_SRM	0.7642	88.67	91.60	32.07
MCT_SRM	0.8410	92.33	96.87	4.82
MST_SRM	0.8887	94.59	99.46	0.65
MSST_SRM	0.8984	95.06	99.99	4.51

.

From the visual and qualitative perspective of the above results, the MSST_SRM-based method is significantly superior to the other four methods. In Figure 5b, the resultant water body map of the HC-based method has many jagged boundaries and shows poor continuity because, in HC, only pixel scale land cover information is considered, making it easy to lose spatial details. In Figure 5c, multiscale maximal spatial dependence was adopted in MSS_SRM, which considers the spatial neighborhood relationship at both pixel and subpixel scales. The resultant water body map of MSS_SRM becomes locally oversmoothed because of the multispatial dependence model and lack of temporal term. Thus, while the MCT_SRM method can provide a relatively continuous water body map, it contains many discontinuous speckles because of the lack of spatial constraint at the subpixel level (Figure 5d). Compared with the three previously mentioned methods, the resultant water body map of MST_SRM has better visual results (Figure 5e) and higher resulting accuracy (Table 2). However, some detailed information on the resultant water body map in Figure 5e was lost, especially for the linear features and small water blocks (highlighted by red circles).

By contrast, as shown in the red circle, the resultant water body map generated by MSST_SRM (Figure 5f) can maintain better spatial patterns and water structure information than the other four results. Moreover, MSST_SRM also had the highest quantitative evaluation results (Table 2). The main reason for the good results is that the MSST_SRM method considers multiscale spatial dependence and additional prior temporal information. With the help of a former water body map of Landsat-5, a valuable spatial pattern can be provided for the current Sentinel-3 image. For unchanged areas, the class label of the subpixels can be obtained directly from the former water body map of Landsat-5 of 2009. Thus, using multiscale spatial dependence, the neighboring subpixels and pixels can provide prior spatial information for the center subpixel.

The proposed approach was compared with a previous approach, which was also calculated for Tibet Plate Lake. Wang et al. [36] used a USWBM method based on a spectral water index to produce a 30-m water body map from the Sentinel-3A image and obtained optimal results with kappa = 0.8430 and OA = 92.37%. By contrast, MSST_SRM generated better results, with kappa = 0.8984 and OA = 95.06%, which indicates that MSST_SRM can produce better results closer to the reference water body maps.

3.2. Experiment 2

3.2.1. Study Area

A Sentinel-3 OLCI image and Landsat multispectral images (Figure 6), taken over at Huangshi City, were employed in this experiment to further explore the proposed MSST_SRM method. Daye Lake is located in the southeast of Daye city; it crosses Daye city, Yangxin county, the Xisai mountain area, and the development zone of Huangshi and then flows into the Yangtze River from west to east at the Sigu floodgate. Daye Lake is rich in natural bait and suitable for fish and algae reproduction, which makes it an important water source for aquatic economic plants and fishery breeding.

3.2.2. Data Sources and Preprocessing

The experiment was implemented with the Sentinel-3A OLCI image acquired on 25 July 2019 and two Landsat images acquired on 29 August 1997 and 25 July 2019 (Figure 7). The Landsat-5 TM image from 1997 was divided into nonwater and water classes and used to offer additional prior information for the processing of subpixel location. The Landsat-8 OLI image from 2019 was also classified into two categories and used as a reference water body map. The acquisition and data preprocessing methods of these three images are the same as those in Experiment 1. The study area in the Sentinel-3A OCLI image contains 70 × 40 pixels and 21 bands. The two Landsat images contain 700 × 400 pixels and 9 bands. Scale factor $z$ was set to 10 and the neighborhood window of subpixels and pixels was set to 7. The changed pixels of water accounted for 32.63% of all water pixels.

3.2.3. Results and Analysis

Comparative experiments among HC, MSS_SRM, MCT_SRM, MST_SRM, and MSST_SRM were conducted to verify the proposed MSST_SRM method for Daye Lake (Figure 8). Confusion matrices of the different methods are showed in

Table 3. Confusion matrices of the different methods for Daye Lake.

Methods		Water	Nonwater
HC	water	65,342	13,658
	nonwater	8212	192,788
MSS_SRM	water	64,904	12,762
	nonwater	8650	193,684
MCT_SRM	water	64,354	9349
	nonwater	9200	197,097
MST_SRM	water	61,739	7562
	nonwater	11,815	198,884
MSST_SRM	water	64,149	8668
	nonwater	9405	197,778

. Four quantitative statistical results were adopted to assess the results (

Table 4. Accuracy statistics of the different methods for Daye Lake.

Methods	Kappa	OA(%)	PULC(%)	PCLC(%)
HC	0.8031	92.18	94.64	66.08
MSS_SRM	0.8061	92.35	94.74	66.88
MCT_SRM	0.8291	93.38	97.60	48.31
MST_SRM	0.8180	93.08	98.81	31.96
MSST_SRM	0.8328	93.55	98.01	45.88

). Tests and errors were used to achieve the best value of the weighting coefficients.

The comparison maps in Figure 8 show that the results of MSST_SRM can maintain better water structure. Results in Figure 8b have boundaries with many jags and low continuity because the HC method classification operation is calculated only on the pixel scale. The resultant water body map produced by MSS_SRM contains many over-smoothed patches (Figure 8c) because of the lack of temporal term. The kappa coefficient and OA of MCT_SRM have a higher value than those of HC and MSS_SRM (Table 4), and thus, the resultant water body map generated by MCT_SRM has low continuity (Figure 8d). In Figure 8e, the resultant water body map contains many speckle-like artifacts. Although MCT_SRM and MST_SRM methods have employed the former finer resolution water body map to learn the spatial pattern, the resultant water body maps still did not perform well because both methods only applied single scale spatial dependence either on the pixel scale or on the subpixel scale.

Compared with Tibet Plate Lake, the water in Daye Lake has more mixture, such as algae and sediments, which may be reflected in the NIR, making it more difficult to extract water from Daye Lake. However, among the above methods, the resultant water body map generated by MSST_SRM has the best performance (Figure 8f) and the highest OA and kappa coefficient values (Table 4). Qualitative and quantitative results show that multiscale spatial dependence and additional temporal terms are significant in increasing the accuracy of the results.

4. Discussion

4.1. Influences of Weighting Coefficients among Spectral, Multispatial, and Temporal Terms

The objective function includes spectral, multispatial, and temporal terms. Weighting coefficients α and β, which are used to balance the three big terms, play significant roles in water body mapping. α is a spatial weight coefficient that adjusts the contribution value of the whole multispatial term. The higher the value of α, the greater the spatial dependence among pixels. β is used as a temporal weigh coefficient. A higher value of β means the previous water body map has a greater effect on the results.

Different water body maps can be generated by using different αs and βs. Four typical water body maps are displayed here to elaborate on the effects of the different coefficient combinations on the results. In Figure 9a, the temporal coefficient β is equal to 0, indicating the ineffectiveness of temporal dependence. Moreover, spatial coefficient α was at a low value of 0.01. Finally, the resultant water body map in Figure 9a has many jagged boundaries and poor continuity because of the absence of prior information and imperfect spatial pattern. In Figure 9b, α was set to 1, and β was set at a low value of 0.1, which means the previous water body map only plays a minor role in the process of water body generation and that the results would be extremely dependent on the spatial term. As a consequence, the results were over-smoothed, and the linear water body features could not be classified correctly. In Figure 9c, the temporal coefficient β was set to 1. Spatial coefficient α was set to 0, indicating that the multispatial term was unavailable in such a situation. The resultant water body map contained many speckle-like artifacts because of the missing constraint of spatial dependence. By contrast, with appropriate weight coefficients, the results displayed in Figure 9d had the best performance, with a better spatial pattern that not only represented linear water body features correctly but also contained fewer speckle-like artifacts.

The quantitative accuracy values of the proposed MSST_SRM method are shown in

Table 5. Accuracy statistics of the MSST_SRM method with different weighting coefficients (δ = 0.6).

Result Statistic		α	0	0.01	0.1	1	10	100
	β
Kappa	0		0.7307	0.7346	0.7506	0.7642	0.7560	0.7595
	0.01		0.7414	0.7458	0.7605	0.7653	0.7590	0.7593
	0.1		0.7757	0.7776	0.7825	0.7731	0.7596	0.7593
	1		0.8418	0.8423	0.8409	0.8181	0.7734	0.7603
	10		0.8960	0.8980	0.8960	0.8961	0.8712	0.7716
	100		0.8979	0.8980	0.8980	0.8980	0.8984	0.8773
OA (%)	0		87.02	87.21	87.99	88.66	88.37	88.51
	0.01		87.54	87.75	88.46	88.71	88.44	88.45
	0.1		89.20	89.28	89.53	89.09	88.47	88.46
	1		92.36	92.38	92.31	91.24	89.13	88.51
	10		94.94	94.95	94.95	94.95	93.77	89.05
	100		95.04	95.04	95.04	95.04	95.06	94.06
PULC (%)	0		88.11	89.92	90.76	91.60	91.59	91.79
	0.01		90.53	90.76	91.37	91.68	91.64	91.73
	0.1		97.92	92.76	92.89	92.22	91.72	91.74
	1		96.73	96.74	96.64	95.28	92.55	91.80
	10		99.86	99.86	99.86	99.87	98.38	92.53
	100		99.96	99.97	99.97	99.97	99.99	98.70
PCLC (%)	0		31.74	34.03	35.77	31.07	26.39	25.28
	0.01		29.91	29.81	32.33	31.50	26.24	25.16
	0.1		21.34	22.19	24.68	28.79	25.89	25.15
	1		8.00	8.28	8.89	13.41	23.15	24.94
	10		0.13	0.13	0.13	0.09	4.79	21.78
	100		0.01	0.01	0.01	0.01	4.51	4.58

. When β is set to 0, no matter how α increases, the kappa coefficient will not exceed 0.76 and OA remains at a relatively low value because, when β equals 0, the temporal term will be useless in the object function, which would result in great uncertainty in subpixel location. Meanwhile, if the value of α is fixed, the kappa coefficient, PULC, and OA will be higher when β is higher. However, the accuracy values tend to be stable. After that, the accuracy values remain almost unchanged, no matter how much β increases. The final optimal parameters were obtained by trial and error. In general, the value of PULC is higher than PCLC, especially when β is higher. The main reason for this result is that the previous finer water body map is ineffective for changed pixels. Therefore, the correct classification probability of changed pixels is lower than for unchanged pixels.

4.2. Influences of Change Rate

Land cover changes always occur, and the change rate of the water body should be considered in practical applications. The change rate of the water body means the proportion of changed water pixels in all water pixels, which is calculated by comparing the current image with the previous image.

In general, the accuracy of the results, such as kappa and OA, is determined by the classification precision of changed and unchanged water bodies. For unchanged pixels, the class labels can be inherited from the former water body map directly. Therefore, the result accuracy will be higher when the change rate is smaller. For changed pixels, the spatial pattern of the water body is obtained from the current image and the spatial dependence rule. Obviously, in the processing of SRM, the uncertainty of changed pixels is greater than with the unchanged ones.

In this study, the change rate of water in the two experiments is 11.31% and 32.63%, respectively. The time interval of the former and latter images in Experiment 2 is 22 years. Even so, the two results show that the MSST_SRM method can still obtain the best result among the contrast experiment methods (HC, MSS_SRM, MCT_SRM, and MST_SRM). The above experiments also suggest the following: (1) A higher precision water body map is more likely to be produced at a smaller change rate because the proposed MSST_SRM method usually gains more accurate results for unchanged pixels than changed ones; (2) among the three STSRM-based methods that employ the former fine resolution water body map as prior information, the proposed MSST_SRM method is superior to the other methods (MCT_SRM and MST_SRM) in quantity and quality, thereby indicating that with the help of multiscale spatial dependence, MSST_SRM can provide better spatial patterns and maintain the water structure well.

The changed and unchanged water body classifications affect the resultant water body map. Therefore, to achieve higher result accuracy, the trade-off between changed and unchanged pixels should be considered. First, a balance should be made between the previous finer water body map and the current low-resolution image. Using training data to adjust the weighting parameters is a significant and credible way to obtain balance and gain higher accuracy. Second, using multiscale spatial dependence is an effective and promising means of promoting the performance of the current image. Third, it is better to use Sentinal-3 OLCI and Landsat-8 images acquired at the nearest time and in the same season to reduce the water change rate and to make full use of the prior information provided by the previous high spatial resolution remotely sensed images. In future research, adopting a series of previous finer resolution water body maps to acquire land-cover change rules may be a worthwhile method for further improving the result accuracy of unchanged pixels.

5. Conclusions

In this work, a multiscale spatiotemporal super-resolution method was proposed to produce a water body map with high spatiotemporal resolution from Sentinel-3 images. The proposed MSST_SRM method not only extends the application field of Sentinel-3 but also provides data support for water monitoring and water mapping. First, to ensure the stable performance of the proposed algorithm under different water features, MSST_SRM utilizes four combined NDWI images as input data from the band calculations of the Sentinel-3 image. Then, MSST_SRM unites the spectral, multispatial, and temporal terms into one objective function, where the spectral term is established according to the MRF criterion; the multispatial term is realized by the maximal spatial dependence principle on the pixel and subpixel scales, and the temporal term is constructed through the transition probability matrix between the latter and former images. As a consequence, by using temporal dependence, the spatial pattern uncertainty caused by a single image can be effectively reduced. With the help of multiscale spatial dependence, the integrity and spatial continuity of the entire image can be maintained well.

Two experiments at Tibet Plate Lake and Daye Lake were employed to validate the performance of MSST_SRM. Compared with the pixel-based method (HC), the no-temporal-information SRM-based method (MSS_SRM), and the two STSRM-based methods of single scale spatial dependence (MST_SRM and MCT_SRM), the proposed MSST_SRM method could generate a water body map with high result accuracy. The comparable visual results and higher precision values of the new MSST_SRM method show that MSST_SRM could be a promising approach in producing high spatiotemporal water body maps from Sentinel-3 OLCI images.