# Mitigating Imbalance of Land Cover Change Data for Deep Learning Models with Temporal and Spatiotemporal Sample Weighting Schemes

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methodology

#### 2.1. Study Area and Datasets

^{2}, with 2076 km

^{2}of average annual change (Figure 1b).

#### 2.2. Capturing Neighborhood Effects in Land Cover Data Samples

^{2}area around each cell in the study region.

#### 2.3. Model Specifications

#### 2.4. Categorical Cross-Entropy Loss

#### 2.5. Calculating Temporal and Spatiotemporal Sample Weights

- Unweighted (base case or “none”), where no sample weights were used;
- Binary weights (BW), where a traditional inverse frequency weighting scheme used the inverse frequency of changed versus persistent sample counts to assign sample weights;
- Temporal weighting scheme 1 (TW1), where the inverse temporal distance weight was computed with respect to the most recent change of the central cell;
- Temporal weighting scheme 2 (TW2), where the inverse temporal distance weight was computed with respect to the most recent change of the cell’s neighborhood;
- Spatiotemporal weighting scheme (STW), where the inverse spatiotemporal distance weight was calculated with respect to the most recent change that occurred within the neighborhood of the central cell.

**Step 1.**Identify whether a change has occurred at the sample location, or central cell of the neighborhood (Figure 2). If the number of change incidents was one or greater from 2001 to 2014, the cell was considered as changed.

**Step 2.**Compute the inverse frequency weight according to the counts of persistent and changed cells. Similar to another research study [54], the initial weights of the samples were calculated on the basis of their overall type. For LC data, the initial weight of the changed samples was ${b}_{{c}_{i}}=P/\left(P+C\right)$, and the weight of persistent samples was ${b}_{{p}_{i}}=C/\left(P+C\right)$, where the number of changed cells is denoted by C, and the number of persistent cells is denoted by P. Persistent samples were assigned non-zero weights because they were still important to the learned model structure. Therefore, persistent sample weights $({w}_{{p}_{i}})$ assumed the values of ${b}_{{p}_{i}}$ and required no further updates. This concludes the calculations required to implement the BW scheme, while temporal and spatiotemporal variation was added to changed sample weights in Step 3 for the TW1, TW2, and STW schemes.

**Step 3.**With the effect of persistent cells managed in Step 2, the sample weight calculations for the TW1, TW2, and STW schemes were applied to the changed sample weights $({w}_{{c}_{i}})$ as a function of the temporal and spatial variation occurring at the central cell and within its neighborhood over time. To implement the temporal weighting schemes (TW1 and TW2), the temporal distance from the most recent year of the training sample was computed with respect to the most recent change at the central cell $({d}_{cc})$ and to the year of the most recent change occurring in the neighborhood of the central cell $({d}_{cn})$, respectively. For the spatiotemporal weighting scheme (STW), the spatiotemporal distance was computed from the location and year of the latest central cell in a sample and the location and year of the most recently changed cell in its neighborhood $\left({d}_{cn}^{ST}\right)$. Table 1 shows the formulations of TW1, TW2, and STW. By expanding the weighting schemes to consider changes taking place within a cell’s neighborhood, the TW2 and STW schemes increased the weight of change samples with recent nearby transitions. This means that more dynamic change samples had greater influence on model parameter adjustments during the training procedure using the TW2 and STW schemes. The IDW power or exponent parameter was set to one because the spatial resolution of this research study was coarse and there were limited timesteps available, as well as to ensure weight values associated with of historical changes were non-zero. The resulting sample weights are presented in Figure 3. No manual adjustments or normalization techniques were applied to the weights to further influence sample importance. Therefore, the maximum possible weight values were constrained by Step 2, where the inverse proportion of changed and persistent cells was calculated. This led to the maximal weight value of 0.9 (Figure 3).

#### 2.6. Model Assessment

_{overall}), the largest classes (AED

_{large}), the classes deemed “medium-sized” (AED

_{medium}), and the smallest classes (AED

_{small}). The purpose was to identify where the most severe AEs stemmed from. The large class size category encompassed evergreen forests, shrublands and savannas, and barren land, comprising 91.1% of the study area. The medium class size category captured permanent snow and ice, water bodies, and deciduous forests, covering 8.7% of the study area. The smallest classes were urban and built-up lands and croplands, occupying less than 1% of the study area.

#### 2.7. Experiment Settings

_{n}, t

_{n+1}, …, t

_{n+9}, with t

_{n+10}as the training label. Each data sample comprised spatiotemporal LC data and static spatial variables with the neighborhood specifications expressed in Section 2.2. Samples for every location are provided to train the model with the sample weights described in Section 2.5 and shown in Figure 3. The four model types were trained considering each sample weight scheme (none, BW, TW1, TW2, and STW), where “none” refers to the experimental combinations or base case in which no sample weighting scheme was applied.

## 3. Results

#### 3.1. Multi-Year Change Assessment

_{STW}, CNN-GRU

_{STW}, ConvLSTM

_{TW2}, ConvLSTM

_{STW}, CNN-LSTM

_{TW1}, and ConvLSTM

_{TW1}(Figure 4). Following the FOM values obtained by these model and sample weight combinations, there was a 19.7% difference between the next model and sample weight combinations (Figure 4) and a substantial drop in FOM values observed for the base case (denoted as “none”). All sample weighting schemes facilitated improved FOM measures over the base case for the 2016 LC change forecasts, regardless of model type. The BW scheme was associated with consistently improved FOM values compared to the base case, although the top combination using BW (CNN-GRU

_{BW}) was 27.6–32.6% lower than the top six performers identified. In addition, the STW scheme was associated with higher FOM values for three of the four model types (CNN-GRU, CNN-TCN, and ConvLSTM) (Figure 4). FOM values obtained with the TW2 scheme also enabled improved performance versus the BW scheme for the same three models. The TW1 scheme worked well for CNN-LSTM and ConvLSTM, but reduced FOM values for CNN-GRU and CNN-TCN.

_{BW}and CNN-GRU

_{BW}showed a slight increase over time. In contrast to the initial trends seen in Figure 4, ConvLSTM

_{TW1}surpassed ConvLSTM

_{STW}and ConvLSTM

_{TW2}after the 2017 projection, while CNN-TCN

_{STW}, CNN-GRU

_{STW}, and CNN-LSTM

_{TW1}continued to yield the highest FOM measures (Figure 5a). The PA measures computed with respect to changed areas followed the same trends (Figure 5b), showing that the proportion of forecasted versus real-world changes was highest in forecasts obtained from the top six model and sample weight combinations. In contrast, the UA showed a trend dissimilar to those seen with FOM and PA measures (Figure 5c). The highest UA measures were obtained by CNN-LSTM

_{None}and ConvLSTM

_{None}, showing that the proportion of correctly simulated changes versus all projected change was high. The low quantities of changes projected with no sample weights boosted UA measures over time, as these measures were inflated by small amounts of projected LC change, as indicated by “hits”, “false alarms”, and “wrong changes” for both 2016 and 2020 (Figure 6a,b). Notably, ConvLSTM

_{TW2}and ConvLSTM

_{STW}exhibited higher UA measures than the other top six models. This indicates that these combinations forecasted higher amounts of correctly changed areas out of all forecasted changes and reduced incorrectly changed areas, despite not attaining the maximal FOM or amount of hits (Figure 5c and Figure 6). Overall, CNN-TCN

_{STW}yielded the highest amount of correctly changed area of all the 2020 LC change forecasts (Figure 6). The highest quantity of hits or correctly changed area for each model type was associated with TW1, TW2, and STW, except for CNN-GRU

_{TW1}. Additionally, CNN-TCN

_{STW}and ConvLSTM

_{TW1}produced the highest number of false alarms of the top six models.

#### 3.2. Multi-Year Error Analysis

_{TW1}, CNN-GRU

_{STW}, and CNN-TCN

_{STW}) provided the lowest EQ for their respective model types, showing that each projected more realistic quantities of changes. ConvLSTM

_{TW1}preserved the lowest EQ for the ConvLSTM model type, while EQ values attributed to ConvLSTM

_{TW2}and ConvLSTM

_{STW}gradually exceed those of ConvLSTM

_{BW}. Conversely, the base case attained the lowest EA measures, corresponding to the minimal quantities of false alarms and wrong changes observed (Figure 6 and Figure 7b). Of the top six models, CNN-LSTM

_{TW1}, CNN-GRU

_{STW}, CNN-TCN

_{STW}, and ConvLSTM

_{TW1}forecasted the highest amounts of changed area allocated incorrectly, while ConvLSTM

_{STW}projected the lowest EA for each step of the 5-year projection.

_{overall}, AED

_{large}, AED

_{medium}, and AED

_{small}maintained similar trends across the 5-year forecast (Figure 8). AED

_{overall}and AED

_{large}values showing the smallest or nearest allocation errors were associated with the 2016 forecast by ConvLSTM

_{TW2}(Figure 8a,b). For AED

_{overall}, a notable deviation was observed for CNN-TCN

_{BW}, where the overall allocation error severity exceeded the unweighted base case. The top six models produced overall allocation errors generally closer to the real-world areas than the unweighted base case in the projections for 2017–2020. The same trend was also noted for AED

_{large}, except for the 2017 forecast produced by ConvLSTM

_{TW1}. However, it was observed that the erroneous large class allocations were either corrected or were more agreeable with the real-world 2018 LC allocations, as the AED

_{large}value decreased for the next timestep. It should be noted that the spread of AED

_{overall}and AED

_{large}values was not substantial, indicating that overall allocation errors and allocation errors attributed to the largest LC classes were marginal with respect to the spatial resolution of the dataset. The AED

_{medium}showed that the top six models produced allocation errors nearer to the real-world classes than the unweighted base case, except for the ConvLSTM

_{STW}model (Figure 8c). While AED

_{small}values computed from the 2016 forecast were zero for ConvLSTM

_{STW}and ConvLSTM

_{TW2}, larger deviations were observed between 2018 and 2020. This implies agricultural or built-up areas were forecasted far from their real-world allocations.

#### 3.3. Visual Assessment

_{STW}and ConvLSTM

_{STW}were considered because the first exhibited the highest FOM, while the latter forecasted the smallest number of false alarms among the top six models. For the 2016 forecasts, the false alarms and misses exhibited a “salt-and-pepper” appearance (Figure 9a,c). The CNN-TCN

_{STW}exhibited a few small clusters of false alarms in the west and southwest of the study region. However, for the 2020 LC forecast, CNN-TCN

_{STW}showed more distinctive clusters of false alarms surrounded by correct changes. Areas that were persistent that were forecasted incorrectly as changed appeared to be near to or at the locations that had higher sample weights (Figure 3d and Figure 9b). The 2020 forecast from ConvLSTM

_{STW}showed smaller and fewer clusters of false alarms. It was shown previously that misses contributed the most errors (Figure 6), and missed changes were shown to be visually consistent across both 2020 projections (Figure 9b,d).

## 4. Discussion

_{STW}, CNN-GRU

_{STW}, ConvLSTM

_{TW2}, ConvLSTM

_{STW}, CNN-LSTM

_{TW1}, and ConvLSTM

_{TW1}. It was observed that the highest FOM measures associated with these six combinations were maintained for the 5-year projection. Furthermore, the top six models attained the highest PA measures. The UA measures highlighted some interesting trends among the weighting schemes, notably for ConvLSTM. The gradually increasing UA observed for ConvLSTM

_{STW}with respect to cumulative forecasted changes indicated consistent increases in correct changes versus all projected changes, while forecasting fewer false alarms or incorrect LC transitions. Overall, STW was most beneficial for CNN-GRU and CNN-TCN across all experiments, while TW1, TW2, and STW similarly benefited ConvLSTM models. The TW1 scheme benefited CNN-LSTM most for all timesteps of the 5-year projection, and the FOM values associated with ConvLSTM

_{TW1}also surpassed those of ConvLSTM

_{TW2}and ConvLSTM

_{STW}after 2017. This aligns with the observation that sample weight values and variations of TW1 and STW are more similar than those characterizing TW2 or BW schemes (Figure 3). However, the similarities did not expound the approximate 2% difference between FOM measures of CNN-LSTM and CNN-GRU with the TW1 and STW schemes (Figure 5a). This requires future investigation of model parameters, structure, and regularization techniques with respect to the weighting schemes. TW2 was associated with only one of the top six models (ConvLSTM

_{TW2}) and did not facilitate similar performance to models trained with STW for the other model types. This may be because TW2 sample weight values were more like those of the traditional inverse frequency weighting (BW) scheme observed across the study area (Figure 3a,c).

_{STW}forecasted the highest amount of correctly changed areas while forecasting the most false alarms for the 2016 projection. However, CNN-TCN

_{STW}forecasted the second most false alarms for the 2020 projection, superseded by ConvLSTM

_{TW1}projecting the most persistent area incorrectly as changed. If maximizing the agreement of changed areas for the 5-year forecast was the sole objective, CNN-TCN

_{STW}would have been regarded as the “best” model and sample weight combination. Meanwhile, ConvLSTM

_{STW}forecasted 86.3% fewer false alarms with less error due to quantity (EQ) after 5 years, which was apparent in the visual assessment (Figure 9). With respect to the AED measures, the expectation was that all sample weighting schemes would help mitigate allocation error severity overall and with respect to the largest classes, which was typically the case. The AED measures indicated that the worst allocation errors were generally associated with medium and small size classes, which makes sense because no techniques were used to address the LC class imbalance problem and since optimizing per-class change allocations was not the objective of this study. However, the AED

_{medium}indicated that the top six models generally produced less severe allocation errors than the unweighted base case. The exception to this trend was ConvLSTM

_{STW}, suggesting that one or more of the medium-sized classes were not well-allocated by this model. AED

_{small}measures also indicated ConvLSTM

_{STW}and ConvLSTM

_{TW2}forecast built-up or agricultural areas far from their real-world allocations. This outcome was expected, as this second dimension of imbalance characterizing LC datasets adds challenges for multi-class change forecasting with DL models. As such, future research studies should investigate further combinations of sample weights with class weights or the focal loss function [67] to reduce the quantity and allocation error distance with respect to non-majority LC categories.

_{STW}and ConvLSTM

_{STW}(Figure 9). The CNN-TCN

_{STW}combination appeared more sensitive to the higher sample weight values in some areas, which was more noticeable in the 2020 forecast (Figure 9b). For instance, the projected hits and false alarms were typically seen clustered around locations with larger sample weights (Figure 3d, Figure 9b). This may suggest that persistent samples still require reduced weights or undersampling strategies to manage their influence on learned model parameters. This outcome may also hinge upon the model type, since the 2020 forecast produced by ConvLSTM

_{STW}appeared less influenced by areas where the sample weight computed for the location was high. Conversely, in each resulting map, both sporadic and clustered areas of missed changes existed at similar spatial locations for areas with low weight values. However, given the appearance of more clustered hits and false alarms in the 2020 projections (Figure 9b,d), the STW scheme may be beneficial for future works seeking to manage properties like spatial variability [68].

_{STW}for the 5-year projection may have been somewhat comparable. Additionally, the FOM measure had a positive linear relationship with net observed changes [56]. For example, a land change model with 10-year temporal resolution obtained an FOM value of 9%, with less than 5% of the study area undergoing changes during that time period [28]. This corroborates the values obtained in this research study, in which CNN-TCN

_{STW}attained an FOM value of 7.8% with only 3.8% of the region undergoing changes from 2016 to 2020. Future work should consider further optimization of models and sample weighting schemes for datasets with finer spatial resolutions, expanded neighborhoods, longer LC sequences, and noisy datasets. Climatic variables are also important drivers of LC change [70] and should also be integrated to further enhance model capacity to forecast changes alongside the sample weighting schemes. Additional adjustments to computed sample weights may also be beneficial, as previous work identified that low weights were negligible in their effect on model training procedures [20]. Lastly, combinations of data augmentation [16] and the removal or undersampling of persistent samples [14] may be beneficial alongside the proposed sample weighting schemes. Nevertheless, the risk of removing potentially important LC data samples remains an open problem.

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Shi, W.; Zhang, M.; Zhang, R.; Chen, S.; Zhan, Z. Change detection based on artificial intelligence: State-of-the-art and challenges. Remote Sens.
**2020**, 12, 1688. [Google Scholar] [CrossRef] - Reichstein, M.; Camps-Valls, G.; Stevens, B.; Jung, M.; Denzler, J.; Carvalhais, N. Prabhat Deep learning and process understanding for data-driven Earth system science. Nature
**2019**, 566, 195–204. [Google Scholar] [CrossRef] [PubMed] - Aburas, M.M.; Ahamad, M.S.S.; Omar, N.Q. Spatio-temporal simulation and prediction of land-use change using conventional and machine learning models: A review. Environ. Monit. Assess.
**2019**, 191, 205. [Google Scholar] [CrossRef] [PubMed] - Wang, J.; Bretz, M.; Dewan, M.A.A.; Delavar, M.A. Machine learning in modelling land-use and land cover-change (LULCC): Current status, challenges and prospects. Sci. Total Environ.
**2022**, 822, 153559. [Google Scholar] [CrossRef] [PubMed] - Rußwurm, M.; Körner, M. Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders. ISPRS Int. J. Geo-Inf.
**2018**, 7, 129. [Google Scholar] [CrossRef] [Green Version] - Lee, C.; Lee, J.; Park, S. Forecasting the urbanization dynamics in the Seoul metropolitan area using a long short-term memory–based model. Environ. Plan. B Urban Anal. City Sci.
**2022**, 59. in press. [Google Scholar] [CrossRef] - Xiao, B.; Liu, J.; Jiao, J.; Li, Y.; Liu, X.; Zhu, W. Modeling dynamic land use changes in the eastern portion of the hexi corridor, China by cnn-gru hybrid model. GIScience Remote Sens.
**2022**, 59, 501–519. [Google Scholar] [CrossRef] - Kubat, M.; Matwin, S. Addressing the curse of imbalanced data sets: One-sided sampling. In Proceedings of the Fourteenth International Conference on Machine Learning, Nashville, TN, USA, 8–12 July 1997; pp. 179–186. [Google Scholar]
- Krawczyk, B. Learning from imbalanced data: Open challenges and future directions. Prog. Artif. Intell.
**2016**, 5, 221–232. [Google Scholar] [CrossRef] [Green Version] - Buda, M.; Maki, A.; Mazurowski, M.A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw.
**2018**, 106, 249–259. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Pontius, R.G.; Shusas, E.; McEachern, M. Detecting important categorical land changes while accounting for persistence. Agric. Ecosyst. Environ.
**2004**, 101, 251–268. [Google Scholar] [CrossRef] - Karpatne, A.; Jiang, Z.; Vatsavai, R.R.; Shekhar, S.; Kumar, V. Monitoring land-cover changes: A machine-learning perspective. IEEE Geosci. Remote Sens. Mag.
**2016**, 4, 8–21. [Google Scholar] [CrossRef] - Samardžić-Petrović, M.; Kovačević, M.; Bajat, B.; Dragićević, S. Machine Learning Techniques for Modelling Short Term Land-Use Change. ISPRS Int. J. Geo-Inf.
**2017**, 6, 387. [Google Scholar] [CrossRef] [Green Version] - Karimi, F.; Sultana, S.; Babakan, A.S.; Suthaharan, S. Urban expansion modeling using an enhanced decision tree algorithm. Geoinformatica
**2021**, 25, 715–731. [Google Scholar] [CrossRef] - Ahmadlou, M.; Karimi, M.; Pontius, R.G. A new framework to deal with the class imbalance problem in urban gain modeling based on clustering and ensemble models. Geocarto Int.
**2022**, 37, 5669–5692. [Google Scholar] [CrossRef] - Douzas, G.; Bacao, F.; Fonseca, J.; Khudinyan, M. Imbalanced learning in land cover classification: Improving minority classes’ prediction accuracy using the geometric SMOTE algorithm. Remote Sens.
**2019**, 11, 3040. [Google Scholar] [CrossRef] [Green Version] - Yu, X.; Wu, X.; Luo, C.; Ren, P. Deep learning in remote sensing scene classification: A data augmentation enhanced convolutional neural network framework. GIScience Remote Sens.
**2017**, 54, 741–758. [Google Scholar] [CrossRef] [Green Version] - Kamel, A.; Issam, B. Data Augmentation for Land Cover Classification Using Generative Adversarial Networks. Int. Geosci. Remote Sens. Symp.
**2021**, 2021, 2309–2312. [Google Scholar] [CrossRef] - Lu, J.; Ren, K.; Li, X.; Zhao, Y.; Xu, Z.; Ren, X. From reanalysis to satellite observations: Gap-filling with imbalanced learning. Geoinformatica
**2022**, 26, 397–428. [Google Scholar] [CrossRef] - Ren, X.; Mi, Z.; Georgopoulos, P.G. Comparison of Machine Learning and Land Use Regression for fine scale spatiotemporal estimation of ambient air pollution: Modeling ozone concentrations across the contiguous United States. Environ. Int.
**2020**, 142, 105827. [Google Scholar] [CrossRef] [PubMed] - Lian, D.; Wu, Y.; Ge, Y.; Xie, X.; Chen, E. Geography-Aware Sequential Location Recommendation. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Association for Computing Machinery, San Diego, CA, USA, 23–27 August 2020; pp. 2009–2019. [Google Scholar]
- Sun, P.; Lu, Y.; Zhai, J. Mapping land cover using a developed U-Net model with weighted cross entropy. Geocarto Int.
**2021**, in press. [Google Scholar] [CrossRef] - Fotheringham, A.S.; Crespo, R.; Yao, J. Geographical and Temporal Weighted Regression (GTWR). Geogr. Anal.
**2015**, 47, 431–452. [Google Scholar] [CrossRef] [Green Version] - Li, D.; Wen, G.; Kuai, Y.; Wang, L. Spatio-temporally weighted multiple instance learning for visual tracking. Optik
**2018**, 171, 904–917. [Google Scholar] [CrossRef] - Statistics Canada “Population and Dwelling Counts: Canada and Census Subdivisions”. Available online: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810000201 (accessed on 1 June 2022).
- Sulla-Menashe, D.; Friedl, M. The Terra and Aqua Combined Moderate Resolution Imaging Spectroradiometer (MODIS) Land Cover Type (MCD12Q1) Version 6 Data Product. Available online: https://lpdaac.usgs.gov/dataset_discovery/modis/modis_products_table/mcd12q1_v006 (accessed on 30 January 2022).
- Wang, M.; Sun, X.; Fan, Z.; Yue, T. Investigation of Future Land Use Change and Implications for Cropland Quality: The Case of China. Sustainability
**2019**, 11, 3327. [Google Scholar] [CrossRef] [Green Version] - Singh, V.G.; Singh, S.K.; Kumar, N.; Singh, R.P. Simulation of land use/land cover change at a basin scale using satellite data and markov chain model. Geocarto Int.
**2022**, in press. [Google Scholar] [CrossRef] - Van Berkel, D.; Shashidharan, A.; Mordecai, R.S.; Vatsavai, R.; Petrasova, A.; Petras, V.; Mitasova, H.; Vogler, J.B.; Meentemeyer, R.K. Projecting urbanization and landscape change at large scale using the FUTURES model. Land
**2019**, 8, 144. [Google Scholar] [CrossRef] [Green Version] - Stobbe, T.E.; Eagle, A.J.; Cotteleer, G.; van Kooten, G.C. Farmland Preservation Verdicts-Rezoning Agricultural Land in British Columbia. Can. J. Agric. Econ.
**2011**, 59, 555–572. [Google Scholar] [CrossRef] - NASA/METI/AIST/Japan Spacesystems and U.S./Japan ASTER Science Team. “ASTER Global Digital Elevation Model V003”. Available online: https://lpdaac.usgs.gov/products/astgtmv003/ (accessed on 10 June 2022).
- Statistics Canada. “2016 Census-Boundary Files”. Available online: https://www12.statcan.gc.ca/census-recensement/2011/geo/bound-limit/bound-limit-2016-eng.cfm (accessed on 10 May 2022).
- Statistics Canada. “2016 Census Road Network File”. Available online: https://open.canada.ca/data/en/dataset/57d5ffae-3048-4a19-9b4c-eab12f6322c5 (accessed on 29 July 2022).
- van Vliet, J.; Naus, N.; van Lammeren, R.J.A.; Bregt, A.K.; Hurkens, J.; van Delden, H. Measuring the neighbourhood effect to calibrate land use models. Comput. Environ. Urban Syst.
**2013**, 41, 55–64. [Google Scholar] [CrossRef] - Roodposhti, M.S.; Hewitt, R.J.; Bryan, B.A. Towards automatic calibration of neighbourhood influence in cellular automata land-use models. Comput. Environ. Urban Syst.
**2020**, 79, 101416. [Google Scholar] [CrossRef] - Masolele, R.N.; De Sy, V.; Herold, M.; Marcos Gonzalez, D.; Verbesselt, J.; Gieseke, F.; Mullissa, A.G.; Martius, C. Spatial and temporal deep learning methods for deriving land-use following deforestation: A pan-tropical case study using Landsat time series. Remote Sens. Environ.
**2021**, 264, 112600. [Google Scholar] [CrossRef] - Gray, P.C.; Chamorro, D.F.; Ridge, J.T.; Kerner, H.R.; Ury, E.A.; Johnston, D.W. Temporally Generalizable Land Cover Classification: A Recurrent Convolutional Neural Network Unveils Major Coastal Change through Time. Remote Sens.
**2021**, 13, 3953. [Google Scholar] [CrossRef] - van Duynhoven, A.; Dragićević, S. Assessing the Impact of Neighborhood Size on Temporal Convolutional Networks for Modeling Land Cover Change. Remote Sens.
**2022**, 14, 4957. [Google Scholar] [CrossRef] - Verburg, P.H.; de Nijs, T.C.M.; van Eck, J.R.; Visser, H.; de Jong, K. A method to analyse neighbourhood characteristics of land use patterns. Comput. Environ. Urban Syst.
**2004**, 28, 667–690. [Google Scholar] [CrossRef] - Hochreiter, S.; Schmidhuber, J. Long Short-Term Memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] - Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar, 26–28 October 2014; Association for Computational Linguistics: Stroudsburg, PA, USA, 2014; pp. 1724–1734. [Google Scholar]
- Bai, S.; Kolter, J.Z.; Koltun, V. An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling. arXiv
**2018**, arXiv:1803.01271. [Google Scholar] - Yin, G.; Huang, Z.; Bao, Y.; Wang, H.; Li, L.; Ma, X.; Zhang, Y. ConvGCN-RF: A hybrid learning model for commuting flow prediction considering geographical semantics and neighborhood effects. Geoinformatica
**2022**, in press. [Google Scholar] [CrossRef] - Yan, J.; Chen, X.; Chen, Y.; Liang, D. Multistep Prediction of Land Cover from Dense Time Series Remote Sensing Images with Temporal Convolutional Networks. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2020**, 13, 5149–5161. [Google Scholar] [CrossRef] - Shi, X.; Chen, Z.; Wang, H.; Yeung, D.Y.; Wong, W.K.; Woo, W.C. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst.
**2015**, 2015, 802–810. [Google Scholar] - Chen, R.; Wang, X.; Zhang, W.; Zhu, X.; Li, A.; Yang, C. A hybrid CNN-LSTM model for typhoon formation forecasting. Geoinformatica
**2019**, 23, 375–396. [Google Scholar] [CrossRef] - Huang, C.-J.; Kuo, P.-H. A Deep CNN-LSTM Model for Particulate Matter (PM2.5) Forecasting in Smart Cities. Sensors
**2018**, 18, 2220. [Google Scholar] [CrossRef] [Green Version] - Sefrin, O.; Riese, F.M.; Keller, S. Deep learning for land cover change detection. Remote Sens.
**2021**, 13, 78. [Google Scholar] [CrossRef] - Pham, V.; Bluche, T.; Kermorvant, C.; Louradour, J. Dropout Improves Recurrent Neural Networks for Handwriting Recognition. In Proceedings of the 2014 14th International Conference on Frontiers in Handwriting Recognition, Crete, Greece, 1–4 September 2014; pp. 285–290. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] [PubMed] - Gordon-Rodriguez, E.; Loaiza-Ganem, G.; Pleiss, G.; Cunningham, J.P. Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning. In Proceedings of the Proceedings on “I Can’t Believe It’s Not Better!” at NeurIPS Workshops, online, 12 December 2020; Volume 137, pp. 1–10. [Google Scholar]
- Maretto, R.V.; Fonseca, L.M.G.; Jacobs, N.; Körting, T.S.; Bendini, H.N.; Parente, L.L. Spatio-Temporal Deep Learning Approach to Map Deforestation in Amazon Rainforest. IEEE Geosci. Remote Sens. Lett.
**2021**, 18, 771–775. [Google Scholar] [CrossRef] - Zimmerman, D.; Pavlik, C.; Ruggles, A.; Armstrong, M.P. An experimental comparison of ordinary and universal kriging and inverse distance weighting. Math. Geol.
**1999**, 31, 375–390. [Google Scholar] [CrossRef] - Kang, M.; Liu, Y.; Wang, M.; Li, L.; Weng, M. A random forest classifier with cost-sensitive learning to extract urban landmarks from an imbalanced dataset. Int. J. Geogr. Inf. Sci.
**2022**, 36, 496–513. [Google Scholar] [CrossRef] - Tong, X.; Feng, Y. A review of assessment methods for cellular automata models of land-use change and urban growth. Int. J. Geogr. Inf. Sci.
**2020**, 34, 866–898. [Google Scholar] [CrossRef] - Pontius, R.G.; Boersma, W.; Castella, J.-C.C.; Clarke, K.; de Nijs, T.; Dietzel, C.; Duan, Z.; Fotsing, E.; Goldstein, N.; Kok, K.; et al. Comparing the input, output, and validation maps for several models of land change. Ann. Reg. Sci.
**2008**, 42, 11–37. [Google Scholar] [CrossRef] [Green Version] - Camacho Olmedo, M.T.; Pontius, R.G.; Paegelow, M.; Mas, J.F. Comparison of simulation models in terms of quantity and allocation of land change. Environ. Model. Softw.
**2015**, 69, 214–221. [Google Scholar] [CrossRef] [Green Version] - Paegelow, M.; Camacho Olmedo, M.T.; Mas, J.; Houet, T. Benchmarking of LUCC modelling tools by various validation techniques and error analysis. Cybergeo Eur. J. Geogr.
**2014**, 701. [Google Scholar] [CrossRef] - van Rossum, G. Python Language Reference; Python Software Foundation: Amsterdam, The Netherlands, 2009; ISBN 9780954161781. [Google Scholar]
- Chollet, F. Keras: The Python Deep Learning Library. Available online: https://keras.io/ (accessed on 26 May 2022).
- Abadi, M.; Agarwal, A.; Barham, P.; Brevdo, E.; Chen, Z.; Citro, C.; Corrado, G.S.; Davis, A.; Dean, J.; Devin, M.; et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation, Savannah, GA, USA, 2–4 November 2016; pp. 265–283. [Google Scholar] [CrossRef]
- Remy, P. Temporal Convolutional Networks for Keras. Available online: https://github.com/philipperemy/keras-tcn (accessed on 1 May 2022).
- Naushad, R.; Kaur, T.; Ghaderpour, E. Deep transfer learning for land use and land cover classification: A comparative study. Sensors
**2021**, 21, 8083. [Google Scholar] [CrossRef] - Xiao, C.; Chen, N.; Hu, C.; Wang, K.; Xu, Z.; Cai, Y.; Xu, L.; Chen, Z.; Gong, J. A spatiotemporal deep learning model for sea surface temperature field prediction using time-series satellite data. Environ. Model. Softw.
**2019**, 120, 104502. [Google Scholar] [CrossRef] - van Duynhoven, A.; Dragićević, S. Exploring the sensitivity of recurrent neural network models for forecasting land cover change. Land
**2021**, 10, 282. [Google Scholar] [CrossRef] - Ma, J.; Ding, Y.; Gan, V.J.L.; Lin, C.; Wan, Z. Spatiotemporal Prediction of PM2.5 Concentrations at Different Time Granularities Using IDW-BLSTM. IEEE Access
**2019**, 7, 107897–107907. [Google Scholar] [CrossRef] - Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollar, P. Focal Loss for Dense Object Detection. IEEE Trans. Pattern Anal. Mach. Intell.
**2020**, 42, 318–327. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Gupta, J.; Molnar, C.; Xie, Y.; Knight, J.; Shekhar, S. Spatial Variability Aware Deep Neural Networks (SVANN): A General Approach. ACM Trans. Intell. Syst. Technol.
**2021**, 12, 1–21. [Google Scholar] [CrossRef] - Costa, H.; Almeida, D.; Vala, F.; Marcelino, F.; Caetano, M. Land cover mapping from remotely sensed and auxiliary data for harmonized official statistics. ISPRS Int. J. Geo-Inf.
**2018**, 7, 157. [Google Scholar] [CrossRef] [Green Version] - da Silva, M.V.; Pandorfi, H.; de Oliveira-Júnior, J.F.; da Silva, J.L.B.; de Almeida, G.L.P.; de Assunção Montenegro, A.A.; Mesquita, M.; Ferreira, M.B.; Santana, T.C.; Marinho, G.T.B.; et al. Remote sensing techniques via Google Earth Engine for land degradation assessment in the Brazilian semiarid region, Brazil. J. South Am. Earth Sci.
**2022**, 120, 104061. [Google Scholar] [CrossRef]

**Figure 1.**Study area of the Columbia-Shuswap Regional District, British Columba, with (

**a**) land cover for 2001 and (

**b**) annual net land cover changes for the region from 2001 to 2020. Data are displayed with the NAD 1983 BC Environment Albers projected coordinate system.

**Figure 2.**Overview of the basic branched model structure used to accommodate the 9 × 9 spatiotemporal land cover sample sequences and the auxiliary spatial variables. The “spatial branch” is implemented with CNN layers. The “spatiotemporal branch” implementation varies according to the model type, characterized by CNN-LSTM, CNN-GRU, CNN-TCN, or ConvLSTM layers. Location x, y in the land cover sample is denoted in red, indicating the central cell of the neighborhood.

**Figure 3.**Maps depicting the variation of sample weights under each weighting scheme. The sample weights assigned to every location represent (

**a**) binary weight (BW), (

**b**) cell-change temporal weight (TW1), (

**c**) neighborhood-change temporal weight (TW2), and (

**d**) spatiotemporal weight (STW). Color swatches adjacent to each sample weight scheme correspond to those used in figures in subsequent sections.

**Figure 4.**Figure of merit (FOM) values obtained for each model type and sample weight combination for the 2016 LC forecast. The model and sample weight combinations with the top six FOM values are denoted with the prefix **.

**Figure 5.**Values obtained for each model type and sample weight combination with respect to cumulative changes from 2016 to 2020 with measures: (

**a**) figure of merit (FOM), (

**b**) producer’s accuracy (PA), and (

**c**) user’s accuracy (UA). The model and sample weight combinations with the top six FOM values are indicated by bold lines.

**Figure 6.**Components of agreement and disagreement as a percentage of the study area for (

**a**) the 2016 forecast and (

**b**) the 2020 forecast. The model and sample weight combinations with the top six FOM values (Figure 4) are denoted with the prefix **.

**Figure 7.**Cumulative errors from 2016 to 2020 for (

**a**) error due to quantity (EQ) and (

**b**) error due to allocation (EA). The model and sample weight combinations with the top six FOM values for the 2016 forecast are indicated by bold lines.

**Figure 8.**Average allocation error distance from real-world allocations in 2016–2020 considering (

**a**) AED

_{overall}, (

**b**) AED

_{large}, (

**c**) AED

_{medium}, and (

**d**) AED

_{small}. The model and sample weight combinations with the top six FOM values are indicated by bold lines.

**Figure 9.**Maps representing the locations of obtained values for components of agreement and disagreement of forecasted LC with CNN-TCN

_{STW}for (

**a**) 2016 and (

**b**) 2020 and with ConvLSTM

_{STW}for (

**c**) 2016 and (

**d**) 2020.

Sample Weight Scheme for Changed Locations | Formula | Description |
---|---|---|

Binary weight (BW) | ${w}_{{c}_{i}}={b}_{{c}_{i}}=P/\left(P+C\right)$ | The inverse proportion of changed versus persistent samples |

Cell-change temporal weight (TW1) | ${w}_{{c}_{i}}={b}_{{c}_{i}}\ast \frac{1}{{d}_{cc}}={b}_{{c}_{i}}\ast \frac{1}{{t}_{n}-{t}_{cc}}$ | Temporal distance (${d}_{cc}$) between most recent year (${t}_{n}$) and the year of the most recent change event of the central cell (${t}_{cc}$) |

Neighborhood-change temporal weight (TW2) | ${w}_{{c}_{i}}={b}_{{c}_{i}}\ast \frac{1}{{d}_{cn}}={b}_{{c}_{i}}\ast \frac{1}{{t}_{n}-{t}_{cn}}$ | Temporal distance (${d}_{cn}$) from the most recent year (${t}_{n}$) and the year of change event occurring in the neighborhood of the central cell (${t}_{cn}$) |

Spatiotemporal weight (STW) | ${w}_{{c}_{i}}={b}_{{c}_{i}}\ast \frac{1}{{d}_{cn}^{ST}}$ $={b}_{{c}_{i}}\ast \frac{1}{\sqrt{{\left(x-{x}_{cn}\right)}^{2}+{\left(y-{y}_{cn}\right)}^{2}+{\left({t}_{n}-{t}_{cn}\right)}^{2}}}$ | Spatiotemporal distance (${d}_{cn}^{ST}$) from the central cell ($x,y,{t}_{n}$) to the nearest changed cell in its neighborhood (${x}_{cn},{y}_{cn},{t}_{cn}$) |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

van Duynhoven, A.; Dragićević, S.
Mitigating Imbalance of Land Cover Change Data for Deep Learning Models with Temporal and Spatiotemporal Sample Weighting Schemes. *ISPRS Int. J. Geo-Inf.* **2022**, *11*, 587.
https://doi.org/10.3390/ijgi11120587

**AMA Style**

van Duynhoven A, Dragićević S.
Mitigating Imbalance of Land Cover Change Data for Deep Learning Models with Temporal and Spatiotemporal Sample Weighting Schemes. *ISPRS International Journal of Geo-Information*. 2022; 11(12):587.
https://doi.org/10.3390/ijgi11120587

**Chicago/Turabian Style**

van Duynhoven, Alysha, and Suzana Dragićević.
2022. "Mitigating Imbalance of Land Cover Change Data for Deep Learning Models with Temporal and Spatiotemporal Sample Weighting Schemes" *ISPRS International Journal of Geo-Information* 11, no. 12: 587.
https://doi.org/10.3390/ijgi11120587