# Deep Learning-Based Phenological Event Modeling for Classification of Crops

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Materials and Methods

#### 2.1. Datasets

#### 2.2. Proposed Approach

#### 2.2.1. Deep Capsule Network Stream

^{th}capsule should be coupled to the j

^{th}capsule.

#### 2.2.2. Conditional Variational Encoding

#### 2.2.3. Loss Functions and Regularizations

#### 2.2.4. Transparency and Interpretability

_{c}, is found by optimizing:

#### 2.3. Implementation of the Proposed VCapsNet

## 3. Results

#### 3.1. Ablation Analysis of VCapsNet

#### 3.2. Comparison of VCapsNet with the Commonly-Used DL Based Approaches

#### 3.3. Comparison of VCapsNet with the Commonly-Used Denoising Approaches

## 4. Discussion

#### 4.1. Modeling Phenological Events

#### 4.2. Interpretability Based Comparison of the Benchmark DL Models

## 5. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Rodriguez-Galiano, V.; Dash, J.; Atkinson, P. Characterising the Land Surface Phenology of Europe Using Decadal MERIS Data. Remote Sens.
**2015**, 7, 9390–9409. [Google Scholar] [CrossRef] [Green Version] - Zhao, K.; Wulder, M.A.; Hu, T.; Bright, R.; Wu, Q.; Qin, H.; Li, Y.; Toman, E.; Mallick, B.; Zhang, X.; et al. Detecting change-point, trend, and seasonality in satellite time series data to track abrupt changes and nonlinear dynamics: A Bayesian ensemble algorithm. Remote Sens. Environ.
**2019**, 232, 111181. [Google Scholar] [CrossRef] - Zeng, L.; Wardlow, B.D.; Xiang, D.; Hu, S.; Li, D. A review of vegetation phenological metrics extraction using time-series, multispectral satellite data. Remote Sens. Environ.
**2020**, 237, 111511. [Google Scholar] [CrossRef] - Chen, J.; Chen, J.; Liu, H.; Peng, S. Detection of Cropland Change Using Multi-Harmonic Based Phenological Trajectory Similarity. Remote Sens.
**2018**, 10, 1020. [Google Scholar] [CrossRef] [Green Version] - Azar, R.; Villa, P.; Stroppiana, D.; Crema, A.; Boschetti, M.; Brivio, P.A. Assessing in-season crop classification performance using satellite data: A test case in Northern Italy. Eur. J. Remote Sens.
**2016**, 49, 361–380. [Google Scholar] [CrossRef] [Green Version] - Peña-Barragán, J.M.; Ngugi, M.K.; Plant, R.E.; Six, J. Object-based crop identification using multiple vegetation indices, textural features and crop phenology. Remote Sens. Environ.
**2011**, 115, 1301–1316. [Google Scholar] [CrossRef] - Zhong, L.; Hu, L.; Yu, L.; Gong, P.; Biging, G.S. Automated mapping of soybean and corn using phenology. ISPRS J. Photogramm. Remote Sens.
**2016**, 119, 151–164. [Google Scholar] [CrossRef] [Green Version] - Zhang, J.; Feng, L.; Yao, F. Improved maize cultivated area estimation over a large scale combining MODIS-EVI time series data and crop phenological information. ISPRS J. Photogramm. Remote Sens.
**2014**, 94, 102–113. [Google Scholar] [CrossRef] - Al-Shammari, D.; Fuentes, I.; Whelan, B.M.; Filippi, P.; Bishop, T.F.A. Mapping of Cotton Fields Within-Season Using Phenology-Based Metrics Derived from a Time Series of Landsat Imagery. Remote Sens.
**2020**, 12, 3038. [Google Scholar] [CrossRef] - Che, M.; Chen, B.; Zhang, H.; Fang, S.; Xu, G.; Lin, X.; Wang, Y. A New Equation for Deriving Vegetation Phenophase from Time Series of Leaf Area Index (LAI) Data. Remote Sens.
**2014**, 6, 5650–5670. [Google Scholar] [CrossRef] [Green Version] - Dong, J.; Xiao, X.; Kou, W.; Qin, Y.; Zhang, G.; Li, L.; Jin, C.; Zhou, Y.; Wang, J.; Biradar, C.; et al. Tracking the dynamics of paddy rice planting area in 1986–2010 through time series Landsat images and phenology-based algorithms. Remote Sens. Environ.
**2015**, 160, 99–113. [Google Scholar] [CrossRef] - Zhang, G.; Xiao, X.; Dong, J.; Kou, W.; Jin, C.; Qin, Y.; Zhou, Y.; Wang, J.; Menarguez, M.A.; Biradar, C. Mapping paddy rice planting areas through time series analysis of MODIS land surface temperature and vegetation index data. ISPRS J. Photogramm. Remote Sens.
**2015**, 106, 157–171. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Dong, J.; Xiao, X.; Wagle, P.; Zhang, G.; Zhou, Y.; Jin, C.; Torn, M.S.; Meyers, T.P.; Suyker, A.E.; Wang, J.; et al. Comparison of four EVI-based models for estimating gross primary production of maize and soybean croplands and tallgrass prairie under severe drought. Remote Sens. Environ.
**2015**, 162, 154–168. [Google Scholar] [CrossRef] [Green Version] - Maselli, F.; Battista, P.; Chiesi, M.; Rapi, B.; Angeli, L.; Fibbi, L.; Magno, R.; Gozzini, B. Use of Sentinel-2 MSI data to monitor crop irrigation in Mediterranean areas. Int. J. Appl. Earth Obs. Geoinf.
**2020**, 93, 102216. [Google Scholar] [CrossRef] - Pan, Y.; Li, L.; Zhang, J.; Liang, S.; Zhu, X.; Sulla-Menashe, D. Winter wheat area estimation from MODIS-EVI time series data using the Crop Proportion Phenology Index. Remote Sens. Environ.
**2012**, 119, 232–242. [Google Scholar] [CrossRef] - Kontgis, C.; Schneider, A.; Ozdogan, M. Mapping rice paddy extent and intensification in the Vietnamese Mekong River Delta with dense time stacks of Landsat data. Remote Sens. Environ.
**2015**, 169, 255–269. [Google Scholar] [CrossRef] - Foerster, S.; Kaden, K.; Foerster, M.; Itzerott, S. Crop type mapping using spectral-temporal profiles and phenological information. Comput. Electron. Agric.
**2012**, 89, 30–40. [Google Scholar] [CrossRef] [Green Version] - Blaes, X.; Vanhalle, L.; Defourny, P. Efficiency of crop identification based on optical and SAR image time series. Remote Sens. Environ.
**2005**, 96, 352–365. [Google Scholar] [CrossRef] - Mingwei, Z.; Qingbo, Z.; Zhongxin, C.; Jia, L.; Yong, Z.; Chongfa, C. Crop discrimination in Northern China with double cropping systems using Fourier analysis of time-series MODIS data. Int. J. Appl. Earth Obs. Geoinf.
**2008**, 10, 476–485. [Google Scholar] [CrossRef] - Jakubauskas, M.E.; Legates, D.R.; Kastens, J.H. Crop identification using harmonic analysis of time-series AVHRR NDVI data. Comput. Electron. Agric.
**2003**, 37, 127–139. [Google Scholar] [CrossRef] - Immerzeel, W.W.; Quiroz, R.A.; De Jong, S.M. Understanding precipitation patterns and land use interaction in Tibet using harmonic analysis of SPOT VGT-S10 NDVI time series. Int. J. Remote Sens.
**2005**, 26, 2281–2296. [Google Scholar] [CrossRef] - Rudorff, B.F.T.; Adami, M.; De Aguiar, D.A.; Gusso, A.; Da Silva, W.F.; De Freitas, R.M. Temporal series of EVI/MODIS to identify land converted to sugarcane. In Proceedings of the International Geoscience and Remote Sensing Symposium (IGARSS), Cape Town, South Africa, 12–17 July 2009; Volume 4. [Google Scholar]
- Huang, X.; Liu, J.; Zhu, W.; Atzberger, C.; Liu, Q. The Optimal Threshold and Vegetation Index Time Series for Retrieving Crop Phenology Based on a Modified Dynamic Threshold Method. Remote Sens.
**2019**, 11, 2725. [Google Scholar] [CrossRef] [Green Version] - You, X.; Meng, J.; Zhang, M.; Dong, T. Remote sensing based detection of crop phenology for agricultural zones in China using a new threshold method. Remote Sens.
**2013**, 5, 3190–3211. [Google Scholar] [CrossRef] [Green Version] - Beck, P.S.A.; Atzberger, C.; Høgda, K.A.; Johansen, B.; Skidmore, A.K. Improved monitoring of vegetation dynamics at very high latitudes: A new method using MODIS NDVI. Remote Sens. Environ.
**2006**, 100, 321–334. [Google Scholar] [CrossRef] - Bórnez, K.; Descals, A.; Verger, A.; Peñuelas, J. Land surface phenology from VEGETATION and PROBA-V data. Assessment over deciduous forests. Int. J. Appl. Earth Obs. Geoinf.
**2020**, 84, 101974. [Google Scholar] [CrossRef] - Luo, Q.; Song, J.; Yang, L.; Wang, J. Improved Spring Vegetation Phenology Calculation Method Using a Coupled Model and Anomalous Point Detection. Remote Sens.
**2019**, 11, 1432. [Google Scholar] [CrossRef] [Green Version] - An, S.; Zhang, X.; Chen, X.; Yan, D.; Henebry, G. An Exploration of Terrain Effects on Land Surface Phenology across the Qinghai–Tibet Plateau Using Landsat ETM+ and OLI Data. Remote Sens.
**2018**, 10, 1069. [Google Scholar] [CrossRef] [Green Version] - Sakamoto, T.; Yokozawa, M.; Toritani, H.; Shibayama, M.; Ishitsuka, N.; Ohno, H. A crop phenology detection method using time-series MODIS data. Remote Sens. Environ.
**2005**, 96, 366–374. [Google Scholar] [CrossRef] - Xiang, M.; Yu, Q.; Wu, W. From multiple cropping index to multiple cropping frequency: Observing cropland use intensity at a finer scale. Ecol. Indic.
**2019**, 101, 892–903. [Google Scholar] [CrossRef] - Lim, C.H.; An, J.H.; Jung, S.H.; Nam, G.B.; Cho, Y.C.; Kim, N.S.; Lee, C.S. Ecological consideration for several methodologies to diagnose vegetation phenology. Ecol. Res.
**2018**, 33, 363–377. [Google Scholar] [CrossRef] - Wang, C.; Li, J.; Liu, Q.; Zhong, B.; Wu, S.; Xia, C. Analysis of differences in phenology extracted from the enhanced vegetation index and the leaf area index. Sensors
**2017**, 17, 1982. [Google Scholar] [CrossRef] [Green Version] - Yuan, Q.; Shen, H.; Li, T.; Li, Z.; Li, S.; Jiang, Y.; Xu, H.; Tan, W.; Yang, Q.; Wang, J.; et al. Deep learning in environmental remote sensing: Achievements and challenges. Remote Sens. Environ.
**2020**, 241, 111716. [Google Scholar] [CrossRef] - Wang, L.; Shi, C.; Diao, C.; Ji, W.; Yin, D. A survey of methods incorporating spatial information in image classification and spectral unmixing. Int. J. Remote Sens.
**2016**, 37, 3870–3910. [Google Scholar] [CrossRef] - Kingma, D.P.; Welling, M. An Introduction to Variational Autoencoders. Found. Trends Mach. Learn.
**2019**, 12, 307–392. [Google Scholar] [CrossRef] - Burgess, C.P.; Higgins, I.; Pal, A.; Matthey, L.; Watters, N.; Desjardins, G.; Lerchner, A. Understanding disentangling in β-VAE. arXiv
**2018**, arXiv:1804.03599. [Google Scholar] - Wang, X.; Tan, K.; Du, Q.; Chen, Y.; Du, P. CVA2E: A Conditional Variational Autoencoder with an Adversarial Training Process for Hyperspectral Imagery Classification. IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 5676–5692. [Google Scholar] [CrossRef] - Mao, J.; Wang, X.; Li, H. Interpolated Convolutional Networks for 3D Point Cloud Understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 1578–1587. [Google Scholar]
- Li, J.; Shen, Y.; Yang, C. An adversarial generative network for crop classification from remote sensing timeseries images. Remote Sens.
**2021**, 13, 65. [Google Scholar] [CrossRef] - Jiang, T.; Li, Y.; Xie, W.; Du, Q. Discriminative reconstruction constrained generative adversarial network for hyperspectral anomaly detection. IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 4666–4679. [Google Scholar] [CrossRef] - Pv, A.; Buddhiraju, K.M.; Porwal, A. Capsulenet-Based Spatial-Spectral Classifier for Hyperspectral Images. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens.
**2019**, 12, 1849–1865. [Google Scholar] [CrossRef] - Iwana, B.K.; Frinken, V.; Uchida, S. DTW-NN: A novel neural network for time series recognition using dynamic alignment between inputs and weights. Knowl. Based Syst.
**2020**, 188, 104971. [Google Scholar] [CrossRef] - Sabour, S.; Frosst, N.; Hinton, G.E. Dynamic Routing Between Capsules. Adv. Neural Inf. Process. Syst.
**2017**, 2017r, 3857–3867. [Google Scholar] - Rajasegaran, J.; Jayasundara, V.; Jayasekara, S.; Jayasekara, H.; Seneviratne, S.; Rodrigo, R. Deepcaps: Going deeper with capsule networks. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019; Volume 2019, pp. 10717–10725. [Google Scholar]
- Shi, Y.; Han, L.; Huang, W.; Chang, S.; Dong, Y.; Dancey, D.; Han, L. A Biologically Interpretable Two-Stage Deep Neural Network (BIT-DNN) for Vegetation Recognition from Hyperspectral Imagery. IEEE Trans. Geosci. Remote Sens.
**2021**. [Google Scholar] [CrossRef] - Zhu, K.; Chen, Y.; Ghamisi, P.; Jia, X.; Benediktsson, J.A. Deep convolutional capsule network for hyperspectral image spectral and spectral-spatial classification. Remote Sens.
**2019**, 11, 223. [Google Scholar] [CrossRef] [Green Version] - Zhou, S.; Zhou, Y.; Liu, B. Using Siamese capsule networks for remote sensing scene classification. Remote Sens. Lett.
**2020**, 11, 757–766. [Google Scholar] [CrossRef] - Paoletti, M.E.; Haut, J.M.; Fernandez-Beltran, R.; Plaza, J.; Plaza, A.; Li, J.; Pla, F. Capsule Networks for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens.
**2019**, 57, 2145–2160. [Google Scholar] [CrossRef] - Zhang, H.; Meng, L.; Wei, X.; Tang, X.; Tang, X.; Wang, X.; Jin, B.; Yao, W. 1D-Convolutional Capsule Network for Hyperspectral Image Classification. arXiv
**2019**, arXiv:1903.09834. [Google Scholar] - Jayasekara, H.; Jayasundara, V.; Rajasegaran, J.; Jayasekara, S.; Seneviratne, S.; Rodrigo, R. TimeCaps: Learning from Time Series Data with Capsule Networks. arXiv
**2019**, arXiv:1911.11800. [Google Scholar] - Peng, P.; He, Z.; Wang, L.; Jiang, Y. Microseismic records classification using capsule network with limited training samples in underground mining. Sci. Rep.
**2020**, 10, 13925. [Google Scholar] [CrossRef] [PubMed] - Cai, Z.; Jönsson, P.; Jin, H.; Eklundh, L. Performance of Smoothing Methods for Reconstructing NDVI Time-Series and Estimating Vegetation Phenology from MODIS Data. Remote Sens.
**2017**, 9, 1271. [Google Scholar] [CrossRef] [Green Version] - Sadeghi, M.; Behnia, F.; Amiri, R. Window Selection of the Savitzky-Golay Filters for Signal Recovery from Noisy Measurements. IEEE Trans. Instrum. Meas.
**2020**, 69, 5418–5427. [Google Scholar] [CrossRef] - Sameni, R. Online filtering using piecewise smoothness priors: Application to normal and abnormal electrocardiogram denoising. Signal Process.
**2017**, 133, 52–63. [Google Scholar] [CrossRef] [Green Version] - Koziol, P.; Raczkowska, M.K.; Skibinska, J.; Urbaniak-Wasik, S.; Paluszkiewicz, C.; Kwiatek, W.; Wrobel, T.P. Comparison of spectral and spatial denoising techniques in the context of High Definition FT-IR imaging hyperspectral data. Sci. Rep.
**2018**, 8. [Google Scholar] [CrossRef] [PubMed] - Kong, D.; Zhang, Y.; Gu, X.; Wang, D. A robust method for reconstructing global MODIS EVI time series on the Google Earth Engine. ISPRS J. Photogramm. Remote Sens.
**2019**, 155, 13–24. [Google Scholar] [CrossRef] - Yang, Y.; Luo, J.; Huang, Q.; Wu, W.; Sun, Y. Sun Weighted Double-Logistic Function Fitting Method for Reconstructing the High-Quality Sentinel-2 NDVI Time Series Data Set. Remote Sens.
**2019**, 11, 2342. [Google Scholar] [CrossRef] [Green Version] - Forkel, M.; Carvalhais, N.; Verbesselt, J.; Mahecha, M.; Neigh, C.; Reichstein, M. Trend Change Detection in NDVI Time Series: Effects of Inter-Annual Variability and Methodology. Remote Sens.
**2013**, 5, 2113–2144. [Google Scholar] [CrossRef] [Green Version] - Roy, D.P.; Yan, L. Robust Landsat-based crop time series modelling. Remote Sens. Environ.
**2020**, 238, 110810. [Google Scholar] [CrossRef] - Yan, L.; Roy, D.P. Spatially and temporally complete Landsat reflectance time series modelling: The fill-and-fit approach. Remote Sens. Environ.
**2020**, 241, 111718. [Google Scholar] [CrossRef] - Vorobiova, N.; Chernov, A. Curve fitting of MODIS NDVI time series in the task of early crops identification by satellite images. Procedia Eng.
**2017**, 201, 184–195. [Google Scholar] [CrossRef] - Al-Nahhal, I.; Dobre, O.A.; Basar, E.; Moloney, C.; Ikki, S. A Fast, Accurate, and Separable Method for Fitting a Gaussian Function [Tips & Tricks]. IEEE Signal Process. Mag.
**2019**, 36, 157–163. [Google Scholar] - Khanal, N.; Matin, M.A.; Uddin, K.; Poortinga, A.; Chishtie, F.; Tenneson, K.; Saah, D. A comparison of three temporal smoothing algorithms to improve land cover classification: A case study from NEPAL. Remote Sens.
**2020**, 12, 2888. [Google Scholar] [CrossRef] - Pan, Z.; Hu, Y.; Cao, B. Construction of smooth daily remote sensing time series data: A higher spatiotemporal resolution perspective. Open Geospat. Data Softw. Stand.
**2017**, 2, 25. [Google Scholar] [CrossRef] [Green Version] - Tepper, M.; Sapiro, G. L1 Splines for Robust, Simple, and Fast Smoothing of Grid Data. arXiv
**2012**, arXiv:1208.2292. [Google Scholar] - Patterson, N.K.; Lane, B.A.; Sandoval-Solis, S.; Pasternack, G.B.; Yarnell, S.M.; Qiu, Y. A hydrologic feature detection algorithm to quantify seasonal components of flow regimes. J. Hydrol.
**2020**, 585, 124787. [Google Scholar] [CrossRef] - Tian, C.; Fei, L.; Zheng, W.; Xu, Y.; Zuo, W.; Lin, C.W. Deep learning on image denoising: An overview. Neural Netw.
**2020**, 131, 251–275. [Google Scholar] [CrossRef] [PubMed] - Kolbæk, M.; Tan, Z.-H.; Jensen, S.H.; Jensen, J. On Loss Functions for Supervised Monaural Time-Domain Speech Enhancement. IEEE/ACM Trans. Audio Speech Lang. Process.
**2019**, 28, 825–838. [Google Scholar] [CrossRef] [Green Version] - Lai, X.; Wu, X.; Zhang, L.; Lu, W.; Zhong, C. Imputations of missing values using a tracking-removed autoencoder trained with incomplete data. Neurocomputing
**2019**, 366, 54–65. [Google Scholar] [CrossRef] - Chai, X.; Gu, H.; Li, F.; Duan, H.; Hu, X.; Lin, K. Deep learning for irregularly and regularly missing data reconstruction. Sci. Rep.
**2020**, 10, 3302. [Google Scholar] [CrossRef] - Guo, Y.; Wang, H.; Hu, Q.; Liu, H.; Liu, L.; Bennamoun, M. Deep Learning for 3D Point Clouds: A Survey. IEEE Trans. Pattern Anal. Mach. Intell.
**2020**. [Google Scholar] [CrossRef] - Soneson, C.; Gerster, S.; Delorenzi, M. Batch Effect Confounding Leads to Strong Bias in Performance Estimates Obtained by Cross-Validation. PLoS ONE
**2014**, 9, e100335. [Google Scholar] [CrossRef] [PubMed] - Barredo, A.A.; Díaz-Rodríguez, N.; Del Ser, J.; Bennetot, A.; Tabik, S.; Barbado, A.; Garcia, S.; Gil-Lopez, S.; Molina, D.; Benjamins, R.; et al. Explainable Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Inf. Fusion
**2020**, 58, 82–115. [Google Scholar] [CrossRef] [Green Version] - Samek, W.; Montavon, G.; Lapuschkin, S.; Anders, C.J.; Müller, K.R. Toward interpretable machine learning: Transparent deep neural networks and beyond. arXiv
**2020**, arXiv:2003.07631. [Google Scholar] - Cuturi, M.; Blondel, M. Soft-DTW: A Differentiable Loss Function for Time-Series. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, Australia, 6–11 August 2017; Volume 2, pp. 1483–1505. [Google Scholar]
- Naesseth, C.A.; Ruiz, F.J.R.; Linderman, S.W.; Blei, D.M. Reparameterization gradients through acceptance-rejection sampling algorithms. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, AISTATS 2017, Fort Lauderdale, FL, USA, 20–22 April 2017; pp. 489–498. [Google Scholar]
- Kim, H.; Mnih, A. Disentangling by Factorising. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, 10–15 July 2018; Volume 6, pp. 4153–4171. [Google Scholar]
- Montavon, G.; Samek, W.; Müller, K.R. Methods for interpreting and understanding deep neural networks. Digit. Signal Process. A Rev. J.
**2018**, 73, 1–15. [Google Scholar] [CrossRef] - Montavon, G.; Lapuschkin, S.; Binder, A.; Samek, W.; Müller, K.R. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit.
**2017**, 65, 211–222. [Google Scholar] [CrossRef] - Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.R.; Samek, W. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE
**2015**, 10, e0130140. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Zhang, Z.; Duan, F.; Sole-Casals, J.; Dinares-Ferran, J.; Cichocki, A.; Yang, Z.; Sun, Z. A Novel Deep Learning Approach with Data Augmentation to Classify Motor Imagery Signals. IEEE Access
**2019**, 7, 15945–15954. [Google Scholar] [CrossRef] - Belgiu, M.; Bijker, W.; Csillik, O.; Stein, A. Phenology-based sample generation for supervised crop type classification. Int. J. Appl. Earth Obs. Geoinf.
**2021**, 95, 102264. [Google Scholar] [CrossRef] - Hird, J.N.; McDermid, G.J. Noise reduction of NDVI time series: An empirical comparison of selected techniques. Remote Sens. Environ.
**2009**, 113, 248–258. [Google Scholar] [CrossRef] - Jiang, Y.; Li, H.; Rangaswamy, M. Deep Learning Denoising Based Line Spectral Estimation. IEEE Signal Process. Lett.
**2019**, 26, 1573–1577. [Google Scholar] [CrossRef] - Srivastava, N.; Hinton, G.; Krizhevsky, A.; Salakhutdinov, R. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res.
**2014**, 15, 1919–1929. [Google Scholar] - Xiang, C.; Zhang, L.; Tang, Y.; Zou, W.; Xu, C. MS-CapsNet: A Novel Multi-Scale Capsule Network MS-CapsNet: A Novel Multi-Scale Capsule Network. IEEE Signal Process. Lett.
**2018**, 25, 1850–1854. [Google Scholar] [CrossRef] - Arun, P.V.; Buddhiraju, K.M.; Porwal, A.; Chanussot, J. CNN-Based Super-Resolution of Hyperspectral Images. IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 6106–6121. [Google Scholar] [CrossRef] - Bochinski, E.; Senst, T.; Sikora, T. Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In Proceedings of the International Conference on Image Processing, Beijing, China, 17–20 September 2018; Volume 2017, pp. 3924–3928. [Google Scholar]
- Herrmann, I.; Shapira, U.; Kinast, S.; Karnieli, A.; Bonfil, D.J. Ground-level hyperspectral imagery for detecting weeds in wheat fields. Precis. Agric.
**2013**, 14, 637–659. [Google Scholar] [CrossRef] - Herrmann, I.; Pimstein, A.; Karnieli, A.; Cohen, Y.; Alchanatis, V.; Bonfil, D.J. LAI assessment of wheat and potato crops by VENμS and Sentinel-2 bands. Remote Sens. Environ.
**2011**, 115, 2141–2151. [Google Scholar] [CrossRef] - Fawaz, H.I.; Forestier, G.; Weber, J.; Idoumghar, L.; Muller, P.-A. Deep learning for time series classification: A review. Data Min. Knowl. Discov.
**2018**, 33, 917–963. [Google Scholar] [CrossRef] [Green Version] - Zhong, L.; Hu, L.; Zhou, H. Deep learning based multi-temporal crop classification. Remote Sens. Environ.
**2019**, 221, 430–443. [Google Scholar] [CrossRef] - Mou, L.; Zhu, X.X. Learning to Pay Attention on Spectral Domain: A Spectral Attention Module-Based Convolutional Network for Hyperspectral Image Classification. IEEE Trans. Geosci. Remote Sens.
**2020**, 58, 110–122. [Google Scholar] [CrossRef] - Honke, G.; Higgins, I.; Thigpen, N.; Miskovic, V.; Link, K.; Duan, S.; Gupta, P.; Klawohn, J.; Hajcak, G. Representation learning for improved interpretability and classification accuracy of clinical factors from EEG. arXiv
**2020**, arXiv:2010.15274. [Google Scholar] - Karim, F.; Majumdar, S.; Darabi, H.; Harford, S. Multivariate LSTM-FCNs for Time Series Classification. Neural Netw.
**2018**, 116, 237–245. [Google Scholar] [CrossRef] [Green Version] - Zhao, A.X.; Tang, X.J.; Zhang, Z.H.; Liu, J.H. The parameters optimization selection of Savitzky-Golay filter and its application in smoothing pretreatment for FTIR spectra. In Proceedings of the 2014 9th IEEE Conference on Industrial Electronics and Applications, ICIEA 2014, Hangzhou, China, 9–11 June 2014; pp. 516–521. [Google Scholar]
- Julien, Y.; Sobrino, J.A. Optimizing and comparing gap-filling techniques using simulated NDVI time series from remotely sensed global data. Int. J. Appl. Earth Obs. Geoinf.
**2019**, 76, 93–111. [Google Scholar] [CrossRef]

**Figure 2.**Proposed variational capsule network (VCapsNet) architecture without pooling (µ and σ denote mean and standard deviation, respectively).

**Figure 3.**Proposed variational capsule network (VCapsNet) architecture with pooling (µ and σ denote mean and standard deviation respectively).

**Figure 5.**Sensitivity analysis of the variational capsule network (VCapsNet) in terms of (

**a**) network depth; (

**b**) size of filters; and (

**c**) number of filters for classification.

**Figure 6.**Sensitivity analysis of the variational capsule network (VCapsNet) in terms of (

**a**) network depth; (

**b**) size of filters; and (

**c**) number of filters for denoising.

**Figure 9.**Accuracy analysis of smoothing approaches with respect to the change in the percentage of training samples.

**Table 1.**Analysis of the effect of the proposed architectures and constraints for 50% of the training samples *.

Architectural Variations/Losses | Kappa Statistic | Overall Accuracy | Z-Score |
---|---|---|---|

1D Capsule based classifier | 0.81 | 83.46 | 2.21 |

Long Short-Term Memory (LSTM) based classifier | 0.82 | 85.91 | 2.18 |

VCapsNet without Capsule Stream | 0.85 | 87.18 | 2.25 |

VCapsNet without Long Short-Term Memory (LSTM) stream | 0.86 | 89.68 | 1.99 |

VCapsNet without variational encoding | 0.86 | 91.24 | 2.32 |

VCapsNet without embedding the label information prior | 0.86 | 92.68 | 2.16 |

VCapsNet without fine-tuning the latent space for classification | 0.86 | 90.12 | 2.17 |

VCapsNet without piece-wise reconstruction loss | 0.87 | 88.56 | 2.09 |

VCapsNet without cosine dissimilarity loss | 0.88 | 91.43 | 1.97 |

VCapsNet without Dynamic Time Wrapping (DTW) loss | 0.87 | 89.08 | 1.99 |

VCapsNet without interpolated convolution | 0.89 | 94.12 | 2.12 |

Proposed VCapsNet implementation | 0.94 | 98.57 | - |

**Table 2.**Comparison of the variational capsule network (VCapsNet) with benchmark deep learning (DL) classifiers for 40% of training samples *.

Benchmark Classifiers | Kappa Statistic | Overall Accuracy |
---|---|---|

Deep learning (DL) based [92] | 0.74 | 80.10 |

Spectral attention CNN [93] | 0.79 | 84.43 |

Phenology metrics based [9] | 0.85 | 87.91 |

Bayesian estimator based [2] | 0.83 | 88.20 |

Representation learning based [94] | 0.86 | 90.08 |

Generative Adversarial Network (GAN) based [39] | 0.83 | 88.36 |

Multivariate Long Short-Term Memory (LSTM) based [95] | 0.87 | 90.82 |

Proposed VCapsNet | 0.92 | 96.23 |

**Table 3.**Z-score-based significance analysis of variational capsule network (VCapsNet) in comparison with the benchmark deep learning (DL) classifiers *.

Benchmark Smoothing Approaches | Z-score of Kappa Statistic as Compared to VCapsNet | Z-score of Overall Accuracy as Compared to VCapsNet |
---|---|---|

Deep learning (DL) based [92] | 2.03 | 1.98 |

Spectral attention Convolutional Neural Network (CNN) [93] | 1.98 | 2.21 |

Phenology metrics based [9] | 2.12 | 2.14 |

Bayesian estimator based [2] | 2.64 | 2.43 |

Representation learning based [94] | 2.07 | 2.07 |

Generative Adversarial Network (GAN) based [39] | 2.59 | 2.46 |

Multivariate Long Short-Term Memory (LSTM) based [95] | 2.18 | 1.99 |

Benchmark Classifiers. | Normalized Cosine Similarity Between the Concepts Learned and the Benchmark Phenological Curves | Normalized RMSE in Fourier Domain Between the Concepts Learned and the Benchmark Phenological Curves | Normalized DTW Based Similarity Between the Concepts Learned and the Phenological Benchmark Curves |
---|---|---|---|

Deep learning (DL) based [92] | 0.9874 | 0.4290 | 0.7742 |

Spectral attention Convolutional Neural Network (CNN) [93] | 0.9870 | 0.3916 | 0.7891 |

Representation learning based [94] | 0.9942 | 0.2396 | 0.8263 |

Generative Adversarial Network (GAN) based [39] | 0.9964 | 0.2246 | 0.8551 |

Multivariate Long Short-Term Memory (LSTM) based [95] | 0.9968 | 0.2389 | 0.8629 |

Proposed VCapsNet | 0.9989 | 0.1059 | 0.9672 |

Benchmark Classifiers. | Overall Accuracy | Kappa Statistic | Normalized DTW Similarity Between the Concepts Learned and the Phenological Benchmark Curves |
---|---|---|---|

Deep learning (DL) based [92] | 67.41 | 0.62 | 0.6591 |

Spectral attention Convolutional Neural Network (CNN) [93] | 71.82 | 0.65 | 0.6983 |

Representation learning based [94] | 76.33 | 0.72 | 0.7150 |

Generative Adversarial Network (GAN) based [39] | 74.49 | 0.66 | 0.6840 |

Multivariate Long Short-Term Memory (LSTM) based [95] | 79.09 | 0.74 | 0.7651 |

Proposed VCapsNet | 89.72 | 0.86 | 0.9439 |

Benchmark Smoothing Approaches | Z-Score of Kappa Statistic as Compared to VCapsNet | Z-Score of Overall Accuracy as Compared to VCapsNet |
---|---|---|

Deep learning (DL)-based [92] | 1.97 | 2.01 |

Spectral attention Convolutional Neural Network (CNN) [93] | 2.17 | 2.34 |

Phenology metric-based [9] | 2.09 | 2.16 |

Bayesian estimator-based [2] | 1.99 | 2.05 |

Representation learning-based [94] | 2.14 | 2.37 |

Generative Adversarial Network (GAN)-based [39] | 2.08 | 2.42 |

Multivariate Long Short-Term Memory (LSTM)-based [95] | 1.97 | 2.14 |

Benchmark Smoothing Approaches | PSNR |
---|---|

Least squares fitting to double logistic functions [57] | 24.32 |

Least square fitting to asymmetric Gaussian functions [62] | 27.92 |

Spline smoothing [66] | 29.05 |

Deep learning (DL) based [69] | 31.93 |

Savitzky-Golay filter based [56] | 29.51 |

Deep learning (DL) based [70] | 33.74 |

Savitzky-Golay filter based [53] | 31.83 |

Proposed VCapsNet implementation | 37.90 |

Benchmark Smoothing Approaches | Z-Score with Respect to VCapsNet |
---|---|

Least squares fitting to double logistic functions [57] | 2.38 |

Least square fitting to asymmetric gaussian functions [62] | 2.20 |

Spline smoothing [66] | 2.09 |

Deep learning (DL) based [69] | 1.99 |

Savitzky-Golay filter based [56] | 2.29 |

Deep learning (DL) based [70] | 2.20 |

Savitzky-Golay filter based [53] | 2.16 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Arun, P.V.; Karnieli, A.
Deep Learning-Based Phenological Event Modeling for Classification of Crops. *Remote Sens.* **2021**, *13*, 2477.
https://doi.org/10.3390/rs13132477

**AMA Style**

Arun PV, Karnieli A.
Deep Learning-Based Phenological Event Modeling for Classification of Crops. *Remote Sensing*. 2021; 13(13):2477.
https://doi.org/10.3390/rs13132477

**Chicago/Turabian Style**

Arun, Pattathal V., and Arnon Karnieli.
2021. "Deep Learning-Based Phenological Event Modeling for Classification of Crops" *Remote Sensing* 13, no. 13: 2477.
https://doi.org/10.3390/rs13132477