Crop yield is the most important indicator in agriculture and has numerous connections with human society. Yield prediction, one of the most challenging tasks in precision agriculture, is of great significance for yield mapping, crop market planning, crop insurance, and harvest management [1
Remote sensing has been widely used in agricultural applications including cropland cover classification, drought stress estimation, and yield prediction by under its macro-performance and periodicity [2
]. Various relevant information can be extracted from remote sensing data for yield prediction. Particularly, vegetation indices (VIs), such as the Normalized Difference Vegetation Index (NDVI), have been widely utilized [3
]. The other indices, such as Green Leaf Area Index (GLAI) [7
], Crop Water Stress Index (CWSI) [8
], Normalized Difference Water Index (NDWI) [9
], Green Vegetation Index (GVI), Soil-Adjusted Vegetation Index (SAVI) [10
], Enhanced Vegetation Index (EVI) [11
], etc., have also been used for crop production forecasting. Also, besides, meteorological variables, such as precipitation; air temperature [12
]; and some soil condition data, including soil moisture, temperature, and quality, were often adopted in yield prediction as crop growth environment indicators [15
Based on the remote sensing data, there are mainly two kinds of approaches for crop yield prediction: crop simulation and empirical statistical models [16
]. Although crop simulation models are precisely simulating the physical processes in crop growth, these models can be barely applied in large spatio-temporal scales due to insufficient data. In contrast, empirical statistical models are simple and require fewer input data, and have therefore been broadly used as a common alternative to process-based models. Machine-learning algorithms, including Support Vector Machine (SVM), Decision Trees (DT), Multilayer Perception (MLP), and Restricted Boltzmann Machine (RBM) [17
], can provide alternatives to traditional regression approaches and overcome many limitations. Besides, artificial neural network (ANN) was also considered as an alternative model. Traditional ANN, the multilayer perceptron model, has been applied successfully to crop yield estimation with various types of crops [18
Recently, Deep Learning (DL) has been considered a breakthrough technology in machine learning and data mining agricultural remote sensing. Most of the DL algorithms, including Stacked Sparse Autoencoder (SSAE), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN), have been applied for yield prediction. Ma et al. proposed an SSAE for rice yield estimation using climatic and MODIS data, the result showed that the SSAE model can outperform the ANN model [21
]. Kuwata and Shibasaki used a Caffe-based deep learning regression model (Gaussian Radial Basis Function) trained with remotely sensed data (satellite data, climate data, and environmental metadata) to model corn crop yield estimation at county-level [22
]. Nevavuori et al. proposed a CNN model for crop yield prediction based on NDVI and RGB data acquired from UAVs, the result showed the CNN architecture performed better with RGB data than the NDVI data [23
]. Yang et al. found that the CNN trained by RGB and multispectral datasets perform much better than a VI-based regression model for rice grain yield estimation at the ripening stage [24
]. Chen et al. proposed a faster region-based convolutional neural network (R-CNN) to detect and count the number of flowers, mature strawberries, and immature strawberries for yield prediction [25
]. Russello designed a 3D CNN architecture for crop yield prediction, the results significantly outperform competing state-of-the-art machine learning methods [26
]. In addition, some researches try to integrate temporal characteristics to predict crop yield by using RNN. Jiang et al. employed the Long Short-Term Memory(LSTM), a special form of RNN method to predict corn yield with weather and soil data, the empirical results from county-level data in Iowa show promising predictive power [27
]. Kulkarni et al. designed an RNN to identify optimal combinations of soil parameters and blended it with the rainfall pattern in a selected region to evolve the expectable crop yield [28
]. Haider et al.developed an accurate wheat production forecasting model using the LSTM; the results verify that the model achieves satisfying performance in terms of forecasting [29
]. You et al. introduced a novel method incorporating a Gaussian Process component into a CNN or LSTM; the results showed that the proposed method can outperform traditional remote sensing-based methods by 30% in terms of root-mean-squared error (RMSE) [30
]. Alhnaity et al. employed the LSTM to predict yield and plant growth variation across two different scenarios—tomato yield forecasting and Ficus benjamina stem growth, in controlled greenhouse environments [31
]. It is generally accepted that CNN can explore more spatial features and LSTM has the ability to reveal phenological characteristics [32
]; nevertheless, to our knowledge, little attention has been devoted to comprehensively utilize the advantages of CNN and RNN for yield prediction at the county-level.
Higher accuracy and better spatial scale are two goals of crop yield prediction, they also issued a challenge for downloading, analyzing, and managing a multidecadal time series of satellite images over large areas, which is not feasible using desktop computing resources. DL has emerged together with big data technologies and high-performance computing to create new opportunities, so as to unravel, quantify, and understand the relationship between remote sensing data and crop yield at a fine scale. With the advent of Google Earth Engine (GEE), a robust connection to the Internet is now all that is required to access, manipulate, and analyze long-term global comprehensive data at various scales [33
]. Whereas, some of the related works based on the GEE merely employ a crop statistical model for yield prediction [37
], and many other DL based methods only use the GEE as a data preprocessing tool and downloaded the raw data to local drive from it, which does not take full advantage of its enormous computing power [26
Generally, the crop yield mission always spans from the in-season to the end-of-season [43
]. In the U.S., USDA provides a crop yield forecasting service, namely, Objective Yield (OY) surveys, which can provide monthly forecasts of crop yield by state-level. The OY survey field work often starts from July 25 for soybean, then the yield forecasts can be issued from Aug to the end of the season. However, the county-level soybean yield estimation cannot be issued from USDA until the next March. An early accurate county-level soybean yield prediction before that issue date is of significance for early marketing decisions and harvesting management at a fine scale. This paper proposed a deep CNN-LSTM model for both end-of-season and in-season yield prediction in CONUS at the county-level. Based on the GEE, several long-term monitored variables, including weather data, MODIS LST, and MODIS SR data, were transformed into tensors for model training; besides, historical soybean yield data was used for label and validation.
The main aims of this study are (1) to evaluate the performance of the proposed method for end-of-season crop yield prediction and (2) to explore how early a satisfied in-season crop yield prediction can be achieved. To verify the prediction power of the proposed CNN-LSTM model, two classic DL network architectures, including CNN and LSTM, were employed for comparison.
Accurate early yield prediction is of great significance for crop market planning, crop insurance, and harvest management. In this paper, a GEE-based CNN-LSTM model was proposed for both in-season and end-of-season soybean yield prediction by county-level in CONUS. From 2011 to 2015, the results demonstrate for the first time evidence that (1) compared with the CNN or LSTM, the prediction performance of the proposed CNN-LSTM model was proven to be the best. Based on the proposed method, the end-of-season yield prediction can obtain high accuracy with RMSE = 329.53 averaged from 2011 to 2015 and for five years together. (2) An early prediction on AUG 21st can achieve a satisfying result with RMSE = 353.74 and , which is comparable to end-of-season result but can win a long time before USDA issue data. (3) The method is highly efficient, as it can benefit from the great computing power of GEE and a dimension reduction method. (4) MODIS surface reflectance played a more important role in the method than environmental features.
However, as a preliminary attempt to investigate a U.S. county-Level soybean yield prediction using CNN-LSTM, a few improvements may be taken into consideration in future work. First, using only weather and LST data may be insufficient for yield prediction, and more features could be added to the training data such as soil moisture, soil quality, transpiration, and irrigation situation, which makes the model more comprehensive. Second, although the proposed method employs a histogram-based tensor transformation that can fuse different remote sensing data into a composite, combining multisource data with different resolution and cadence for feature extraction remains challenging, for example, some data is monthly or yearly while some of the data may be constant. To accommodate the data, some optimization should be adopted on the model architecture. Third, the resolution of tensors depends on the bin number, different bin numbers, such as 64, 128, or higher, will be tested for performance comparison. This method can offer exciting opportunities for other kinds of early crop yield predictions at larger scales in the future.