Evaluating the Influence of Missing Data from the Crop Vegetation Index Time Series on Copernicus HR-VPP Phenological Products ^†

Abstract

Phenological parameters extracted from time series (TS) of spectral indices are essential to characterizing crops. However, the lack of data in the TS can affect their accuracy. The Copernicus Land Monitoring Service (CLMS) provides these parameters and their temporal quality. This paper evaluates the impact of missing vegetation index data on phenological parameters, namely, SOS, EOS, and MAX, for extensive arable crop between 2018 and 2023. The TSGenerator package was developed to download, process, and analyze the data. We used 252 images from the BIOPAR-VI module, 6 phenology parameters, and 2025 plots of barley and maize in Monegros and Zaidín, Spain. In barley, SOS and MAX showed 42.9% and 40.9% of missing data, while in maize, SOS and EOS showed 36.6% and 41.0%. The correlation between the Copernicus VPP quality parameter and the proposed one was r = 0.89 for barley and r = 0.74 for maize. This study advances the understanding of the effect of missing data on SOS, EOS, and MAX.

1. Introduction

Crop phenological characterization is fundamental to optimizing agricultural management and resource sustainability. Vegetation indices (VIs) derived from satellite images are key tools for monitoring crop growth [1]. From the VI time series, several phenological metrics are extracted, such as the start, end, and time of maximum vegetation development (SOS, EOS, and MAX), among others, which allow for monitoring of the state of health of vegetation and its response to climate change [2]. Focusing on crops, the accurate extraction of these metrics plays a fundamental role; it allows farmers to make informed decisions on crop management, fertilizer and pesticide application, and irrigation [3]. Phenological metrics are obtained through the analysis of temporal curves that identify critical transitions in the growth cycle [3]. Copernicus High Resolution Vegetation Phenology and Productivity (HR-VPP) products provide these parameters operationally from Sentinel-2 [4], offering a standardized alternative that avoids individual time series processing for large-scale phenological analyses.

However, despite these operational advantages, the reliability of these metrics is sometimes compromised by the influence of clouds and cloud shadows in the VI time series [5], elements that cause missing data in these series. An incorrect analysis of the missing data patterns in VI time series can lead to a misselection of the methods and approaches needed to address this issue or an inaccurate assessment of their influence on the estimated phenological metrics [6].

The aim of this work is to assess, for extensive herbaceous crops, the impact of missing vegetation index time data on the phenological parameters provided by the CLMS of Copernicus at the time of occurrence.

2. Materials and Methods

2.1. Study Areas and Crops

We focused the study on two typical arable crops, barley and maize, in two irrigated areas, Zaidín and Monegros, in northern Spain (Figure S1). We selected 2025 plots of at least 5 hectares, with information on the studied crops derived from the anonymous records of the Common Agricultural Policy, obtained from the Government of Aragon for the period 2018–2023. The spatial information for the plots was collected through the Geographic Information System for Agricultural Plots (SIGPAC, for its acronym in Spanish) in shapefile format [7]. This system allows for the geographical identification of the parcels declared by farmers in any aid scheme related to the cultivated area.

2.2. Proposed Library and Procedure

A library and procedure called TSGenerator is proposed in this study for downloading, processing, and analyzing the data. In addition, this procedure allows for the evaluation of the parameters related to SOS, EOS, and MAX. The package incorporates several libraries [8]. The reticulate library, a powerful tool used by R users who wish to take advantage of Python’s (version 3.9.2) capabilities, is the most important [9]. Reticulate facilitates the integration and use of Python libraries within the R environment, allowing for more efficient data analysis and manipulation. Specifically, TSGenerator uses the Python package “Harmonized Data Access API” (hda). The procedure proposed in this study, using the TSGenerator library, consists of two main blocks: (i) time series processing and (ii) data analysis and visualization (Figure 1).

2.2.1. Time Series Processing

In the present study, high spatial resolution products on phenology and vegetation productivity (HR-VPP) from the Copernicus Land Monitoring Service were used. These products are derived from the Sentinel-2 satellite constellation (Figure 2A,B) [4].

A time series of 252 (average of 42 per year) images of the Normalized Difference Vegetation Index (NDVI) and the quality product QFLAG2, processed in the BIOAPAR-VI module, and 6 images of the vegetation phenology and productivity parameters (VPP): SOS, EOS, and MAX(day of the year) and the quality parameter QFLAG-VPP (Figure S2 and Table S1), processed in the TIMESAT module of the HR-VPP System (tile 30/TYM), all between the years 2018 and 2023, were downloaded. The download was performed using two of the main functions of the TSGenerator library: Download.VI and Download.HRVPP. These functions use the Python package “Harmonized Data Access API” (hda); this is a REST-based API that all users can use for free to download data from WEkEO (for more information, see https://help.wekeo.eu/en/articles/9515753-what-is-the-harmonized-data-access-hda-api (accessed on 30 March 2025)). Noisy pixels in each of the NDVI series images were removed in two steps using the functions get.Stack and get.CleanVI: first creating a combination between both products corresponding to each date (raster stack NDVI + QFLAG2, Figure 2C) and then removing those pixels where QFLAG2 ≠ 1 (Figure 2D).

We used the functions get.Series.median and get.Series.VPP to extract and store in .CSV format the median time series of NDVI, VPP, and QFLAG-VPP product values over the period 2018–2023 for each of the selected 2025 plots.

2.2.2. Data Analysis and Visualization

We focused on analyzing the percentage of missing data around the critical moments of the phenology of the crops studied in each zone. For this purpose, we calculated the global mean values (expressed in days of the year) of SOS, EOS, and MAX for each crop from the values extracted in each plot, as described in Section 2.2.1. From these, we calculated the percentage of missing data around these parameters ±21 days. Subsequently, Pearson’s correlation between the mean percentage value of non-missing values between the two phenological moments most affected in each crop and the Copernicus QFLAG-VPP quality product (Table S2) was performed. Finally, we estimated the probability of missing data for each VI observation on each day of the year using a Generalized Additive Model (GAM) [10], according to the following equation:

$$log \left({\displaystyle \frac{Missing}{No-Missing}}\right)={\beta }_0+f\left(DOY\right)+{\beta }_1\times Year$$

(1)

where:

$log \left({\displaystyle \frac{Missing}{No-Missing}}\right)$: The link function that models the probability of missing data (Missing) versus non-missing data (No-Missing).

${\beta }_0$: It represents the reference log probabilities of the missing data when all other variables are equal to zero. It is estimated from the data through maximum likelihood estimation during model fitting.

${\beta }_1\times Year$: Linear term to capture trends over years, which is useful if the probability of missing data is expected to change over time. ${\beta }_1$ was estimated using penalized maximum likelihood within the GAM framework, representing the linear trend in the log-odds of missing data over time.

$f\left(DOY\right)$: A polynomial spline function that allows for capturing the non-linear relationship between the day of the year and the probability of missing data. The smoothing parameter was automatically selected using generalized cross-validation to optimize the trade-off between model fit and complexity, allowing the model to adapt to the underlying seasonal patterns in data availability without overfitting.

3. Results and Discussion

3.1. Percentage of Missing Data at Phenological Moments and Correlation with the QFLAG-VPP Product

When estimating the percentage of missing data ±21 around the phenological moments SOS, MAX, and EOS estimated from the Copernicus HR-VPP products in each year of study, we observed that the SOS and MAX parameters were the moments most affected by the absence of data with 42.69% and 39.58% in the Monegros region and 48.15% and 42.36% in the Zaidín region for the barley crop, respectively. On the other hand, in the maize crop, the most affected moments were SOS and EOS with 35.86% and 40.10% in Monegros and 37.5% and 41.81% in Zaidín, respectively (Figure 3, Tables S3 and S4). These results show how the reliability of these phenological products is susceptible to the level of cloudiness and cloud shadows that may be present in the VI time series [2,11].

We correlated the estimated mean percentage of non-missing data between the two phenological moments most affected in each crop and the quality product QFLAG-VPP, showing a strong positive correlation of r = 0.89 in barley and r = 0.74 in maize. It should be mentioned that, although a strong positive correlation was determined between both products, they were determined differently. The authors of Ref. [12] explain that QFLAG-VPP estimates the data availability in green up, green peak, and green down zones of the growing season, which are defined as the left 20–left 80%, left 80–right 80%, and right 80–right 20% of the season amplitude, so it is not possible to infer whether one is superior to the other; however, they could complement each other to give an accurate assessment of the reliability of phenological parameters.

3.2. Probability of Missing Data Estimated Using the Generalized Additive Model

Julien and Sobrino [13], Militino et al. [14], and Vuolo et al. [15] mention that cloudy periods that affect the quality of VI time series can persist for some periods of time (seasonally localized), although they may vary according to the region studied. The GAM results reaffirm what was mentioned by the authors of [13], that the existence of missing data is not random, but follows a very marked temporal pattern. The identified temporal patterns can serve as a basis for the development of phenology-specific interpolation methods that take into account the non-random nature of data gaps, potentially improving traditional time series interpolation approaches.

For the barley crop, a probability of about 50% of missing data was determined for the phenological moments SOS and MAX (Figure 4 and Figures S3 and S4). The same probability was found for the maize crop, but at the SOS and EOS moments. This high percentage of missing data reduces the reliability of phenological moments and increases the degree of uncertainty during their estimation. Efficient crop management can be influenced by this, especially the optimal estimation of water requirements and thus irrigation water management. The phenological moments with the highest probability of missing data occur during the period of greatest cloud cover in the study region.

4. Conclusions

It was demonstrated that the proposed methodology using the TSGenerator package is effective for downloading, processing, and analyzing time series, allowing for the retrieval and evaluation of missing data for the SOS, MAX, and EOS parameters. Moreover, it is independently replicable in any region, vegetation type, and climatic conditions. There was a strong correlation between the product resulting from the proposed methodology and the Copernicus QFLAG-VPP quality product. The phenological moments with the highest probability of being affected by missing data in the time series of vegetation indices were identified: SOS and MAX for barley, as well as SOS and EOS for maize. The missing data analyses evaluated in the study can be applied to other types of spectral data; however, the downloading and pre-processing procedures are limited to the phenology products provided by the CLMS. This work represents an advance in the knowledge of the effect of missing data at the specific times of SOS, EOS, and MAX. The results obtained open a clear opportunity to develop and evaluate specific data reconstruction techniques aimed at the most critical phenological moments identified, incorporating the knowledge of the temporal patterns characterized herein to optimize gap-filling algorithms.