Generation of Synthetic Hyperspectral Image Cube for Mapping Soil Organic Carbon Using Proximal Remote Sensing ^†

Abstract

The advent of hyperspectral remote sensing represented a breakthrough in the accurate, fast, and non-invasive estimation of important soil fertility parameters. The present study utilizes non-imaging hyperspectral data in the spectral range of 350–2500 nm for estimating soil organic carbon (SOC) content. When partial least squares (PLS) scores were taken as independent variables, support vector machine (SVM) outperformed artificial neural network (ANN) and partial least squares regression (PLSR), achieving an R² value of 0.83. After pre-processing, the proximal spectral values were spatially interpolated to construct a synthetic hyperspectral image of the experimental fields. By applying the regression model to this synthetic hyperspectral imagery, a high-resolution SOC map showing the variability of organic carbon content in the soil was generated.

1. Introduction

The organic content of soil acts as a strong indicator of soil fertility, which impacts agricultural production and the global carbon cycle. The timely assessment of soil organic carbon is essential for managing its deficiencies to enhance crop growth. However, traditional approaches of laboratory-based SOC estimation are time-consuming and costly with respect to larger sample sizes [1]. Therefore, there exists a growing demand for novel sensor-based approaches for estimating soil fertility parameters without compromising accuracy and precision. The recent advances in hyperspectral remote sensing and the development of sophisticated machine learning regression models have facilitated accurate sensor-based estimation and mapping of SOC. Multiple hyperspectral sensors mounted on different platforms ranging from ground proximal, unmanned aerial vehicles (UAVs), and satellites have proved to be effective in quantifying the organic content in soil using hundreds of high-resolution, contiguous and narrow bands in the spectral range of 350–2500 nm [2]. Kriging-based mapping relies on the spatial distribution of sampling points and provides a statistical interpolation of sparse data collected from the field. In contrast, sensor-based approaches provide the real-time quantitative characterization of soil properties through high-resolution maps using their spectroscopic reflectance. Spectral pre-processing, band selection, and regression modeling, including machine learning and deep learning techniques, play a critical role in extracting information from these SOC response bands [3]. This study introduces an extended kriging method to interpolate spectral data, rather than soil properties, to generate a synthetic hyperspectral image cube for creating soil prediction maps with the aid of machine learning models. Although proximal non-imaging hyperspectral sensors, such as ASD spectroradiometers, provide high-resolution spectral bands for accurate SOC estimation, they cannot deliver high-resolution fertility maps compared with UAV-based or space-borne sensors [4]. Therefore, the underlying novel part of the proposed workflow is to generate imagery showing the spatial distribution of soil fertility using non-imagery spectral data.

Thus, the present study proposes a novel approach to generate a synthetic hyperspectral image cube using the spectral information from a proximal hyperspectral non-imaging sensor. Multiple machine learning models will be evaluated to choose the best one for developing a SOC prediction model. The best prediction model will be applied to a synthetic hyperspectral image for generating a high-resolution SOC map.

2. Materials and Methods

A flowchart showing the major steps adopted for this proposed methodology is shown in Figure 1. The major steps are (i) study area and spectral data collection, (ii) generation of synthetic hyperspectral image, and (iii) optimizing PLS scores and machine learning regression. Each step is explained in detail below.

2.1. Study Area and Spectral Data Collection

The study was conducted in a 5-acre experimental plot at the research farm of the ICAR—Indian Agricultural Research Institute (IARI), New Delhi, India, situated at geographical coordinates of 28°37′49.81″ N and 77°9′32.00″ E, with an altitude of 228 m above mean sea level. The entire plot was divided into 200 grids, each with a grid dimension of 10 m × 10 m, for soil sampling and spectral data collection. A total of 200 soil samples were collected from the center of each grid at a depth of 0–15 cm. A study area map showing the location of the experimental plot and the spatial distribution of sampling points is shown in Figure 2. A semi-arid climate with hot and dry summers and cold winters exists in the study area. The soil is taxonomically Typic Haplustept, which belongs to the sandy loam textural class. The collected samples were air-dried, finely ground, and passed through a 2 mm pore diameter sieve for chemical analysis in the laboratory. The SOC was determined using the wet chromic acid oxidation technique, also known as the Walkley–Black method [5]. The global Moran’s I index was used to examine the autocorrelation test of the SOC distribution.

The ASD Field Spec 4^® Standard-Res portable spectroradiometer (PANalytical Spectral Devices, Inc., Boulder, CO, USA) equipped with a spectral gun was used for in situ collection of spectral data over the 350–2500 nm range at 1 nm spectral intervals. At every sampling location, the spectral gun with a field-of-view of 25° was kept at 1 m above the soil surface at the nadir position for capturing the soil spectra, ensuring the maximum field-of-view was fully covered. The standardization of the sensor was carried out using a white calibration board, called spectralon (Labsphere, North Sutton, NH, USA), made up of barium sulfate, before starting the spectral measurements. The integration time was fixed at 100 milliseconds. The spectral pre-processing involves the following steps: (i) splice correction, (ii) removal of marginal bands with low signal-to-noise ratio (350–400 nm and 2350–2500 nm), (iii) removal of water absorption bands (1350–1430 nm and 1800–1950 nm). The splice correction removes the discontinuity in the spectra around 1001 and 1831 nm, allowing smooth merging of the visible–NIR and SWIR ranges. The splice correction was carried out using the View Spec™ Pro software version 6.20 (PANalytical Spectral Devices, Inc., Boulder, CO, USA). A total of 30 scans were taken at each location and subsequently averaged to generate a single representative spectrum of that location.

2.2. Generation of Synthetic Hyperspectral Image

Geostatistical interpolation of the spectral values of all 200 sampling points, belonging to each band, was performed using ordinary kriging (OK) to generate spatially continuous hyperspectral bands. Many studies have reported the use of the OK method in generating accurate spatial maps of SOC [6,7]. Since the sampling grid size was fixed at 10 m, the spatial resolution for the interpolated spatial maps was also chosen as 10 m. For each band, four variogram models—linear, Gaussian, spherical, and exponential—were evaluated using leave-one-out cross-validation to select the optimal one based on the lowest RMSE scoring. These optimal semivariograms will be used to interpolate the band values across the grids for each band. Finally, these layers were stacked to generate a synthetic hyperspectral image cube with a coordinate reference system of UTM Zone 43N, WGS84.

2.3. Optimizing PLS Scores and Machine Learning Regression

The partial least squares (PLS) method reduces spectral dimensionality by transforming data to a new space of a lower dimension without compromising their information content [8]. The first few latent variables or PLS scores, obtained by decomposing the independent (X) variables, proved to account for the majority of data variance and thereby reduced the computational complexity of using hyperspectral data for SOC mapping [4]. After an exhaustive literature review, three multivariate regression models—ANN, SVM, and PLSR—were evaluated using k-fold cross-validation (k = 5) to select the best model for SOC mapping. A fixed random state of 42 was implemented in data splitting during 5-fold cross-validation to ensure reproducibility of the model. The number of PLS components was also optimized during the model development by checking their cross-validation R² values. SVM sees the margin between data points from multiple classes to generate a decision boundary, called an optimal hyperplane, that separates them [9]. ANN mimics the biological neural networks of the human brain, which consists of fundamental computational units called neurons. A three-layer architecture consists of input, intermediate (or hidden), and output layers, each occupied by neurons, which facilitate the transfer of data and generation of the output [10]. PLSR is a linear multivariate approach using the properties of principal component analysis and multiple linear regression models [11]. The models were implemented using the Scikit-learn machine learning module in the Python 3.11.3 programming environment. The parameters for each model were optimized using GridsearchCV. The model performance was evaluated using the coefficient of determination (R²), mean absolute error (MAE), root mean square error (RMSE), and ratio of performance to deviation (RPD). The RPD ranges of 1.4–1.8, 1.8–2.0, and 2.0–2.5 signify fair, good, and very good prediction accuracy, respectively [12].

3. Results

3.1. Descriptive Statistics of Measured SOC

The basic statistical metrics calculated for the measured SOC values are provided in

Table 1. Descriptive statistics of the measured SOC.

Minimum	Maximum	Mean	Median	SD 1	%CV 2	Skewness	Kurtosis
0.18	0.52	0.33	0.33	0.06	18.05	0.16	−0.16

¹ SD = standard deviation, ² CV = coefficient of variation.

. The samples showed good variability in terms of CV, 18.05%. The SOC values ranged from 0.18 to 0.52% with a mean of 0.33, median of 0.33, SD of 0.06, and kurtosis of −0.16. Positive skewness indicates that the SOC distribution is predominantly composed of low values. These basic statistics indicate that the datasets are ideal for effective training and validation during model development. Spatial autocorrelation analysis reveals that SOC exhibits a strong and significant spatial positive correlation (Moran’s I > 0, p < 0.01, Z > 2.58). The Moran’s Index is 0.638, and the z-score is 8.665.

3.2. Synthetic Hyperspectral Imagery

The spectral data consists of 2151 bands, which were pre-processed and reduced to 1693 bands after removing noisy bands. The raw spectra collected from 200 sampling points in the field are shown in Figure 3a.

The synthetic hyperspectral imagery with a 10 m pixel size, generated from 1693 bands through ordinary kriging, is shown in Figure 3b. A representative spectrum taken from a pixel is shown in Figure 3c. The kriging was carried out 1693 times for generating 1693 bands in order to construct a synthetic hyperspectral image of 10 m spatial resolution. Each time, the selection of semivariogram models was completed based on the lowest RMSE obtained from leave-one-out cross-validation. Exponential semivariograms, followed by spherical and Gaussian, gave the lowest RMSE compared with the others. The average RMSE values reported for exponential, spherical, and Gaussian semivariograms were 0.026, 0.030, and 0.030, respectively. The nugget, sill, and range parameters were estimated individually for each band while executing the kriging. For example, the ordinary kriging was carried out for band 10 using an exponential variogram model selected based on the lowest RMSE of 0.008 and MSE of 0.004. Here, the corresponding variogram parameters were a nugget of 0.000046, a major range of 0.0012, and a partial sill of 0.000079.

3.3. Model Evaluation and SOC Mapping

SOC prediction models of SVM, ANN, and PLSR were constructed using PLS components as the explanatory variables, and their accuracy was evaluated using five-fold cross-validation. The prediction accuracy was evaluated using multiple values of the number of PLS components to determine the optimal setting. The R² shows an increase and becomes stable at 40 PLS components (as shown in Figure 4a), which was selected as the optimal number for prediction model development. The scatterplots of the predicted and measured SOC values, along with goodness-of-fit statistics, are shown in Figure 4b1–b3. The SVM shows the best fit and higher accuracy, with R² = 0.83, RMSE = 0.02, MAE = 0.02, and RPD = 2.41, followed by PLSR and ANN. Through GridSearchCV, the optimal parameters chosen for SVM were kernel: linear, C: 0.1, and gamma: scale. For the ANN, the optimal parameters were batch size: 32, epochs: 200, learning rate: 0.005, and number of hidden layers: 4. The latent component count for PLSR was also set to 4. The best-performing SVM model was applied to synthetic imagery to generate a spatial map of SOC, as shown in Figure 5. The predicted SOC values exhibit variability across the field, indicating comparable results with the measured values.

4. Discussion

The superior performance shown by the optimized SVM model indicates an acceptable operational delivery of SOC maps at the field scale by using proximally collected spectral values. The outperformance of SVM in SOC prediction using hyperspectral data over other multivariate traditional models has been previously reported for hyperspectral data collected using proximal, airborne, and satellite spectroscopy [13,14,15,16]. The outperformance of SVM could be attributed to the model’s ability to manage the nonlinear relationships between SOC and spectral ranges [17]. Moreover, it was reported that, along with N, the carbon content can also be accurately predicted using spectral bands due to the presence of various overtones and blends of chemical bonds arising from carbon–carbon and carbon–hydrogen bonds [18,19]. A Variable Importance in Projection (VIP) plot (as shown in Figure 6) was generated to quantify the relative contribution of each wavelength across all 40 PLS latent variables calculated from the measured spectra. Wavelengths with VIP scores greater than 1 are considered the most influential for SOC prediction. The important bands obtained in the range of 400–900 nm are due to the presence of iron oxides (goethite, hematite), bands located around 1330 nm are associated with C-H bonds, around 2015 belong to C=O bonds, and more than 2200 nm are related to aliphatic C-H, C-O, C-N, etc. [20,21]. Most of the bands showing VIP scores > 1 closely coincide with previous findings [4,13].

The spatial variability of soil parameters depends on underlying materials, surface accumulations, and cultivation and input practices [13]. The present approach of generating synthetic hyperspectral data for soil OC mapping was tested on one field site with a limited number of samples, denoted by 16% of CV. A relevant follow-up study is needed to test this approach in multiple fields, covering larger sampling points to ensure transferability [21]. Beyond these three evaluated models, more machine learning models and deep learning frameworks, in combination with advanced feature selection techniques, can be considered in the future to attain higher accuracy [22]. As a continuation of the present work, synthetic imagery of mid-infrared spectral regions collected through Fourier Transform Infrared Spectroscopy will be employed in future for the more accurate prediction of soil fertility parameters [23]. At the same time, comparative studies with different imaging spectroscopic technologies, such as space and airborne hyperspectral missions, will also be considered in the future. Another critical point to consider is the application of a similar approach to other important soil fertility parameters, including micronutrients [24].

5. Conclusions

Synthetic hyperspectral imagery was generated using an ordinary kriging interpolation technique applied to non-imaging proximal spectroscopy collected from the field. Upon evaluating three machine learning models, the support vector machine demonstrated better accuracy (R² value of 0.83) in predicting soil organic content. A high-resolution SOC map was also generated by applying the best prediction model to the synthetic imagery. The present work has opened up new research practices for the near-real-time delivery of accurate soil fertility maps using non-imaging spectral information.