Randomness in Data Partitioning and Its Impact on Digital Soil Mapping Accuracy: A Comparison of Cross-Validation and Split-Sample Approaches

Abstract

Digital soil mapping has become increasingly important for large-scale soil organic carbon (SOC) assessments, yet the choice of accuracy assessment method significantly influences model performance interpretation. This study investigates the impact of cross-validation fold numbers on accuracy metrics and compares cross-validation with split-sample validation approaches in national-scale SOC mapping. Five machine learning algorithms (Random Forest, Cubist, Support Vector Regression, Bayesian Regularized Neural Networks, and ensemble modeling) were evaluated to predict SOC content across France (539,661 km²) and Czechia (78,873 km²) using 2731 and 445 soil samples, respectively. Environmental covariates included satellite imagery (Sentinel-1, Sentinel-2, and MODIS), climate data (CHELSA), and topographic variables. Four cross-validation approaches (k = 2, 4, 5, 10) were utilized with 100 repetitions each and the results were compared with the existing literature using both cross-validation and split-sample methods. Ensemble models consistently produced the highest prediction accuracy and lowest variance per fold across all validation approaches. Higher fold numbers (k = 10) also produced higher accuracy estimates compared to lower folds (k = 2) and had the greatest value ranges of accuracy assessment metrics. This confirmed the observations from previous studies, in which split-sample validation reported higher R² values (0.10–0.90) compared to cross-validation studies (0.03–0.68), suggesting a strong effect of randomness in training and test data split in the split-sample approach. This suggests that k-fold cross-validation should preferably be used in reporting prediction accuracy in similar studies, with the split-sample approach being strongly affected by value properties from training and test data from particular splits used for validation.

1. Introduction

Digital soil mapping revolutionized the ability to quantify and map soil properties at various spatial scales, from local to global extents, fundamentally transforming traditional soil survey approaches that relied on limited point observations and expert knowledge [1]. Soil organic carbon (SOC), as one of the most critical soil properties influencing fertility, water retention, and nutrient cycling has received particular attention in digital soil mapping applications due to its fundamental role in ecosystem functioning and global carbon cycling [2]. The modern digital soil mapping framework utilizes extensive environmental covariates derived from remote sensing imagery, climate datasets, topographic analyses, and geological information to establish quantitative relationships between soil properties and environmental factors [3,4]. Satellite-based remote sensing data, including multispectral imagery from platforms such as Landsat, Sentinel-2, and MODIS, provide crucial information about vegetation patterns, land use dynamics, and surface reflectance characteristics that correlate with soil properties [5,6,7]. Climate variables, including temperature, precipitation, and derived indices such as aridity and seasonality metrics, capture the environmental drivers of pedogenesis and organic matter accumulation [8,9]. Topographic derivatives from digital elevation models, such as slope, aspect, curvature, and compound topographic indices, quantify the influence of terrain on soil formation processes, water flow patterns, and erosion–deposition cycles [10]. The integration of machine learning algorithms with these extensive environmental datasets has enabled moderately high prediction accuracy in SOC mapping across large spatial domains, especially for national- and regional-scale applications [11]. Machine learning algorithms, and more recently, deep learning, have demonstrated superior performance compared to traditional geostatistical methods by capturing complex non-linear relationships between environmental covariates and soil properties [12,13]. Among these methods, Random Forest (RF), Cubist (CUB), Support Vector Regression (SVR), and Bayesian Regularized Neural Networks (BRNNs) were frequently used for digital SOC mapping, achieving high prediction accuracy due to their ability to model non-linear relationships between SOC and environmental covariates [14,15]. Ensemble methods that combine multiple algorithms have shown particular promise in improving prediction robustness and reducing uncertainty in digital soil mapping applications [16], particularly when these models are constructed from methods with complementary properties.

Despite these advances of technology, the reliability and comparability of digital soil mapping studies is fundamentally subject to the assessment of accuracy used, which varies considerably in the literature (

Table 1. Accuracy assessment approaches and statistical metrics from recent digital SOC mapping studies.

Country	Soil Samples	Sampling Depth (cm)	Sampling Density (km2 per Sample)	Mean SOC	Accuracy Assessment
					Method	R2	RMSE	Lowest NRMSE	Reference
China	313	0–20	7987.22	4.85 kg C m−2	CV	0.33–0.42	2.64–2.84	0.54	[26]
USA	6213	0–30	1582.81	94.90 mg·ha−1	SS	0.38–0.53	0.51–0.54	0.01	[27]
China	8021	0–5	1196.48	19.09 g·kg−1	CV	0.06–0.42	0.47–1.11	0.02
		5–15		17.22 g·kg−1		0.03–0.46	0.45–0.99	0.03	[28]
		15–30		12.84 g·kg−1		0.03–0.40	0.51–1.16	0.04
China	644	0–30	1009.32	3.86 kg·m−2	CV	0.28–0.40	5.76–11.76	1.49	[21]
USA	673	0–15	641.90	1.16%	CV	0.51	1.69	1.46	[29]
	630	15–30	685.71	0.85%		0.39	1.48	1.74
China	105	0–10	480.00	13.42 g·kg−1	CV	0.30–0.43	5.28–8.51	0.39	[30]
China	23,103	0–800	415.39	3.85 kg·m–2	CV	0.83	1.93	0.50	[31]
Colombia	653	0–30	398.16	15.00 g·kg−1	CV	0.50	0.46	0.03	[32]
Switzerland	150	0–20	273.33	43.93 g·kg−1	CV	0.12–0.47	0.44–0.56	0.01	[33]
China	733	0–20	191.00	13.11 g·kg−1	CV	0.10–0.60	4.50–6.00	0.34	[34]
Dominican Republic	268	0–15	179.84	110.35 mg· ha−1	SS	0.77–0.83	35.00–38.60	0.32	[35]
Brazil	81	0–20	178.56	17.6 g·kg–1	CV	0.04–0.51	1.75–3.02	0.10	[36]
Germany	475	0–30	149.47	2.63%	CV	0.42–0.68	1.42–1.60	0.54	[19]
			75.79	1.74%		0.30–0.48	1.37–1.44	0.79
Germany	3104	0–30	115.21	28.00 g·kg−1	CV	/	21.00–34.00	0.75	[20]
Italy	414	0–30	60.39	1.51%	SS	/	0.70	0.46	[37]
Iran	201	0–20	24.02	0.32%	CV	0.41–0.54	0.08–0.18	0.25	[38]
China	308	0–20	18.08	12.62 g·kg−1	CV	0.23–0.35	5.00–5.50	0.40	[39]
China	186	0–20	14.34	23.78 g·kg−1	SS	0.11–0.49	3.90–5.28	0.16	[40]
China	396	0–10	9.92	12.56 g·kg−1	SS	0.46–0.58	3.49–3.83	0.28
		10–20		10.11 g·kg−1		0.63–0.71	3.49–3.60	0.35	[39]
		20–30		7.58 g·kg−1		0.67–0.73	2.95–3.03	0.39
Iran	180	0–10	8.33	0.86%	SS	0.86	0.24	0.28	[41]
China	395	0–20	6.64	11.60 mg·kg−1	SS	0.32–0.42	1.67–1.90	0.14	[42]
Switzerland	2071	0–5	6.28	6.05%	SS	0.10–0.22	4.85–5.16	0.80
		5–15		3.66%		0.21–0.29	3.78–4.00	1.03	[43]
		15–30		2.20%		0.23–0.32	2.83–3.03	1.29
China	181	0–15	4.08	1.70%	SS	0.20–0.56	0.20–0.26	0.12	[44]
				1.03%		0.19–0.53	0.25–0.33	0.24
India	83	0–30	3.73	26.48 mg·ha−1	SS	0.90	8.21	0.31	[45]
France	162	0–30	2.06	8.9 g·kg–1	CV	0.36–0.73	24.8	2.7	[46]

CV: cross-validation, SS: split-sample, R²: coefficient of determination, RMSE: root mean square error, NRMSE: normalized RMSE.

). The selection of different validation strategies, specifically, cross-validation (CV) or split-sample (SS) methods, and the choices of cross-validation parameters like number of folds can have strong effects on the interpretation of model performance. CV has become the most dominant method of assessing accuracy in digital soil mapping due to its efficiency in using all available data to train and validate an accuracy assessment [17]. The k-fold cross-validation approach, where data are partitioned into k subsets with each subset serving as validation data while the remaining k − 1 subsets are used for training, was frequently used in previous studies [18,19,20,21]. However, the selection of k values varies considerably across studies, ranging from 2-fold to 10-fold or even leave-one-out cross-validation, with limited systematic investigation of how this choice affects accuracy metrics. SS validation, which involves randomly dividing the dataset into independent training and testing sets (typically 70:30 or 80:20 ratios), represents a simpler approach that may provide more conservative accuracy estimates. The fundamental difference between CV and SS lies in data utilization, as CV uses all data points for both training and validation during iterative split in k folds, while SS reserves a portion of data exclusively for validation [22,23]. If the split is biased, it may result in misleading results of the accuracy if any types of soil are overrepresented in the training dataset and undersrepresented in the testing set. Such an approach can produce very high variation in accuracy assessment metrics per split depending on the properties of input data [24,25]. Therefore, it is possible that the model works very well on the training data, but not very well on unseen data, producing an exaggerated estimate of accuracy. For this reason, a single split might not necessarily provide the overall effectiveness of the model across various scenarios, thus the need for numerous iterations resulting in a higher-reliability estimate.

Furthermore, there is much variation in the choice of accuracy metrics across the studies reviewed, as different emphasis has been placed upon the coefficient of determination (R²), root mean square error (RMSE), mean absolute error (MAE), and correlation coefficients. Some of the studies have multiple measures and some have single indicators, making direct comparisons difficult, for which reason a normalized RMSE (NRMSE) was calculated for each of the papers listed in Table 1, enabling a cross-study analysis of achieved absolute accuracy levels. The spatial scale of analysis also affects the assessment of accuracy, poined for local-scale studies to be more accurate due to a lower environmental variability, and for regional-scale or global application, with increasing heterogeneity affecting the model’s capability to perform correctly. The implications of choosing validation methods go beyond the individual studies to the knowledge of the capabilities of digital soil mapping. Systematic biases triggered by particular validation approaches can result in overly optimistic or pessimistic judgments on model performance and have an impact on decision making in agricultural management and environmental policy.

This variation in methodology has produced considerable cause for difficulty in comparing the results produced by previous studies, and a general lack of suitable benchmarks for establishing realistic levels of digital soil mapping accuracy. Therefore, the research hypothesis is that randomness in data partitioning strongly influences national-scale SOC estimates, limiting the reproducibility of digital soil mapping studies. To evaluate this hypothesis, the main aims of this research were to (1) systematically analyze the influence of the k (2, 4, 5, 10 folds) of cross-validation on digital soil mapping accuracy metrics using these identical datasets and modeling approaches relative to split-sample approach; and (2) assess the robustness and validation effect of different machine learning algorithms under varying validation scenarios to evaluate the consistency of regional validation effects of different algorithms.

2. Materials and Methods

The workflow of the study consisted of four main steps: (1) preprocessing of SOC samples used for digital soil mapping; (2) modeling and preprocessing of environmental covariates relevant for SOC prediction; (3) machine learning prediction of SOC levels; and (4) an accuracy assessment based on the k-fold cross-validation approach in multiple iterations to determine the influence of the k (2, 4, 5, 10 folds) of cross-validation on digital soil mapping accuracy metrics and randomness effect of individual split-sample components.

2.1. Study Area and SOC Sampling Data

This study area included two European countries representing diverse environmental conditions, including mainland France and Czechia, covering 539,661 km² and 78,873 km², respectively (Figure 1). Mainland France has a diverse topography that comprises wide lowland plains in the north and the west, the central Massif Central highland regions, and the high mountain ranges in the south and east. Conversely, the Czech Republic is a landlocked country located in Central Europe, whose topography is made up of upland plateaus and basins with mountain ranges. According to the Köppen–Geiger climate classification [47], France’s mainland possesses a wide range of climates, with temperate oceanic (Cfb) as a major climate class, Mediterranean (Csa), and humid subtropical climate (Cfa) being prevalent in its southern region, and several minor climate classes present in its highlands, including humid continental (Dfb) and alpine (Dfc, ET) climates. By contrast, the Czech Republic is more climatically homogeneous, being dominated by the humid continental (Dfb) class. SOC is directly linked to soil genesis, and this relationship can also be expressed through the formula “S = f(c, o, p)t,”, where “S” represents soil as a function of the factors “c” (climate), “o” (organisms), and “p” (parent material) that interact over time (t) to determine its formation [48].

Soil organic carbon data for France consisted of 2731 samples, resulting in a sampling density of 197.6 km² per sample, while the Czechia dataset included 445 samples with a density of 177.2 km² per sample. The 2018 LUCAS (Land Use/Cover Area Frame Statistical survey) SOC dataset [49], created by the Joint Research Centre of the European Commission, was used as a data source for soil samples for the study. Soil samples taken at a depth of 0–20 cm and expressed in g kg^–1 were selected to match the geographic extent of the study area. Preprocessing of input soil samples included the removal of outliers using the interquartile range (IQR) approach. All soil samples with SOC values below 1.5 IQR of the first and above 1.5 IQR of the third quartile were identified as outliers using the boxplot approach, following the conventional rule.

2.2. Environmental Covariates Used for Digital SOC Mapping

Environmental covariates were modeled from three main data sources representing abiotic conditions which correlated with SOC levels (Table 2), including climate, topographic, and remote sensing covariates [50]. All environmental covariates were processed to a 1000 m spatial resolution using bilinear resampling and projected to the European Terrestrial Reference System (EPSG:3035) for spatial consistency. Covariate values were extracted at soil sampling locations using point extraction methods to create the modeling dataset. Multicollinearity analysis was performed using variance inflation factor (VIF) assessment, with variables exhibiting VIF > 10 removed to prevent numerical instability in machine learning algorithms. Recursive feature elimination (RFE) was conducted using Random Forest importance measures to identify optimal variable subsets for each study area.

Climate covariates were obtained from the climatologies at high resolutions for the Earth’s land surface areas (CHELSA) dataset, providing 1981–2010 climatological data at approximately 1 km spatial resolution [51]. These included 19 bioclimatic variables which quantified annual trends of air temperature and precipitation, their seasonality and extreme values relative to monthly and quarterly averages. Topographic covariates were derived from the Shuttle Radar Topography Mission (SRTM) 90 m digital elevation model (DEM) [52], which consisted of elevation data and derived properties, including slope, aspect, flow direction, terrain ruggedness index, topographic position index, and surface roughness calculated using neighborhood analysis with 8-connected pixels [53]. From remote sensing covariates, Sentinel-1 Synthetic Aperture Radar (SAR) data included VV and VH polarization backscatter coefficients, which represented surface roughness, moisture content, and vegetation structure independent of cloud cover. Twelve bands from multispectral Sentinel-2 satellite missions included reflectance from visible, near-infrared, and short-wave bands, based on scenes with below 20% cloud coverage. Nine spectral indices were additionally calculated from derived annual band medians, including vegetation, soil, and moisture indices.

Table 2. Spectral indices used as environmental covariates for SOC prediction in the study.

Spectral Index	Equation	Reference
Normalized Difference Vegetation Index (NDVI)	$\mathrm{NDVI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}4}{\mathrm{B}8+\mathrm{B}4}}$	[54]
Green Normalized Difference Vegetation Index (GNDVI)	$\mathrm{GNDVI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}3}{\mathrm{B}8+\mathrm{B}3}}$	[55]
Enhanced Vegetation Index (EVI)	$\mathrm{EVI}=2.5·{\displaystyle \frac{\mathrm{B}8 - \mathrm{B}4}{\mathrm{B}8+6·\mathrm{B}4-7.5·\mathrm{B}2+1}}$	[56]
Soil Adjusted Vegetation Index (SAVI)	$\mathrm{SAVI}= {\displaystyle \frac{\mathrm{B}8-\mathrm{B}4}{(\mathrm{B}8+\mathrm{B}4+0.5)·1.5}}$	[57]
Normalized Difference Moisture Index (NDMI)	$\mathrm{NDMI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}11}{\mathrm{B}8+\mathrm{B}11}}$	[58]
Moisture Stress Index (MSI)	$\mathrm{MSI}= {\displaystyle \frac{\mathrm{B}11}{\mathrm{B}8}}$	[59]
Green Chlorophyll Index (GCI)	$\mathrm{GCI}= {\displaystyle \frac{\mathrm{B}9}{\mathrm{B}3}}-1$	[60]
Bare Soil Index (BSI)	$\mathrm{BSI}= {\displaystyle \frac{(\mathrm{B}11+\mathrm{B}4)-(\mathrm{B}8+\mathrm{B}2)}{(\mathrm{B}11+\mathrm{B}4) + (\mathrm{B}8+\mathrm{B}2)}}$	[61]
Normalized Difference Water Index (NDWI)	$\mathrm{NDWI}= {\displaystyle \frac{\mathrm{B}3 - \mathrm{B}8}{\mathrm{B}3+\mathrm{B}8}}$	[62]

B3: Reflectance in green band of Sentinel-2 (560 nm), B4: reflectance in red band of Sentinel-2 (665 nm), B8: reflectance in near-infrared band of Sentinel-2 (842 nm), B9: reflectance in near-infrared band of Sentinel-2 (945 nm), B11: reflectance in shortwave infrared band of Sentinel-2 (1610 nm).

Table 2. Spectral indices used as environmental covariates for SOC prediction in the study.

Spectral Index	Equation	Reference
Normalized Difference Vegetation Index (NDVI)	$\mathrm{NDVI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}4}{\mathrm{B}8+\mathrm{B}4}}$	[54]
Green Normalized Difference Vegetation Index (GNDVI)	$\mathrm{GNDVI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}3}{\mathrm{B}8+\mathrm{B}3}}$	[55]
Enhanced Vegetation Index (EVI)	$\mathrm{EVI}=2.5·{\displaystyle \frac{\mathrm{B}8 - \mathrm{B}4}{\mathrm{B}8+6·\mathrm{B}4-7.5·\mathrm{B}2+1}}$	[56]
Soil Adjusted Vegetation Index (SAVI)	$\mathrm{SAVI}= {\displaystyle \frac{\mathrm{B}8-\mathrm{B}4}{(\mathrm{B}8+\mathrm{B}4+0.5)·1.5}}$	[57]
Normalized Difference Moisture Index (NDMI)	$\mathrm{NDMI}= {\displaystyle \frac{\mathrm{B}8 - \mathrm{B}11}{\mathrm{B}8+\mathrm{B}11}}$	[58]
Moisture Stress Index (MSI)	$\mathrm{MSI}= {\displaystyle \frac{\mathrm{B}11}{\mathrm{B}8}}$	[59]
Green Chlorophyll Index (GCI)	$\mathrm{GCI}= {\displaystyle \frac{\mathrm{B}9}{\mathrm{B}3}}-1$	[60]
Bare Soil Index (BSI)	$\mathrm{BSI}= {\displaystyle \frac{(\mathrm{B}11+\mathrm{B}4)-(\mathrm{B}8+\mathrm{B}2)}{(\mathrm{B}11+\mathrm{B}4) + (\mathrm{B}8+\mathrm{B}2)}}$	[61]
Normalized Difference Water Index (NDWI)	$\mathrm{NDWI}= {\displaystyle \frac{\mathrm{B}3 - \mathrm{B}8}{\mathrm{B}3+\mathrm{B}8}}$	[62]

B3: Reflectance in green band of Sentinel-2 (560 nm), B4: reflectance in red band of Sentinel-2 (665 nm), B8: reflectance in near-infrared band of Sentinel-2 (842 nm), B9: reflectance in near-infrared band of Sentinel-2 (945 nm), B11: reflectance in shortwave infrared band of Sentinel-2 (1610 nm).

Additionally, MODIS satellite products provided derived spectral information (Justice et al., 2002) [63], including surface reflectance data (MOD09A1, MODOCGA) covering 16 spectral bands, land surface temperature data (MOD11A1) for day and night observations, land cover classification (MCD12Q1), leaf area index and fraction of photosynthetically active radiation (MOD15A2H), evapotranspiration components (MOD16A2/105), gross primary productivity (MOD17A2H), and phenological metrics (MCD12Q2).

2.3. Ensemble Machine Learning Prediction of SOC Levels

Ensemble machine learning methods involved the combination of regression algorithms with different prediction principles, including RF, CUB, SVR, and BRNN, whose predictions were aggregated using a generalized linear model. The resulting ensemble provided a theoretical foundation for a better generalization ability, less prediction variance, and increased robustness to variances in data distribution than any single constituting algorithm [64]. Moreover, previous studies suggested that this approach is especially useful in complex regression tasks in which there is no single model that provides the highest prediction accuracy in all parts of the feature space [65,66]. The ensemble models were constructed using the stacking approach, with four base learners being trained independently. These predictions were used as input features of a generalized linear model meta-learner that used the base models for prediction, minimizing the out-of-fold prediction error by assigning higher weight to the better performing models, causing less bias from each individual model. Hyperparameter optimization was performed using a random grid search within each cross-validation fold in 10 instances per model to prevent data leakage and ensure a more robust comparison across validation approaches. A variable importance assessment was conducted for each algorithm to identify key predictors based on the model-agnostic permutation feature importance.

RF regression was based on a series of decision trees to make strong and effective predictions based on the concepts of bootstrap aggregating and random feature selection [67]. The algorithm was based on building a fixed size of regression trees, each of which is trained on a bootstrap sample of the original dataset, randomizing both samples and features in the process [68]. During tree construction, at each internal node, randomly chosen “mtry” hyperparameter features out of the available total number of predictors are used and the best split is selected only between these candidate variables [69]. The overall prediction was achieved by averaging the results of all constituent trees, effectively decreasing the variance of predictions. The random selection of features controlled by the “mtry” hyperparameter allowed the forest to retain both predictive power and generalization ability even in situations where highly correlated or irrelevant features exist, and thus it is especially useful when using high-dimensional datasets where traditional regression models can encounter challenges of multicollinearity or overfitting.

CUB regression is a rule-based algorithm that uses model trees with instance-based corrections to provide predictions [70]. The algorithm built a sequence of model trees, with the leaf node of each tree holding a linear regression model instead of a predictor constant, to allow for the identification of local linear relationships in different parts of the feature space. The “committees” hyperparameter defined how many model trees were produced by boosting, with each successive committee using the residual errors of the preceding ensemble, which becomes more accurate and prevents overfitting by adaptive reweighting of training an instance [71]. The resulting prediction tree was then narrowed down by the “neighbors” hyperparameter, which defined how many closest training cases were used for rule-based prediction to make local modifications [71]. This instance-based correction system then found the k-most similar cases in the training set and averaged their weighted residuals, presenting a combination of the global patterns of the rule-based models with the local neighborhood information.

SVR was based on the principles of support vector machines applied to regression tasks by projecting input features in a high-dimensional feature space in which the relations could be represented in linear form [72]. The algorithm was built by determining a small set of training examples, known as support vectors, which determined the optimal hyperplane that best approximated the target function. The radial basis function (RBF) kernel, optimized by hyperparameter “σ”, regulating the radius of influence of individual training samples, where larger values generated smooth, more generalized predictions [73]. Regularization hyperparameter “C” controlled the trade-off between the model’s complexity and tolerance to prediction errors, with larger C values resulting in models with a higher model complexity, but with poorer generalization properties [73]. Prediction was based on the evaluation of the kernel function between the test instance and each of the support vectors were weighted by their addressed Lagrange multipliers to generate a continuous output [74].

The BRNN combined the flexibility of multilayer perceptron with the Bayesian principle of inference to overcome overfitting and provide a measure of uncertainty in predictions [75]. This algorithm used a three-layer architecture, with an input layer, a hidden layer, where the number of neurons is specified, and a final output layer, with the network weights being random variables who have prior probability distributions instead of fixed parameters. The number of hidden units was defined by a “neurons” hyperparameter to learn complex non-linear relationships in data, and there is a tradeoff between the complexity of the model and the ability to generalize [76]. The Bayesian framework also regulated the network automatically during the training process by integrating prior calculations concerning weight distributions, and updating them using the Bayes theorem, which can avoid overfitting without explicitly specifying validation sets to be used to early terminate the training process. Prediction was performed by marginalizing the posterior distribution of network weights, which does not only generate point prediction but also generates uncertainty estimates.

2.4. Accuracy Assessment of Predicted SOC Levels per Data Fold

To evaluate the impact of fold numbers on the accuracy assessment of predicted SOC levels, four k-fold cross-validation approaches were implemented with varying levels of data partitioning. Based on evaluated k-fold numbers (k = 2, 4, 5, 10), the input dataset was split into k equal parts, with each part serving as validation data in a single fold and as a part of training data in the remaining folds. To ensure robust statistical estimates and account for variability inherent in random data splitting procedures, each cross-validation approach was repeated 100 times using different random partitions which produced a significant randomness effect on the creation of training and validation datasets [77]. Accuracy assessment metrics from individual folds were saved for each repetition, simulating training and validation data obtained by a single split-sample procedure. The selected k-fold number thus covered frequently used ratios for training and validation dataset creation using the split-sample approach, with k = 2, 4, 5, 10 representing 50:50, 75:25, 80:20, and 90:10 ratios, respectively [78].

Model performance evaluation was conducted using four complementary accuracy metrics, including the coefficient of determination (R²), root mean square error (RMSE), normalized RMSE (NRMSE), and mean absolute error (MAE). R² quantified the proportion of variance explained by the model, serving as a relative indicator of overall predictive capability and model fit quality. RMSE measured the average magnitude of prediction errors expressed in the original units of measurement, with NRMSE enabling scale-independent comparison across different datasets or variables. Additionally, MAE was computed as the average absolute prediction error, offering a metric less sensitive to outliers compared to RMSE, thereby providing a more robust assessment of typical prediction accuracy [79]. These metrics were calculated according to Equations (1)–(4):

$$R^2 = 1-{\displaystyle \frac{\sum _{i = 1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}{\sum _{i = 1}^n{\left(y_i-\overline{y}\right)}^2}},$$

(1)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2},$$

(2)

$$NRMSE={\displaystyle \frac{RMSE}{\overline{y}}},$$

(3)

$$MAE={\displaystyle \frac{1}{n}}\sum _{i=1}^n\left|y_i-{\widehat{y}}_i\right| ,$$

(4)

where $y_i$—measured SOC levels from the LUCAS 2018 dataset; ${\stackrel{\wedge }{y}}_i$—predicted SOC levels; $\overline{y}$—mean of measured SOC levels from the LUCAS 2018 dataset; and n—sample count. The higher R² and lower RMSE, NRMSE, and MAE indicated higher prediction accuracy.

3. Results

3.1. Descriptive Statistics of Used SOC Samples

The median SOC levels in the French and Czech soils were similar (21.5 g·kg^–1 and 19.0 g·kg^–1, respectively), with both study areas producing high coefficients of variation of 1.695 and 2.266, respectively (Figure 2). This observation is contrary to variability in soil types in France and Czechia, since French soils were much more heterogeneous regarding soil types. According to the Soil Geographical Database of Europe [80] and World Reference Base Soil Classification of the Food and Agriculture Organization of the United Nations [81], Dystric Cambisol (17.7%), Orthic Rendzina (13.7%), and Orthic Luvisol (11.1%) were the most representative soil classes in terms of LUCAS 2018 soil samples count. Notably higher mean SOC values than the overall average were observed for samples collected in Dystric Cambisol (36.5 g·kg^–1) and Orthic Rendzina (31.9 g·kg^–1), while Orthic Luvisol soils produced below average SOC levels (18.2 g·kg^–1). Unlike France, which had 55 soil types with at least one soil sample, Czech soils were more homogeneous, Dystric Cambisol soils being dominant, on which 51.2% of all used soil samples were collected. While the same soil class had the highest representation in collected soil samples in both France and Czechia, its mean SOC value was lower in Czechia (29.7 g·kg^–1), with the same applying for the second largest class of Orthic Luvisol soils, with a mean SOC of 17.6 g·kg^–1.

3.2. Accuracy Assessment of Predicted SOC Levels Based on Aggregated Metrics from k-Fold Cross-Validation

The accuracy assessment of evaluated machine learning models produced a notable difference between the prediction accuracy results in France and Czechia (

Table 3. Cross-validation accuracy assessment results for SOC prediction in France and Czechia based on k-fold variation.

Cross-Validation Approach	Machine Learning Method	France				Czechia
		R2	RMSE	NRMSE	MAE	R2	RMSE	NRMSE	MAE
k = 10	RF	0.409	11.60	0.454	8.64	0.243	8.21	0.387	6.25
	CUB	0.373	12.01	0.470	8.76	0.197	8.56	0.403	6.39
	SVR	0.392	11.94	0.467	8.42	0.228	8.44	0.398	6.09
	BRNN	0.382	11.87	0.464	8.83	0.236	8.27	0.390	6.27
	ENS	0.412	11.56	0.452	8.49	0.229	8.21	0.387	6.15
k = 5	RF	0.406	11.63	0.455	8.67	0.227	8.27	0.390	6.27
	CUB	0.371	12.04	0.471	8.79	0.180	8.65	0.407	6.43
	SVR	0.387	11.98	0.469	8.45	0.218	8.47	0.399	6.07
	BRNN	0.378	11.90	0.466	8.84	0.220	8.34	0.393	6.28
	ENS	0.409	11.58	0.453	8.52	0.225	8.24	0.388	6.16
k = 4	RF	0.404	11.65	0.456	8.69	0.224	8.28	0.390	6.26
	CUB	0.369	12.05	0.472	8.81	0.186	8.61	0.406	6.40
	SVR	0.385	12.01	0.470	8.47	0.213	8.49	0.400	6.07
	BRNN	0.378	11.90	0.466	8.85	0.211	8.39	0.395	6.31
	ENS	0.408	11.60	0.454	8.53	0.223	8.25	0.389	6.16
k = 2	RF	0.392	11.77	0.461	8.80	0.206	8.38	0.395	6.32
	CUB	0.372	12.15	0.476	8.67	0.174	8.72	0.411	6.30
	SVR	0.374	12.11	0.474	8.56	0.206	8.53	0.402	6.08
	BRNN	0.369	11.99	0.469	8.91	0.192	8.52	0.402	6.41
	ENS	0.398	11.70	0.458	8.64	0.211	8.31	0.392	6.21

Statistical metrics indicating the highest prediction accuracy per k-fold number and study area are bolded.

). The optimal hyperparameters of all evaluated models are presented in

Table 4. Optimal hyperparameters for all evaluated machine learning models for SOC prediction.

Cross-Validation Approach	Machine Learning Method	Optimal Hyperparameters
		France	Czechia
k = 10	RF	mtry = 10	mtry = 5
	CUB	committees = 20, neighbors = 9	committees = 20, neighbors = 9
	SVR	σ = 0.020, C = 1	σ = 0.019, C = 0.5
	BRNN	neurons = 2	neurons = 1
k = 5	RF	mtry = 10	mtry = 5
	CUB	committees = 20, neighbors = 9	committees = 20, neighbors = 9
	SVR	σ = 0.022, C = 1	σ = 0.019, C = 0.5
	BRNN	neurons = 2	neurons = 1
k = 4	RF	mtry = 6	mtry = 5
	CUB	committees = 20, neighbors = 9	committees = 20, neighbors = 9
	SVR	σ = 0.021, C = 1	σ = 0.019, C = 0.5
	BRNN	neurons = 2	neurons = 1
k = 2	RF	mtry = 6	mtry = 2
	CUB	committees = 20, neighbors = 0	committees = 20, neighbors = 0
	SVR	σ = 0.022, C = 1	σ = 0.017, C = 0.5
	BRNN	neurons = 2	neurons = 1

. The ensemble approach proved to be the most accurate for SOC prediction in France with k = 10 cross-validation (R² = 0.412, RMSE = 11.56 g·kg^–1, NRMSE = 0.452). All evaluated k-fold numbers agreed on the optimal model, with a minor difference in obtained accuracy assessment metrics, as 10-fold cross-validation produced the most accurate results and 2-fold cross-validation suggested lower accuracy across all four metrics and both study areas. Among the evaluated machine learning methods, ENS produced superior accuracy to individual models according to RMSE and NRMSE, while RF produced the highest R² for k = 4, 5, and 10. For individual machine learning models, RF produced the highest overall prediction accuracy, followed by SVR in terms of relative prediction accuracy quantified by R² and BRNNs in terms of absolute prediction accuracy according to RMSE and NRMSE.

3.3. Stability of Accuracy Assessment Metrics According to Randomness in Training and Validation Dataset Creation per Fold

The cross-validation results per fold indicate a strong relationship between k-fold numbers and the variability of all evaluated statistical metrics (Figure 3). In both study areas, 10-fold cross-validation produced the largest variability in all the accuracy measures, indicated by the largest interquartile ranges and the broadest outlier distributions, due to the largest discrepancy between quantity of training and validation data. This variability tended to diminish as the k-fold number reduced down to two, with median accuracy levels also being very slightly reduced [19,21,43,82,83]. The same effect was observed in final model fit accuracy assessment metrics, which were all very stable regardless of variations in k values (Figure 4).

3.4. Stability of Variable Importances According to Number of Folds in Cross-Validation [84,85]

The selection of k-fold values had a negligible effect on the variable importances of the most accurate evaluated machine learning model for SOC prediction (Figure 5). The DEM is consistently the most important variable in France, with changes in relief, slope, and topographic position being key drivers of SOC distribution. Following the DEM, other significant variables include land cover data, followed by MODIS satellite data (B10, B11) and climatic data (CHELSA). In the case of the Czech Republic, the most important predictors are vegetation indices, respectively, the GCI and BSI, followed by the Leaf Area Index (LAI) and MODIS satellite data (B4). Interestingly, the climatic variables from the CHELSA dataset, which were more important in France, have a lower relative importance here.

4. Discussion

4.1. Evaluation of Machine Learning Prediction Accuracy According to Input Data Properties

The variability in input dataset tends to produce less robust prediction accuracy depending on value properties of training data, as well as to hinder generalization in the initial stages of machine learning [86,87]. Digital soil mapping on a national scale is often subjected to highly heterogeneous soil sampling values [88], which suggests that the randomness in training and validation data split during the accuracy assessment tends to have a larger impact on comparison, less variable landscape, such as in-field predictions.

The lower prediction accuracy in Czechia compared to France according to R² is concurrent with the increased coefficient of variation in Czech SOC data, indicating that the strong spatial heterogeneity of Czech soils prevented evaluated machine learning models from sufficiently explaining the proportion on variance in SOC values based on the used environmental covariates. However, the absolute accuracy assessment metrics were in disagreement with R², producing NRMSE down to 0.387, since Czechia had a smaller range of extreme SOC values in the input LUCAS 2018 soil samples, leading to smaller absolute errors but not explaining much variance in SOC values. This disagreement suggests that multiple statistical metrics are needed to evaluate the prediction accuracy of machine learning models across regions with different data characteristics, thus confirming the observations of previous studies [89,90]. The likely reason for the reduced accuracy of 2-fold cross-validation is the lower quantity of training data relative to other k-fold numbers and exaggerated sensitivity to randomness in data splitting [91]. Among the evaluated methods, SVR consistently produced the lowest MAE for all evaluated models and k-fold variations, suggesting that it produced the lowest number of smaller residuals during prediction [79] and thus provides complementary prediction properties to other methods in the ensemble.

In most cases from previous studies, the accuracy of prediction tended to decrease with spatial extent of the study because of higher landscape heterogeneity and limitations related to sampling density [92], having the R² values generally lower than 0.5, while national- and regional-scale studies frequently produced values lower than 0.4 (Table 1). Therefore, SOC prediction accuracy obtained in France (R² = 0.412) is similar or slightly higher than previous large-scale studies, with similar large-scale mapping studies in mainland France producing an R² in the range between 0.10 and 0.47 [93,94]. On the other hand, the reduced accuracy achieved in Czechia is indicative of the challenges for digital SOC mapping in complex topography, which correlates with problematic prediction accuracy in mountainous areas, such as Switzerland, where R² values ranged from 0.19 to 0.32 across different depth intervals [43]. In terms of the effectiveness of machine learning algorithms used in previous studies with cross-validation, tree-based models, especially RF, frequently outperformed other evaluated methods, including CUB and SVR [95]. Ensemble learning models were less represented in previous studies and are more difficult to compare across studies due to various constructions, but a hybrid model based on RF and kriging produced higher prediction accuracy in comparison to individual methods [96]. Moreover, other complex methods also outperformed frequently used individual machine learning methods, among which deep neural networks were superior on a local scale [82], as well as on regional- and continental-scales [13,34].

4.2. The Impact of Selected Accuracy Assessment Approaches and Statistical Metrics on Prediction Accuracy

The predictive performance through accuracy measures based on SS overestimated predictive performance in both study areas relative to CV, and that SS therefore might be misleading in its ability to give an inflated feeling of model reliability depending on the randomness in data partition. This implies that reported accuracies of SOC prediction in previous studies can vary, not only because of model or data quality, but also because of the selected accuracy assessment approach. Although 10-fold cross-validation is widely used in digital SOC mapping [19,21,43,82], moderate fold sizes (k = 4, 5) can be slightly more robust approaches when using heterogeneous national or regional data. The negative correlation between k-fold number and metric variability in both study areas occurred as larger k values and produced more disproportional training-validation ratios, thereby increasing the randomness effect of sample selection, which aligns with theoretical assumptions of k-fold cross-validation [83].

A greater sensitivity of selected k values was observed on R² in comparison to RMSE or MAE, suggesting that the absolute accuracy assessment metrics were a more reliable measure of robustness of the evaluated machine learning models. Despite this observation, a previous study noted the comparative advantages of R² in comparison to RMSE and MAE in terms of comparing prediction accuracy in regression tasks across studies and sensitivity to value distribution of ground truth data [97]. Therefore, the use of multiple complementary statistical metrics for the accuracy assessment, as was performed in this study, is justified and should be replicated in future studies based on regression tasks. RF was the most sensitive to data partitioning, resulting in high levels of outliers and broad distributions ranges in all k-fold settings. This might be explained by the bootstrap resampling inherent in the bagging algorithm within RF and the extrinsic randomness due to data partitioning, causing RF to be highly sensitive to noise instead of real signals when training sets are small, as in high k-fold situations [98]. CUB and BRNNs were more consistent in prediction accuracy but produced lower median accuracy than RF. Compared to individual machine learning methods, ENS was by far the most resistant to randomness in training and validation data split while producing the highest prediction accuracy, indicating that model averaging is an effective way of eliminating the effect of randomness caused by particular cross-validation splits. This implies that model-specific biases can be canceled by using ensemble averaging and it minimizes variance by using the capacities of constituent algorithms as RFs ability to model complex non-linear structures is offset by the smoother and more conservative forecasts of neural network-based models [64].

While cross-validation is superior to the split-sample approach in terms of resistance to randomness in training and validation data split, it might be susceptible to data leakage as there is no hold-out validation dataset used [84]. Therefore, the results from this study addressed a component of robustness in accuracy assessment of digital SOC mapping and future studies should explore additional approaches, such as nested cross-validation, despite it being computationally expensive [99]. In contrast to standard cross-validation, which shares the same partitions of the data with the two tasks of hyperparameter tuning and performance estimation, nested cross-validation has a two-loop design, which strictly separates the model selection and performance estimation. This hierarchical structure is very useful in avoiding data leakage, which is a widespread problem in which information in validation sets accidentally permeates model-training or hyperparameter optimization, resulting in potentially over-optimistic model predictions [85].

4.3. The Impact of Selected Accuracy Assessment Approaches on Relative Importance of Environmental Covariates

In contrast to France, SOC changes in the Czech Republic are more strongly associated with surface soil characteristics and vegetation cover, including patterns of agricultural production, vegetation cycles, types of tillage and other soil surface dynamics [100]. This is consistent with the fact that the Czech Republic is more climatically homogeneous, making the variability driven by agronomic practices more pronounced [51]. Carbon accumulation is also significantly affected by the type of economic land use [101]. Agriculture dominates approximately 54% of the nation’s land area and Mišurec et al. detected that SOC levels declined over five years due to variations in agricultural practices [102]. Agricultural approaches depending only on mineral fertilizers (without the use of organic manures) often lead to degradation of the soil environment, such as soil pH decrease, reduction in the SOC and soil organic matter content or reduced availability of nutrients [103]. The mineral, N, fertilizers significantly promote decomposition processes and may cause the depletion of the SOC over time [103]. There is a fundamental difference in the factors controlling SOC at the national level. The higher average SOC in France (36.5 g·kg⁻¹) is likely a result of French soils often encompassing areas with favorable topography for organic matter accumulation, with forested, mountainous areas with low erosion, and sufficient amounts of exchangeable Ca²⁺, as well as the parent material from which the soil is formed [104]. It consequently resulted in an underestimation of the effect of some controlling factors driving high SOC values, such as forest or grassland land uses and high elevations [105]. Soils located in plots with a mountainous climate had higher C content [106]. Erosion influences the higher SOC content in France. Organic matter transported by erosion accumulates at the mountain foothills, where soils are formed in valleys, leading to an increase in SOC levels [107]. It is important to note that when estimating SOC content, the effectiveness of predicting soil surface texture varies depending on the acquisition date and the presence of crop residues on the surface [108].

4.4. Study Limitations and Future Considerations

Although the research has shown that cross-validation is effective for reducing the element of randomness in data partitioning, the present method has few limitations that should be addressed in future research. Annual median values of the covariates calculated from satellite image bands are used to simplify the aspect of temporal variability and may not capture dynamics of vegetation and the state of the soil surface intra-annually, especially in agricultural landscapes with numerous cropping cycles or rapid phenological changes [88]. This temporal aggregation may conceal the short-term variation that is important to the soil organic carbon variation. Future research efforts should thus consider temporally explicit modeling schemes on the basis of seasonal or phenological composites, or time-series characteristics on the basis of remote sensing information in order to better represent variability within-year. Moreover, data-related uncertainty can also have an effect on prediction reliability as the environmental covariates were modeled using multiple datasets with varying native spatial resolutions. Although point-based extraction of these values for vector soil samples reduced technical resampling errors, subpixel variations in input datasets can still cause scale-related uncertainty, especially in discontinuous or heterogeneous landscapes [109]. While outlier removal was implemented to avoid numerical instability and impact of extreme values, this also created a spatial bias. LUCAS data are also not spatially homogeneous and the process of the removal of outliers without accounting for their geographic distribution may lead to the loss of the capacity of the model to capture the actual natural heterogeneity. Future research should examine the spatial distribution of unobservable samples and determine whether they may cause model bias or regional bias such as through declustering or geographically weighted resampling methods.

5. Conclusions

This study aimed to systematically analyze the influence of the k folds of cross-validation on digital SOC mapping accuracy relative to the split-sample approach, and to assess the robustness of different machine learning algorithms under varying validation scenarios. The main conclusions based on the results of the comprehensive evaluation of k-fold cross-validation and split-sample approaches based on the randomness in data partition for the creation of training and validation datasets in two study areas on a national scale are as follows:

• The ensemble machine learning approach proved to be the most accurate for SOC prediction in France with 10-fold cross-validation, producing R² of 0.412, which is on-par with prediction accuracy achieved in similar previous studies.
• All evaluated k-fold numbers agreed on the optimal model, with 10-fold cross-validation producing the most accurate results and 2-fold cross-validation suggesting lower accuracy across all four metrics and both study areas, likely due to the lower quantity of training data relative to other k-fold number and exaggerated sensitivity to randomness in data splitting.
• In both study areas, 10-fold cross-validation produced the largest variability in all the accuracy measures due to the largest discrepancy between quantity of training and validation data. These results suggest that moderate fold sizes (k = 4, 5) can be slightly more robust approaches when using heterogeneous national or regional data.
• The selection of the k-fold number did not have a notable impact on relative variable importance values of the most accurate evaluated machine learning model. DEM had dominant importance for SOC prediction in France, while two spectral indices were the most important for the prediction in Czechia.
• While cross-validation is superior to the split-sample approach in terms of resistance to randomness in training and validation data split, it might be susceptible to data leakage as there is no hold-out validation dataset used. Therefore, the results from this study addressed a component of robustness in the accuracy assessment of digital SOC mapping and future studies should explore additional approaches, such as nested cross-validation.