A Machine Learning-Based Predictive Model for Maintenance Management of Combustion Engines in the Agricultural Sector

Abstract

Maintenance management of stationary combustion engines in the agricultural sector remains largely manual, increasing the risk of unplanned downtime. This study developed a machine learning-based predictive model to anticipate failures within a 60-day horizon, enabling the transition from reactive to proactive maintenance. Following the CRISP-DM (Cross-Industry Standard Process for Data Mining) framework, a sliding-window feature engineering pipeline was built from 2250 historical records spanning 59 engines. Four ensemble learners (Random Forest, LightGBM, XGBoost, and CatBoost) were then compared under two complementary protocols: a strict 60/40 chronological split simulating deployment, and a stratified leave-engine-group-out cross-validation withholding entire engines from training. Nonparametric testing (DeLong test and engine-level cluster bootstrap) showed that the four learners are statistically equivalent, whereas the feature engineering layer contributes a large, significant discrimination gain (ΔAUC $\approx +0.06$, $p\approx {10}^{-27}$) on engines unseen during training. Random Forest, selected as the final model, achieved an AUC of 0.90 with 84.2% recall under the deployment protocol and 0.96 with 90.7% recall under engine-grouped validation. Temporal extrapolation, rather than cross-engine generalization, appeared to be the primary challenge, indicating that rigorously engineered degradation features, more than the choice of ensemble algorithm, drive predictive performance in agricultural maintenance planning.

1. Introduction

In the context of Agriculture 4.0, the digitization and automation of processes have become central to improving efficiency and sustainability in the sector [1]. Within this ecosystem, stationary internal combustion engines play a critical role in powering irrigation systems, electric generators, and drainage pumps, often operating under constant load and in harsh environmental conditions. Unlike mobile machinery, an unexpected failure in a stationary engine can halt entire agricultural production lines, resulting in significant economic losses from unplanned downtime.

Traditionally, the management of these assets has relied on corrective maintenance strategies (repairing after a failure) or static preventive maintenance based on hours of use or fixed time intervals recommended by the manufacturer. However, this deterministic approach is often inefficient: it can lead to unnecessary interventions on healthy equipment (thereby wasting component lifespan) while simultaneously failing to detect premature degradation [2,3].

The application of machine learning (ML) to predictive maintenance has been explored across multiple industrial domains, ranging from sensor-based fault diagnosis to administrative-record-driven failure prediction. Section 2 reviews the state of the art in these areas and identifies the gap that motivates the present work.

Predictive Maintenance (PdM) offers a superior alternative by leveraging historical and operational data to anticipate failures before they occur [4,5]. The implementation of artificial intelligence (AI) techniques has enabled a shift from basic monitoring to complex predictive models [6]. Recent studies have explored various architectures, ranging from artificial neural networks (ANNs) for optimizing combustion parameters [7,8] to deep learning approaches for sensor-based diagnostics [9]. Niroomand and Bach [10] applied Random Forest to predict emissions, and earlier comparative studies have validated the effectiveness of classification algorithms for decision-making [11]. Aliramezani et al. [12], meanwhile, highlight the potential of model-based predictive control (MPC). However, the volume and variability of data in agricultural settings require techniques that balance accuracy and computational efficiency. Although classical algorithms have proven effective [13,14], gradient boosting methods have gained increasing traction. Nevertheless, a gap persists in the literature regarding predictive models specifically adapted to the constant-load dynamics of stationary engines, as distinct from vehicular approaches or non-invasive acoustic monitoring techniques [15].

Rather than proposing a novel algorithm, this study addresses an applied engineering problem: identifying, through rigorous empirical comparison, which among the most competitive existing ensemble methods is best suited for deployment in a future agricultural maintenance management system. Specifically, the objectives are: (i) to construct a domain-adapted predictive pipeline from administrative maintenance records using sliding-window feature engineering, and to quantify the contribution of each pipeline component through a statistically tested ablation study; (ii) to compare Random Forest, LightGBM [16], XGBoost, and CatBoost under two complementary validation protocols (a strict chronological split simulating deployment and a leave-engine-group-out scheme measuring generalization to unseen engines), following the CRISP-DM framework [17] and supported by nonparametric significance testing [18,19]; and (iii) to generate actionable failure probability estimates within a 60-day horizon that can serve as the decision layer of a real-world maintenance application.

2. Related Work

2.1. Evolution of Maintenance Strategies and the Role of Data-Driven Methods

The management of industrial assets has evolved through three successive paradigms. Corrective maintenance (repairing equipment after failure) dominated early industrial practice but results in unplanned downtime and high replacement costs [2,3]. Preventive maintenance introduced fixed-interval servicing based on manufacturer recommendations, reducing catastrophic failures at the expense of unnecessary interventions on healthy components [3]. Predictive Maintenance (PdM) represents the current frontier: by continuously analyzing operational data, it aims to anticipate failures only when degradation indicators cross a learned threshold, optimizing the trade-off between intervention cost and asset availability [4].

Carvalho et al. [4] conducted a systematic review of 43 studies applying ML to PdM across industrial domains, identifying decision-tree ensembles and support vector machines as the most frequently adopted architectures. Zonta et al. [2] similarly reviewed Industry 4.0 PdM implementations, concluding that the availability and quality of historical operational records is a more critical bottleneck than algorithmic choice, a finding that directly motivates the feature engineering strategy adopted in this study. Amruthnath and Gupta [5] further demonstrated that unsupervised methods can detect early fault signatures in unlabeled datasets, underscoring the utility of clustering as a preprocessing layer for supervised models.

2.2. Machine Learning Applied to Internal Combustion Engine Diagnostics

Within the specific domain of internal combustion engines, ML applications have followed two main trajectories. The first focuses on predicting combustion performance parameters. Bhatt and Shrivastava [7] reviewed over 80 studies applying artificial neural networks (ANNs) to this task, reporting consistent accuracy gains over physics-based simulators. Mohasab et al. [8] extended this analysis to recent architectures, confirming that deep feedforward networks can predict brake thermal efficiency with errors below 5% under steady-state conditions. Niroomand and Bach [10] applied Random Forest to predict segment-level CO₂ emissions from internal combustion engines, noting that RF performance degraded with imbalanced temporal datasets; this study addresses class imbalance through class-weighting and re-examines the relative merits of bagging and boosting ensembles under formally tested validation protocols (Section 4.3).

The second trajectory targets fault diagnosis and remaining useful life estimation, typically relying on continuous sensor signals. Giordano et al. [9] applied data-driven strategies to predictive maintenance in an automotive context using deep learning over sensor time series, reporting strong performance but requiring instrumented fleets with real-time telemetry. Torres et al. [15] systematically reviewed non-invasive monitoring techniques, concluding that while such approaches offer diagnostic value, they impose significant sensor deployment costs poorly suited to distributed agricultural fleets. Aliramezani et al. [12] reviewed model-based predictive control for diesel engines, highlighting their dependence on detailed physical engine models unavailable for aging or undocumented assets. Collectively, these studies reveal a methodological gap: dominant paradigms either require rich sensor infrastructure or detailed physical models, neither available in the agricultural settings addressed here.

2.3. Gradient Boosting Methods in Industrial Maintenance

Among ensemble methods, gradient boosting frameworks have emerged as strong performers in maintenance prediction tasks involving tabular, imbalanced, and temporally structured data. Maione et al. [20], comparing multiple ML algorithms for condition-based maintenance of diesel engines, reported that Random Forest exhibited the highest prediction error and longest training times, while gradient boosting variants consistently delivered superior generalization. Roman et al. [6] demonstrated that combining feature selection with gradient-boosted trees outperforms classical ensemble methods in smart agricultural classification tasks. The LightGBM algorithm [16] extends standard gradient boosting via a leaf-wise tree growth strategy and histogram-based approximation, substantially reducing memory consumption and training time, properties that are particularly relevant when scaling to multi-fleet agricultural applications.

2.4. Predictive Maintenance in Agricultural Contexts and Identified Gaps

The digitization of agriculture under the Industry 4.0 paradigm has accelerated the deployment of monitoring systems for field machinery [1], yet research on PdM specifically for stationary combustion engines (irrigation pumps, generators, and drainage systems) remains scarce. The majority of agricultural ML studies address mobile machinery or crop-process optimization, leaving stationary assets underrepresented despite their critical role in irrigation infrastructure. Furthermore, published PdM models for combustion engines are almost exclusively validated on instrumented test benches or vehicular fleets with continuous telemetry, limiting their applicability to small and medium agricultural enterprises in developing economies where sensor infrastructure is absent and maintenance records are manual and transactional. Here, this gap is addressed by proposing a model architecture that relies exclusively on historical administrative records and incorporates usage-profile segmentation, a combination not previously evaluated for stationary agricultural engines.

3. Materials and Methods

3.1. Problem Formulation

• Dataset and observation space. Let $\mathcal{E}=\{e_1,e_2,\dots ,e_M\}$ denote a fleet of $M=59$ stationary combustion engines. For each engine $e_i$, a sequence of $n_i$ operational records is available, ordered chronologically. The full dataset is:

$$\mathcal{D}={\left\{({\mathbf{x}}_{i,t},y_{i,t})\right\}}_{i=1}^M,t=1,\dots ,n_i$$

(1)

where ${\mathbf{x}}_{i,t}\in {\mathbb{R}}^d$ is the feature vector for engine $e_i$ at operational snapshot t, and $y_{i,t}\in \{0,1\}$ is the binary label defined below. The total number of valid records is $N={\sum }_{i=1}^M{n}_i=2250$, with $d=47$ predictor variables (46 engineered features plus one cluster-assignment variable, after the exclusion of two label-associated event descriptors; see Section 3.4, Section 3.5 and Section 3.7).

• Feature transformation. The raw transactional record ${\mathbf{r}}_{i,t}$ is transformed via a sliding-window function $\varphi$:

$${\mathbf{x}}_{i,t}=\varphi \left({\mathbf{r}}_{i,t-W:t}\right)$$

(2)

where W is the window width (7, 14, or 60 days depending on the feature), yielding degradation-indicative trend indicators.

• Prediction horizon and label definition. The target variable is defined over a forward-looking window $\mathsf{\Delta }=60$ days. A record is labelled positive if an unscheduled corrective maintenance intervention occurs within the next $\mathsf{\Delta }$ days:

$$y_{i,t}=\left\{\begin{array}{cc}1\hfill & \mathrm{if}\exists t^{\prime }\in (t,t+\mathsf{\Delta }]\mathrm{with}\mathrm{a}\mathrm{corrective}\hfill \\ & \mathrm{intervention}\mathrm{at}{t}^{\prime }\hfill \\ 0\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

Predictive model. The goal is to learn a function:

$$f:{\mathbb{R}}^d\to [0,1],{\widehat{p}}_{i,t}=f\left({\mathbf{x}}_{i,t};\theta \right)$$

(4)

A classification decision is obtained by applying a threshold $\tau$: ${\widehat{y}}_{i,t}=\mathbf{1}[{\widehat{p}}_{i,t}\ge \tau ]$.

• Optimization objective. Given the asymmetric cost structure of predictive maintenance, the primary criterion is:

$$\mathrm{AUC}={\int }_0^1\mathrm{TPR}\left(\tau \right)d\mathrm{FPR}\left(\tau \right)$$

(5)

subject to a secondary constraint on recall: $\mathrm{Recall}=TP/(TP+FN)\ge {\mathrm{Recall}}_{min}$.

• Temporal data partition. Let $t*$ denote the split point (October 2022):

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{train}}& =\{({\mathbf{x}}_{i,t},y_{i,t}):t\le t*\},\left|{\mathcal{D}}_{\mathrm{train}}\right|=1350(60\%)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathcal{D}}_{\mathrm{test}}& =\{({\mathbf{x}}_{i,t},y_{i,t}):t>t*\},\left|{\mathcal{D}}_{\mathrm{test}}\right|=900(40\%)\hfill \end{array}$$

(7)

This strict temporal ordering guarantees that no future information is available during training, preventing data leakage and simulating a real-world deployment scenario.

3.2. Research Design

The proposed approach follows CRISP-DM phases 1 through 5 (business understanding, data understanding, data preparation, modeling, and evaluation [17]), whereas deployment (phase 6) is left for future work. Translating the predictive pipeline into a field-ready decision-support tool for technicians would require a substantial engineering effort, encompassing system architecture, real-time data ingestion, interface design, and field validation, which falls outside the scope of the present study.

The methodological approach was quantitative and experimental, centering on supervised binary classification to estimate the probability of future failure in critical machinery.

3.3. Dataset and Preprocessing

The dataset was sourced exclusively from historical administrative maintenance records of a real-world fleet managed by an agricultural enterprise in the coastal region of Ecuador (Province of Los Ríos). Records were generated manually by field technicians at each service event using a proprietary fleet management system, with no automated sensor instrumentation; consequently, data acquisition is event-driven rather than time-series based, with one record per maintenance intervention (either preventive or corrective). The original dataset covered the period from December 2018 to June 2025 and comprised 59 stationary diesel engines deployed across irrigation and drainage pumping stations. The fleet spans nine manufacturers, including Caterpillar, Cummins, Weichai, Detroit, MWM, and Maxx-Force, with rated power ranging from 75 HP to 470 HP (mean: 281 HP), reflecting the heterogeneous composition typical of agricultural fleets in the region. Each engine contributed between 22 and 61 operational records, with a median of 38 records per engine. Of the 59 engines, 47 (79.7%) experienced at least one failure event during the observation period; the remaining 12 operated without recorded failures and contributed exclusively to the “normal operation” class.

A failure event (class 1) was defined operationally as any unscheduled corrective maintenance intervention recorded in the system, including the repair or replacement of critical components such as the injection pump, timing belt, turbocharger, or starter motor. The labeling rule applied a forward-looking window of $\mathsf{\Delta }=60$ days: any record at time t was assigned label $y=1$ if at least one such corrective intervention occurred in the interval $(t,t+60\mathrm{days}]$, and $y=0$ otherwise. Scheduled preventive interventions and the 18 unclassified event records were excluded from the failure class.

Data cleaning addressed three types of issues: (i) duplicate records caused by data entry errors (removed by deduplication on engine ID and service date, $n=43$ records removed); (ii) records with missing values in key date fields ($n=28$ records imputed using the median inter-service interval per engine); and (iii) outliers in the “Engine Hours” field identified via the IQR method ($n=17$ records with values exceeding $Q_3+3·\mathrm{IQR}$ were capped at the 99th percentile). After cleaning, 2250 valid records were retained.

The definition of the prediction horizon within a 60-day window inherently increases the proportion of positive labels in the target variable, since multiple operational records preceding a breakdown are flagged as imminent failures. Consequently, of the 2250 total valid records, failure events (class 1) represented 55.0% of the dataset (n = 1238), while normal operation (class 0) accounted for 45.0% (n = 1012). These failure records are distributed across the 47 engines that experienced at least one failure event, with individual per-engine contributions ranging from 3 to 45 records (mean: 22.5, median: 25, SD: 11.0), indicating moderate heterogeneity in per-engine failure load. For transparency, the training set exhibited a 53.3% failure rate, whereas the test set contained 57.6% failures (n = 518). Although this rules out the severe minority-class imbalance typical of raw maintenance data, the moderate variation in class distribution across time, together with potential engine-specific bias, was addressed during modeling through class-weighting strategies.

3.4. Feature Engineering

Because the raw data lacked direct predictive power, transformation techniques were applied to derive 48 explanatory variables capturing trends and degradation patterns, of which 46 were retained in the final feature space (Section 3.5). Key indicators explicitly used by the model’s decision nodes were constructed, such as “Days since last maintenance”, “Cumulative failure rate”, “Accumulated Maintenance Count” (Accum. Maint. Count), “Engine Age” (in days), “Total Station Failures”, and cumulative “Engine Hours”. Additionally, to capture the operational seasonality inherent in agriculture (harvests or droughts), a sliding window technique was applied to the time series of engine-hours. This yielded the variable “Daily Usage Intensity,” which captures the asset’s recent mechanical stress as distinct from its absolute age. A complete dictionary of the derived features, grouped by category and with their corresponding units, is provided in Appendix A (

Table A1. Complete dictionary of derived features for the predictive model.

Variable	Description	Unit
Group 1: Raw operational inputs
potencia_hp	Rated engine power (manufacturer specification)	HP
horas_motor_actuales	Current hour-meter reading at the time of intervention	Hours
tipo_mant_actual	Type of current intervention (0 = Preventive, 1 = Corrective) (excluded from the final feature space, Section 3.5)	Cat.
actividad_actual_enc	Specific maintenance activity, target-encoded by failure rate (excluded from the final feature space, Section 3.5)	Ratio
Group 2: Time-based features
dias_desde_ultimo_mant	Days elapsed since the last service intervention	Days
horas_desde_ultimo_mant	Engine hours accumulated since the last service	Hours
antiguedad_motor_en_dias	Total operational age of the asset since commissioning	Days
dias_desde_primer_mant	Days elapsed since the first recorded maintenance event	Days
año_mantenimiento	Calendar year in which the intervention was recorded	Year
trimestre_mantenimiento	Quarter of the year in which the intervention occurred	Cat.
mes_mantenimiento	Month in which the intervention was recorded	Cat.
Group 3: Cumulative maintenance history
num_mantenimientos_previos	Total number of prior interventions recorded for the engine	Count
num_correctivos_previos	Total number of prior corrective interventions	Count
conteo_mantenimientos_acum	Accumulated maintenance count up to the current record	Count
correctivos_acumulados	Accumulated corrective interventions over engine lifetime	Count
horas_totales_acumuladas	Total accumulated operating hours over engine lifetime	Hours
ratio_fallas	Historical failure frequency (failures/total interventions)	Ratio
ratio_fallas_acumuladas	Cumulative failure rate up to the current record	Ratio
horas_por_falla_correctiva	Mean operating hours elapsed between corrective failures	Hrs/Fail
Group 4: Sliding-window features
media_movil_horas_3	3-event rolling mean of engine hours at intervention	Hours
std_dev_dias_3	3-event rolling standard deviation of inter-service intervals	Days
fallas_ultimos_3_mant	Number of corrective events in the last 3 interventions	Count
Group 5: Station-level contextual features
fallas_historicas_estacion	Historical corrective interventions at the pumping station prior to record	Count
total_fallas_estacion_hist	Total cumulative corrective interventions recorded at the station	Count
promedio_horas_estacion	Mean operating hours across all engines at the station	Hours
Group 6: Outlier indicator flags
outlier_horas_motor_act.	Binary flag indicating anomalous hour-meter reading (IQR method)	Binary
outlier_dias_ultimo_mant	Binary flag indicating anomalous inter-service day interval	Binary
outlier_horas_ultimo_mant	Binary flag indicating anomalous hours since last service	Binary
Group 7: Brand dummy variables (one-hot encoding)
marca_CATERPILLAR	Binary indicator: engine manufactured by Caterpillar	Binary
marca_CUMMINS	Binary indicator: engine manufactured by Cummins	Binary
marca_DETROIT	Binary indicator: engine manufactured by Detroit	Binary
marca_MARATHON	Binary indicator: engine manufactured by Marathon	Binary
marca_MAXX-FORCE	Binary indicator: engine manufactured by Maxx-Force	Binary
marca_MWM	Binary indicator: engine manufactured by MWM	Binary
marca_SIEMENS	Binary indicator: engine manufactured by Siemens	Binary
marca_US_ELEC_MOTOR	Binary indicator: engine manufactured by US Electrical Motor	Binary
marca_WEICHAI	Binary indicator: engine manufactured by Weichai	Binary
Group 8: Categorical encodings
modelo_target_enc	Engine model encoded by observed failure rate	Ratio
estacion_hacienda_enc	Hacienda identifier encoded by observed failure rate	Ratio
estacion_tipo_tipo	Station type label-encoded (irrigation/drainage)	Cat.
tipo_mant_actual_mant	Maintenance type label-encoded (0 = Preventive, 1 = Corrective) (excluded from the final feature space, Section 3.5)	Cat.

). The present section describes in detail only the features that appear as primary contributors to the final ensemble (see Section 4.5).

3.5. Exclusion of Label-Associated Event Descriptors

During the revision of the evaluation protocol, two predictors derived from the current maintenance event, namely the event-type indicator (tipo_mant_actual) and the target-encoded activity description (actividad_actual_enc), were found to act as proxies of the label rather than as degradation signals: given the event-driven structure of the records and the 60-day forward-looking window, the conditional failure rate is 96.1% for records logged at preventive events versus 24.5% for records logged at corrective events. Because this association reflects how interventions are recorded (a corrective repair resets the short-term failure risk, whereas preventive services are scheduled irrespective of it) rather than the physical condition of the engine, both variables were excluded from the final feature space, which therefore comprises $d=47$ predictors (46 engineered features plus the cluster assignment). Their presence in earlier versions of the model compressed all learners into a narrow performance band and masked the contribution of the remaining pipeline components (Section 4.4).

3.6. Definition of the Prediction Horizon

The task was approached as a time-series classification problem, in which the target variable represents the probability of failure within a future 60-day window. The selection of this time horizon is justified by two fundamental criteria:

• Logistical-Operational Criterion: In agricultural contexts with large fleets, the supply chain for critical spare parts requires extended planning times. A shorter window would be insufficient for supply management, whereas the 60-day horizon enables tactical planning of downtime without affecting production peaks.
• Feature-Horizon Alignment: The 60-day horizon matches the time scale of the engineered predictors, which encode cumulative load and failure-rate trajectories over multi-week windows. The study does not define or evaluate a taxonomy of failure types; the contribution of the trajectory-based features to predictive performance is quantified directly in the ablation study (Section 4.4).

3.7. Usage Profile Segmentation (Clustering)

In the context of stationary agricultural engines, a load pattern is defined as the characteristic combination of daily usage intensity (hours/day), cumulative operating hours, and maintenance frequency that describes how an engine is habitually deployed across its operational life. Engines subjected to intensive irrigation campaigns exhibit fundamentally different degradation trajectories from those used intermittently for drainage. Failing to account for this heterogeneity would conflate structurally distinct failure signatures, reducing the classifier’s discriminatory power. K-Means was selected over hierarchical or model-based clustering because: (i) the dataset size ($N=2250$) and the low dimensionality of the profiling variables favor simple, centroid-based prototypes that are directly interpretable and can be communicated to field technicians; and (ii) the expected number of operational regimes was hypothesized to be small (2–3) based on domain knowledge about irrigation scheduling in the region.

To mitigate variability arising from different load patterns, a hybrid strategy was adopted, beginning with the unsupervised K-Means algorithm. The profiles were constructed from dynamic behavior variables derived through feature engineering, specifically daily usage intensity (hours/day), hourly accumulation trend, and monthly frequency of interventions. To determine the optimal number of clusters (k) and avoid arbitrary segmentation, a comprehensive internal validation study was conducted using four metrics: (1) the elbow method (distortion score), (2) the Silhouette coefficient, (3) the Calinski–Harabasz index, and (4) the Davies–Bouldin index.

Two of the four metrics (Silhouette coefficient and Davies–Bouldin index) indicated $k=2$ as the optimal partition. Although the Elbow method suggested k = 5 and the Calinski–Harabasz index pointed to $k=6$, the interpretability of a two-cluster solution and its alignment with the operational context led to the selection of $k=2$, ruling out higher-order partitions that exhibited lower internal cohesion. This confirmed the existence of two clear latent profiles: “Standard Operation” and “Intensive Operation” [21].

The resulting cluster assignment for each operational record, encoded as a binary variable ($c_{i,t}\in \{0,1\}$, where $0=$ Standard Operation and $1=$ Intensive Operation), was incorporated as an additional predictor variable into the feature matrix ${\mathbf{x}}_{i,t}$, yielding a final feature space of $d=47$ predictors. This integration follows a hybrid unsupervised–supervised pipeline: rather than training separate classifiers per cluster, which would fragment the dataset and reduce statistical power, the cluster label acts as a contextual flag that allows the supervised model to learn cluster-specific failure thresholds within a unified framework. The predictive contribution of this variable is tested formally in the ablation study (Section 4.4); as reported there, the segmentation adds no measurable discriminative power, and its value in the pipeline is descriptive and operational (fleet characterization for maintenance planning) rather than predictive.

3.8. Model Configuration and Algorithms

For the supervised modeling phase, the input feature matrix consisted of the 46 engineered variables retained after the exclusion described in Section 3.5 plus the binary cluster assignment described in Section 3.7, yielding a final feature space of $d=47$ predictors.

Decision tree-based ensemble algorithms were selected due to their robustness to noise and ability to handle nonlinear relationships. Four state-of-the-art ensembles were compared under identical conditions: Random Forest [13], a bagging ensemble of decorrelated deep trees; and three gradient boosting frameworks: LightGBM [16], XGBoost, and CatBoost. Random Forest was ultimately adopted as the final model on the basis of the comparative results (Section 4.3), while the boosting variants serve as competitive references.

All models were trained in Python 3.10.3 using the Scikit-learn [22] and LightGBM libraries. The target variable was defined as binary: [1: Failure, 0: Normal]. To address the class imbalance identified during preprocessing, a class-weighting strategy increased the model’s sensitivity to the minority class.

For all four models, hyperparameter tuning was performed using GridSearchCV coupled with TimeSeriesSplit ($n=5$) on the training set. In response to the reviewers’ observations, the search spaces of the present revision were widened symmetrically so that all learners receive comparable capacity budgets; in particular, the original grid unintentionally restricted Random Forest to shallow trees (≤5 levels), an architecture in which the algorithm cannot express its characteristic variance-reduction behavior. The widened search spaces and the parameters selected for each model are detailed in

Table 1. Best hyperparameters selected by GridSearchCV (TimeSeriesSplit, $n=5$).

Hyperparameter	Model	Search Space	Selected Value
n_estimators	Random Forest	{50, 100, 300}	300
max_depth	Random Forest	{3, 5, None}	None
min_samples_split	Random Forest	{2, 5}	2
min_samples_leaf	Random Forest	{1, 2}	1
n_estimators	LightGBM	{50, 100, 500}	500
learning_rate	LightGBM	{0.05, 0.1}	0.05
num_leaves	LightGBM	{15, 31, 63}	63
min_child_samples	LightGBM	{20, 30}	20
scale_pos_weight	LightGBM	{0.8762, 1.051}	1.051
n_estimators	XGBoost	{50, 100, 300}	300
learning_rate	XGBoost	{0.05, 0.1}	0.05
max_depth	XGBoost	{3, 5, 6}	6
min_child_weight	XGBoost	{1, 5}	1
scale_pos_weight	XGBoost	{0.8762, 1.051}	0.8762
iterations	CatBoost	{50, 100, 300}	300
learning_rate	CatBoost	{0.05, 0.1}	0.05
depth	CatBoost	{3, 5, 6}	6
l2_leaf_reg	CatBoost	{1, 3}	1
scale_pos_weight	LightGBM/XGBoost	$N_{\mathrm{neg}}/N_{\mathrm{pos}}$ (computed)	1.051/0.8762
class_weight	Random Forest	‘balanced’	‘balanced’
auto_class_weights	CatBoost	‘Balanced’	‘Balanced’

Note: Selection criterion was AUC on validation folds. The computed base negative-to-positive ratio for class balancing was 0.88. Search spaces were widened in this revision so that all learners receive comparable capacity budgets; the original grid capped Random Forest at depth 5, which under-represented its attainable performance.

.

3.9. Training and Validation Strategy

To ensure the model’s robustness and avoid overfitting, a comparative experiment tested two candidate validation strategies:

• Scenario A: Classic 80/20 split (training/testing) with 3-fold cross-validation ($k=3$).
• Scenario B: Strict 60/40 chronological (temporal) split with 5-fold cross-validation ($k=5$). To rigorously evaluate the predictive horizon, the split was performed chronologically: historical records from December 2018 up to October 2022 formed the training set (60%), while the subsequent records from November 2022 to June 2025 constituted the test set (40%). This strict temporal ordering ensures that no future information is available during training, effectively preventing data leakage and simulating a real-world deployment scenario.

Preliminary results indicated that Scenario B yielded a more stable estimate of generalization error and was therefore adopted for the final evaluation.

In addition to the chronological deployment protocol, and in response to the reviewers’ request for an evaluation setting capable of separating methods, all comparative and ablation experiments were replicated under a stratified leave-engine-group-out (LOGO) protocol: a stratified group 10-fold cross-validation (StratifiedGroupKFold) in which the grouping unit is the engine, so that every record of a test engine is excluded from training (5–6 engines per test fold, with groups stratified by the presence of failure events). To prevent any information flow across partitions, all data-dependent transformations (the target encodings of categorical variables and the K-Means usage-profile segmentation) were re-fitted within each fold using training engines only. Performance is reported on pooled out-of-fold predictions, in which each record is predicted exactly once by a model that never observed its engine. The two protocols answer complementary questions: the chronological split measures the model’s capacity to extrapolate into future time periods for known assets (the deployment scenario), whereas the engine-grouped protocol measures generalization to entirely unseen engines and provides the discriminative setting for validating the pipeline components (Section 4.4).

3.10. Evaluation Metrics and Class Balancing Techniques

Although the 60-day labeling rule yields a moderate class balance (55.0% failure vs. 45.0% normal operation), overall accuracy remains an uninformative primary metric under the asymmetric cost structure of predictive maintenance, in which an undetected failure is far more costly than an unnecessary inspection. Following the robust validation strategies proposed by [23], model performance is reported using four metrics: the Area Under the ROC Curve (AUC-ROC), which measures discrimination capability independently of the classification threshold and serves as the primary criterion; Sensitivity (Recall), critical for minimizing false negatives (undetected failures); and F1-score and Precision, which characterize the trade-off between failure detection and false alarms. Confusion matrices complement these metrics by visualizing the distribution of prediction errors for the leading models.

Statistical Significance Testing

Because point differences between competing classifiers may fall within sampling noise, all pairwise comparisons in this revision are supported by nonparametric hypothesis tests, following the recommendations of Demšar [19] for classifier comparison without distributional assumptions. Differences in AUC computed on the same evaluation records are tested with the DeLong test for correlated ROC curves [18], a nonparametric procedure based on the Mann–Whitney U-statistic representation of the AUC that accounts for the covariance between curves estimated from identical samples. Differences in Recall and F1-score, which depend on the operating threshold and violate independence across records from the same engine, are tested with a cluster bootstrap [24]: engines (not records) are resampled with replacement ($B=2000$ replicates), preserving the within-engine correlation structure, and two-sided p-values are obtained from the empirical distribution of the paired metric differences together with 95% percentile confidence intervals. For the cross-validated protocol, per-fold means and standard deviations are reported alongside pooled out-of-fold estimates. Where several pairwise model comparisons form a family, the Holm–Bonferroni procedure is applied to control the family-wise error rate. All tests use a significance level of $\alpha =0.05$.

4. Results and Discussion

4.1. Characterization of Operational Patterns

Unsupervised exploratory analysis revealed latent structures in the operational behavior of the fleet. Internal validation was conducted using four metrics (elbow, Silhouette, Calinski–Harabasz, and Davies–Bouldin), as shown in Figure 1. The Silhouette coefficient and the Davies–Bouldin index both indicated $k=2$, whereas the elbow method and the Calinski–Harabasz index suggested higher-order partitions ($k=5$ and $k=6$, respectively). The $k=2$ solution was selected on the basis of two of the four metrics together with its operational interpretability and alignment with the irrigation/drainage usage regimes of the fleet.

The cluster centroids were then characterized using a radar plot (Figure 2), allowing the two groups to be labeled. Cluster 0 (Standard Operation) groups engines with moderate workloads and regular maintenance intervals. In contrast, Cluster 1 (Intensive Operation) shows markedly higher values of “Daily Usage Intensity” and “Hour Trend,” consistent with equipment subjected to severe mechanical stress. This distinction was essential for providing the supervised model with operational context.

4.2. Comparison of Evaluation Protocols

A key technical finding concerned the relative difficulty of the two evaluation protocols. Figure 3 compares ROC curves for the final Random Forest model under the chronological 60/40 split (AUC = 0.90) and the engine-grouped LOGO protocol on pooled out-of-fold predictions (AUC = 0.96). Contrary to the assumption that unseen engines are harder to predict, all models performed better under LOGO than under the chronological split (Section 4.4). This suggests the degradation signal in maintenance records (intervention recency, cumulative load, failure-rate trajectories) is largely engine-agnostic, whereas extrapolating across future time periods, with its shift in fleet composition and operating conditions, is the binding difficulty for deployment. The chronological split is thus retained as the primary deployment-oriented evaluation, and LOGO as the discriminative setting for component validation.

4.3. Predictive Performance Evaluation

Under the chronological deployment protocol (60/40 split), the final comparison among the four ensembles was performed with the widened hyperparameter grids of Table 1 and the revised feature space of Section 3.5. Figure 4 illustrates the overall performance using ROC curves. Random Forest achieved the highest test AUC (0.9032), followed by LightGBM (0.8916), XGBoost (0.8864), and CatBoost (0.8860). Random Forest also attained the highest Recall (84.2%) and F1-score (84.5%) at the default operating threshold. Consistent with the reviewers’ observations on the previous version of this comparison, we emphasize that the margins among learners are modest; their statistical assessment is presented under the discriminative protocol in Section 4.4, where the learners prove statistically equivalent once multiple comparisons are controlled, and the substantive performance driver is shown to be the feature engineering layer.

This result diverges from part of the comparative literature: [20] reported that RF yielded the highest prediction error among models benchmarked for diesel-engine maintenance, and [6] found gradient-boosted ensembles superior in agricultural classification tasks. The present findings indicate that such rankings are sensitive to the hyperparameter budgets granted to each learner: when Random Forest is allowed deep, fully grown trees, it matches or exceeds the boosting frameworks on this administrative-records task. This underscores the importance of symmetric tuning budgets in applied benchmarking studies.

For a detailed analysis, it is crucial to examine the confusion matrices, since the primary objective in predictive maintenance is to minimize false negatives: engines predicted as healthy that subsequently fail, leading to costly unplanned downtime.

As shown in Figure 5a, the proposed Random Forest model produced 82 false negatives on the chronological test set, detecting 436 of the 518 actual failures. The strongest boosting alternative, LightGBM (Figure 5b), produced 116 false negatives (402 detected failures) with fewer false positives (54 vs. 78). The two matrices therefore represent different operating points on their respective ROC curves; since operating points can be shifted by re-thresholding, the comparison of learners rests on the threshold-independent AUC and its statistical assessment (Section 4.4), not on the confusion matrices alone.

Table 2. Comparison of performance metrics across all evaluated models under the chronological deployment protocol ($n=900$).

Model	AUC	Recall	F1-Score	Precision
Random Forest (proposed)	0.90	84.2%	84.5%	84.8%
LightGBM	0.89	77.6%	82.5%	88.2%
XGBoost	0.89	75.9%	81.7%	88.5%
CatBoost	0.89	68.9%	78.0%	89.7%

Note: Calculations based on the confusion matrices at the default threshold ($\tau =0.5$); Figure 5 shows the matrices for the two leading models. Statistical comparison of the learners is reported under the engine-grouped protocol in Table 3.

quantifies the deployment-protocol comparison across all four models. Random Forest leads every metric except Precision, where the boosting methods trade sensitivity for a more conservative positive class. Recall is the decisive metric in this maintenance context: the 84.2% achieved by Random Forest means that roughly five of every six imminent failures are anticipated at least 60 days in advance.

4.4. Component Validation Under Leave-Engine-Group-Out Cross-Validation

Following the reviewers’ recommendation, all comparative and ablation experiments were replicated under the stratified engine-grouped protocol described in Section 3.9, with formal significance testing as specified in Section 3.10. Figure 6 shows the ROC curves computed on pooled out-of-fold predictions, and

Table 3. Model comparison under stratified leave-engine-group-out validation (10 folds grouped by engine; pooled out-of-fold predictions). p-values vs. Random Forest: DeLong test [18] for AUC; engine-level cluster bootstrap [24] for Recall and F1.

Model	AUC	AUC/Fold ($\mathit{\mu }\pm \mathit{\sigma }$)	Recall	F1	p (AUC)	p (F1)
Random Forest (proposed)	0.956	$0.948\pm 0.045$	90.7%	89.6%	—	—
CatBoost	0.953	$0.947\pm 0.043$	88.8%	89.5%	0.042	0.826
XGBoost	0.952	$0.950\pm 0.038$	89.2%	89.2%	0.038	0.644
LightGBM	0.951	$0.947\pm 0.044$	88.0%	88.5%	0.011	0.120

Note: nominal p-values are reported; after Holm–Bonferroni correction of the comparison family, only the AUC difference vs. LightGBM remains significant. The learners are statistically equivalent from a practical perspective, and the substantive performance driver is the feature engineering layer (Table 4).

summarizes the comparison. Random Forest attains the highest pooled AUC (0.956); however, the pairwise differences are small in magnitude (ΔAUC $\le 0.006$; nominal DeLong $p=0.011$–$0.042$) and, after controlling the family-wise error rate with the Holm–Bonferroni procedure, only the comparison against LightGBM remains below the significance threshold. Differences in F1 are not significant (engine-level bootstrap, $p\ge 0.12$). The four learners are therefore regarded as statistically equivalent from a practical perspective, and the model comparison plays a confirmatory role: it verifies that the pipeline’s performance is robust to the choice of ensemble learner, rather than establishing algorithmic superiority. Random Forest is adopted as the final model because it ranks first numerically in every metric under both evaluation protocols. More importantly, all models perform better under this protocol than under the chronological split, confirming that cross-engine generalization is not the binding difficulty for this class of data.

To quantitatively evaluate the individual contributions of cluster-based segmentation and sliding-window feature engineering (FE), an ablation study was conducted under the engine-grouped protocol, which, unlike the chronological split of the previous version, proved capable of separating the components. Four configurations of the proposed Random Forest model were compared: (i) M1, using only raw database features without clustering; (ii) M2, incorporating feature engineering but omitting clustering; (iii) M3, incorporating raw features and cluster-based segmentation; and (iv) M4, representing the full pipeline (both clustering and FE features).

As shown in

Table 4. Ablation study under leave-engine-group-out validation (Random Forest): contribution of feature engineering (FE) and cluster-based segmentation. p-values: DeLong test vs. the full configuration M4.

Configuration	AUC	Recall	F1-Score	Precision	p vs. M4
M1: Raw features, no clustering	0.900	86.1%	84.0%	82.0%	<10−26
M2: FE features, no clustering	0.956	90.2%	89.2%	88.3%	0.511
M3: Raw features + clustering	0.900	85.8%	83.7%	81.7%	<10−26
M4: FE features + clustering (full)	0.956	90.7%	89.6%	88.5%	—

Note: sliding-window feature engineering contributes ΔAUC $=+0.056$ ($p\approx {10}^{-27}$) on engines unseen during training; the cluster variable adds no discriminative gain and is retained for descriptive fleet characterization.

, the sliding-window feature engineering is the decisive component: it raises the pooled AUC from 0.900 (M1) to 0.956 (M4), a gain of ΔAUC $=+0.056$ with $p\approx {10}^{-27}$ (DeLong), accompanied by a Recall improvement from 86.1% to 90.7%. Because this gain is measured entirely on engines unseen during training, it demonstrates that the engineered trend features capture degradation signal that generalizes across assets rather than memorizing engine-specific histories; the effect is furthermore model-agnostic, as an equivalent gain is obtained when the ablation is repeated with LightGBM (ΔAUC $=+0.068$, $p\approx {10}^{-26}$). In contrast, the cluster-assignment variable adds no measurable discriminative power (M4 vs. M2: ΔAUC $=+0.0004$, $p=0.51$). The claim advanced in the previous version of this manuscript, that clustering improves predictive performance, is therefore withdrawn: the K-Means segmentation is retained in the pipeline exclusively for its descriptive and operational value (Section 3.7), namely the identification of two validated usage profiles that inform differentiated maintenance planning.

4.5. Model Interpretability

To characterize the decision structure of the final Random Forest model, a global feature importance analysis was conducted from two complementary perspectives: mean decrease in impurity (Gini importance, computed on the training ensemble) and permutation importance (mean AUC degradation on the held-out chronological test set when each feature is randomly permuted, 10 repetitions). The permutation perspective is reported because impurity-based rankings can overstate the relevance of high-cardinality features, whereas permutation importance measures each variable’s contribution to out-of-sample discrimination directly. Figure 7 details the top 15 features under both criteria.

Both criteria agree that the decision structure is dominated by the engineered degradation-trajectory features: the cumulative failure rate, the number of failures in the last three interventions, the historical failure rate, and the mean hours elapsed per corrective failure occupy the leading positions, followed by cumulative usage measures such as total accumulated hours and engine age. This ranking is consistent with, and explains, the ablation result of Section 4.4: the variables that the model relies on most heavily are precisely those constructed by the feature engineering layer whose removal degrades AUC by 0.056. We note that this analysis describes which inputs the fitted model uses; it does not by itself establish claims about the physical failure mechanisms involved.

Consistent with the ablation findings, the cluster-assignment variable does not appear among the top-ranked predictors of the final model (it ranks 38th of 47 by impurity importance). The operational profiles identified in the unsupervised stage (Section 3.7) therefore retain descriptive value for fleet segmentation and maintenance planning, but the earlier interpretation that the cluster signal carries measurable predictive weight is not supported once the label-associated event descriptors are removed, and is accordingly withdrawn.

5. Conclusions

This work developed and validated a predictive maintenance model tailored to the operating conditions of stationary combustion engines in the agricultural sector. Guided by the CRISP-DM framework, sliding-window feature engineering transformed raw transactional operational data into trend indicators capable of capturing recent mechanical stress patterns, overcoming the fundamental limitation of static approaches based solely on accumulated operating hours. This contribution is now supported statistically: under leave-engine-group-out validation, the feature engineering layer raises AUC from 0.900 to 0.956 ($p\approx {10}^{-27}$, DeLong test), an effect that is model-agnostic and measured entirely on engines unseen during training. This demonstrates that the quality and relevance of input variables, rather than the choice of ensemble algorithm, is the decisive factor in predictive performance for agricultural maintenance contexts.

A key structural finding confirmed operational heterogeneity within the fleet. K-Means clustering, validated with k = 2 using four internal metrics, revealed two distinct usage profiles: a standard operation cluster and an intensive use cluster characterized by significantly higher daily usage intensity and accumulated hours trend. The ablation study showed that this segmentation does not add discriminative power to the supervised model; its contribution is descriptive and operational, indicating that a one-size-fits-all maintenance policy is insufficient for fleets with diverse loading conditions and providing an interpretable basis for differentiated planning.

In the comparative algorithmic evaluation, conducted with symmetric tuning budgets and nonparametric significance testing under two protocols, the four ensemble learners were statistically indistinguishable. Random Forest attained the best numerical performance (AUC 0.956 with 90.7% recall under engine-grouped validation and AUC 0.903 with 84.2% recall under the chronological deployment split) and was adopted as the final model on the basis of this consistent numerical primacy across all metrics and protocols, not on a claim of algorithmic superiority. The model comparison is thus confirmatory in nature: it verifies that, once the feature space is properly constructed, well-tuned bagging and boosting ensembles converge to similar performance on administrative maintenance records, and that the pipeline’s effectiveness does not hinge on the choice of learner.

The resulting model offers a 60-day failure prediction window with 84.2% recall and 84.8% precision under the deployment protocol, supporting a shift from reactive and time-based maintenance toward a proactive, data-driven strategy with potential implications for equipment availability during agricultural production peaks. The quantification of economic impact through cost-per-downtime indicators constitutes a relevant direction for future work.

The contribution of this work is methodological and applied rather than algorithmic. The systematic evaluation of state-of-the-art ensemble methods under domain-specific conditions (sparse administrative records, operational heterogeneity, and temporal data structure) provides the empirical foundation for the predictive core of a future agricultural maintenance management application, and yielded a methodological finding of independent interest: under a dual-protocol evaluation, temporal extrapolation, not cross-engine generalization, is the dominant difficulty when predicting failures from administrative maintenance records, which has direct implications for how predictive maintenance models on such data should be validated. This applied benchmarking approach, well-established in the predictive maintenance literature [4,20], serves a concrete engineering purpose: reducing the gap between available ML techniques and their deployment in resource-constrained agricultural environments.

Despite these promising results, several limitations should be acknowledged. The model was developed and validated exclusively on historical transactional records from 59 stationary engines located in a specific geographic and agroclimatic context in Ecuador, which may constrain its direct generalizability to fleets under substantially different environmental or mechanical conditions. Generalization to engines unseen during training, identified as an open question in the previous version of this work, has now been evaluated directly through the leave-engine-group-out protocol (Section 4.4); the remaining open question is generalization to entirely different fleets, regions, and record-keeping practices. Likewise, the study does not define or evaluate a taxonomy of failure types (e.g., progressive wear versus sudden accidental failures); constructing a domain-validated typology from activity descriptions and testing the model’s ability to separate such classes remains beyond the present scope. Furthermore, the dataset relied on administrative maintenance records rather than continuous sensor measurements, meaning that variables with high diagnostic potential such as operating temperature, vibration signatures, or oil pressure were not available as model inputs. This reflects a deliberate design constraint rather than an oversight: the target deployment environment lacks IoT sensor infrastructure, and the applied objective is to maximize predictive performance using only the data that field technicians currently generate. This absence of real-time physical signals likely introduces a ceiling on the model’s sensitivity that cannot be overcome through feature engineering alone.

Future research should prioritize the integration of IoT sensor infrastructure to capture continuous operational variables, including temperature, vibration, and fuel consumption that reflect the internal physical state of the engine rather than its accumulated service history. This enrichment of the input space would allow the model to detect abrupt failure modes that the current 60-day horizon is structurally ill-suited to anticipate. Given the resulting trade-offs among sensor cost, installation feasibility, and diagnostic value, Multi-Criteria Decision Analysis methods (e.g., AHP, TOPSIS, fuzzy hybrids), whose growing methodological maturity and cross-domain applicability are documented in a recent bibliometric review [25], offer a structured framework for prioritizing which sensor variables and integration strategies to adopt.