Physics-Informed Neural Network-Assisted Imaging for Oil Palm Fruit Ripeness Classification

Abstract

In this work, we present a Physics-Informed Neural Network (PINN) framework for the classification of oil palm fresh fruit bunch (FFB) ripeness using RGB images. Unlike conventional Convolutional Neural Networks (CNNs) that learn solely from visual patterns, the proposed PINN integrates a physics-based index—derived from the red-to-green pixel intensity ratio—directly into the network architecture and loss function. This hybrid design embeds wavelength-dependent physical knowledge related to chlorophyll degradation during ripening, enabling the model to learn more robust and generalizable features even with limited and imbalanced training data. The PINN model achieves a peak accuracy of 0.73, outperforming the purely data-driven CNN baseline (0.68) by a margin of 5%. Overall, the PINN demonstrates superior performance in minority-class detection and maintains stable convergence under three different lighting conditions (different light spectra). These results highlight the effectiveness of integrating domain-specific physical insights into deep learning models, offering a promising pathway toward reliable, non-destructive, and automated ripeness assessment for agricultural applications.

1. Introduction

The accurate and objective determination of ripeness in oil palm fresh fruit bunches (FFBs) is a critical factor influencing both oil yield and quality, holding significant economic importance for the global agricultural industry [1]. Traditional ripeness assessment methods are predominantly manual, subjective, and labor-intensive, frequently leading to inconsistencies, human error, and potential economic losses. Consequently, there is a strong and pressing demand for automated, non-destructive, and objective quality control methods [2].

In recent years, deep learning, particularly via Convolutional Neural Networks (CNNs), has emerged as a powerful tool for image-based classification tasks in agriculture. Various CNN-based architectures have been successfully applied for FFB ripeness classification [3,4]. These models excel at learning complex visual patterns directly from image data. However, purely data-driven CNNs face significant practical challenges in real-world agricultural settings. Their performance is highly contingent on large, diverse, and well-annotated datasets [5,6]. Acquiring such extensive and balanced datasets is often impractical in agriculture. Standard CNNs may therefore struggle to generalize, especially when faced with (1) limited or imbalanced training data and (2) variations in environmental conditions, such as non-uniform lighting and shadows. These limitations highlight a gap in “black-box” deep learning approaches, which often fail to incorporate known domain-specific or physical knowledge that governs the outcome of the modeling.

Physics-Informed Neural Networks (PINNs) offer a promising hybrid modeling paradigm to bridge this gap. By integrating governing physical principles or domain knowledge directly into the neural network architecture—either as input features or as constraints within the loss function—PINNs can significantly enhance model robustness, improve data efficiency, and provide more generalizable solutions, particularly when training data is scarce.

Most research on PINNs is currently focused on simulation and inversion problems in mechanical engineering (e.g., computational fluid dynamics, finite difference/element) [7] and, to a lesser extent, in biomedical sciences [8,9]. To our knowledge, the use of PINNs for agricultural applications remains scarce.

The use of color-based pixel intensities has been widely adopted for ripeness classification for various fruits, including oil palm fruits [10,11]. This can be attributed to the strong correlation between the detected color intensities and the pigment concentrations in the fruit epidermis tissues [12,13]. However, a classification model that solely relies on color intensity has limitations, as this simple indicator indiscriminately includes noise from non-fruit elements such as thorns, empty fruit sockets, stains, or specular highlights, potentially leading to erroneous predictions.

This study investigates the potential of a PINN model to achieve improved classification of oil palm FFB ripeness. We propose an easy-to-implement hybrid model that fuses deep visual features extracted from a standard CNN with a physical-based metric index—the ratio of red to green pixel values, γ_RG. This ratio is an indirect, yet effective, measure of this underlying pigment change. The proposed PINN model aims to leverage the respective strengths of both methods, balancing the robust, high-level visual feature extraction of the CNN against the reliable, low-level physical guidance provided by γ_RG. The model is further guided by a loss function that regularizes the network’s predictions to be consistent with this underlying physical heuristic. In this investigation, the performance of the proposed PINN will be compared against a standard, purely data-driven CNN. The technique demonstrated is also applicable and extendable to other agricultural areas where physics-informed classifications are required.

2. Methods

2.1. Dataset Acquisition

A total of 112 images of oil palm fresh fruit bunches were acquired using a CMOS camera for this study. To ensure model robustness against environmental variations, images were captured under three distinct lighting conditions: a standard halogen lamp, a halogen lamp filtered by a blue filter, and a halogen lamp filtered by a red filter (Figure 1).

The dataset was categorized into three classes of ripeness (see Figure 2): 18 unripe (Class 1), 49 underripe (Class 2), and 45 ripe (Class 3) bunches. To maintain high data quality and relevance, care was taken to ensure that for at least 50% of the images, the frame was predominantly filled with pure oil palm fruitlets, minimizing the presence of confounding elements such as thorns, empty fruit sockets, or stains. The numbers of unripe, underripe, and ripe samples with different lighting conditions are listed in

Table 1. Number of unripe, underripe, and ripe samples with different lighting conditions.

Ripeness Class	Halogen Lamp	With Red Filter	With Blue Filter	Total
Unripe	6	5	7	18
Underripe	14	18	17	49
Ripe	16	15	14	45

.

2.2. Data Preprocessing and Feature Extraction

For Physics-Informed Neural Networks (PINNs), a specific feature is extracted from the raw images to serve as the physical indicator. For each image, the median pixel values of the red (R) and green (G) channels were computed. The median was used instead of the mean because it is less affected by extreme values originating from non-fruit artifacts such as thorns, empty fruit sockets, stains, or specular reflections. Using these median values, the colorimetric index γ_RG is calculated based on the red-to-green ratio (R/G). This ratio serves as the scalar physical indicator for the PINN model. As a wavelength-dependent parameter, γ_RG captures the color transition from dark to red, which is primarily driven by the reduction in chlorophyll content during ripening [12,13]. Importantly, this ratio is inherently more robust and less sensitive to variations in ambient lighting intensity. In this investigation, a total of 112 RGB images were prepared; 62 of them were randomly selected for the training set, while the remaining 50 images were reserved for the testing set (Due to the small unripe sample sets, 13 random unripe samples were always allocated to the training data set). A total of 15 random splits were carried out to ensure generalization, and results are reported in Section 3.

2.3. Model Architectures

Two distinct models were developed in this study: a Convolutional Neural Network (CNN) and Physics-Informed Neural Network (PINN).

2.3.1. Convolutional Neural Network (Pure CNN)

This conventional CNN baseline model uses only fruit images. Its architecture includes a convolutional base with three Conv2D layers (8, 16, and 32 filters, respectively) and MaxPooling2D layers to extract visual features. These features are then flattened and passed to a classifier head with two Dense layers for final classification into the three ripeness stages (Figure 3).

2.3.2. Physics-Informed Neural Network (PINN)

A PINN can be seen as a hybrid model that fuses information from both the images and the physical parameter, γ_RG (see Figure 4). The image is first processed by the same convolutional base as the CNN. The resulting feature vector is then concatenated with γ_RG. This combined, multimodal feature vector is then processed by the classifier head to produce the final prediction, allowing the model to learn the interplay between visual appearance and the underlying physical color index. Custom training loops based on the following loss function were implemented:

$$L_{total}=L_{cls}+\lambda \times L_{phys}$$

(1)

where $L_{total}$ is total loss and $L_{cls}$ is the sparse categorical cross-entropy, which measures the error in the model’s classification based on the actual data. $L_{phys}$ denotes the mean squared error (MSE) between the PINN’s predicted class and the class predicted by the γ_RG linear model equations (Equation (2)).

$$L_{phys}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(C_{phys}^i-{\widehat{E}}_{PINN}^i\right)}^2$$

(2)

where N is the number of samples in the batch and $C_{phys}^i$is the deterministic class index (a hard value of 0, 1, or 2) predicted by γ_RG for the i-th sample.${\widehat{E}}_{PINN}^i$is the expected class index predicted by the PINN for the i-th sample, calculated as a weighted average of the class indices (Equation (3)), and $P_j^i$ is the probability assigned to class $j$ for the i-th sample.

$${\widehat{E}}_{PINN}^i=\sum _{j=0}^{2}j·P_j^i$$

(3)

This term constrains the network’s predictions to be consistent with the physical model. $L_{phys}$ can be any function that is able to classify the ripeness stage. A comprehensive summary of all symbols is provided in

Table 2. Comprehensive summary of all symbols.

Symbol	Definition
${\gamma }_{RG}$	Median of red-to-green ratio from raw image
$L_{total}$	Total loss
$L_{cls}$	Sparse categorical cross-entropy loss
$\lambda$	Physics loss weighting coefficient
$C_{phys}^i$	Deterministic class index predicted by γRG for i-th sample
${\widehat{E}}_{PINN}^i$	Expected class index predicted by PINN for i-th sample
$P_j^i$	Probability assigned to class $j$ for i-th sample

, while the network architecture and training settings for the CNN and PINN are detailed in Table 3. In our case, linear equations are fit using least squares regression from the training ${\gamma }_{RG}$ data (see Table 4) and are limited to the range of [0, 3], as any value greater than 3 is clipped to 3. λ is physics loss weighting coefficient, a hyperparameter that balances the influence of the data-driven classification loss and the physics-based regularization term.

2.4. Training and Validation

For both the CNN and PINN models, the dataset was partitioned using an 80/20 stratified random split to maintain proportional representation of all three ripeness classes (unripe, underripe, ripe) in both the training and validation sets. This ensured that class imbalance did not bias model performance. All models were trained with a batch size of 16 using the Adam optimizer (learning rate = 0.001) [14]. To mitigate the issue of overfitting and improve generalization, early termination with a patience of 10 epochs was applied, in which the total validation loss for the PINN was carefully monitored and assessed in each epoch. The training was halted when no further improvement was observed within the patience window, and the model weights corresponding to the best validation performance were registered (Table 3).

Table 3. Summary of network architecture and training settings for CNN and PINN.

Category	Details
CNN Architecture	Input: Image (128, 128, 3) Layers: 1. Conv2D (8 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 2. Conv2D (16 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 3. Conv2D (32 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 4. GlobalAveragePooling2D 5. Dense (32 units, ReLU, L2 reg) 6. Dropout (0.4) 7. Output Dense (Softmax)
PINN Architecture	Inputs: Image (128, 128, 3) and Scalar ${\gamma }_{RG}$ Value (1) Structure: • Image branch identical to CNN • Concatenates image features with the scalar ${\gamma }_{RG}$ input • Followed by same Dense block (32 units) → Dropout → Output
Optimizer	Adam (used for both CNN and PINN)
Learning Rate	0.001 (used for both CNN and PINN)
Batch Size	16
Epochs	500 (maximum; early stopping triggered for all training sets before this limit was reached)
Augmentation	None
Regularization	• L2 Regularization: Factor of 0.01 on 32-unit Dense layer • Dropout: Rate of 0.4 • Early Stopping: Patience of 10 epochs, restoring best weights. • Physics Loss: Weighted regularization applied to PINN predictions based on linear regression constraints

Table 3. Summary of network architecture and training settings for CNN and PINN.

Category	Details
CNN Architecture	Input: Image (128, 128, 3) Layers: 1. Conv2D (8 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 2. Conv2D (16 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 3. Conv2D (32 filters, 3 × 3, ReLU) → MaxPooling2D (2 × 2) 4. GlobalAveragePooling2D 5. Dense (32 units, ReLU, L2 reg) 6. Dropout (0.4) 7. Output Dense (Softmax)
PINN Architecture	Inputs: Image (128, 128, 3) and Scalar ${\gamma }_{RG}$ Value (1) Structure: • Image branch identical to CNN • Concatenates image features with the scalar ${\gamma }_{RG}$ input • Followed by same Dense block (32 units) → Dropout → Output
Optimizer	Adam (used for both CNN and PINN)
Learning Rate	0.001 (used for both CNN and PINN)
Batch Size	16
Epochs	500 (maximum; early stopping triggered for all training sets before this limit was reached)
Augmentation	None
Regularization	• L2 Regularization: Factor of 0.01 on 32-unit Dense layer • Dropout: Rate of 0.4 • Early Stopping: Patience of 10 epochs, restoring best weights. • Physics Loss: Weighted regularization applied to PINN predictions based on linear regression constraints

This procedure was repeated for each value of λ, and the best-performing validation model for each setting is presented in the following section. A comprehensive evaluation was then conducted on a held-out test set, using confusion matrices to quantitatively compare the predictions of the PINN with those of the CNN. Table 4 tabulates linear model equations with slope, intercept, and the R² of 15 randomly split training sets’ data.

Table 4. RG-based linear model equations ($m$ is the slope and c is the intercept of the linear equation) and R² acquired from 15 randomly split training sets’ data.

Linear Model Equations	$\mathit{R}\mathit{i}\mathit{p}\mathit{e}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\mathit{I}\mathit{n}\mathit{d}\mathit{e}\mathit{x}=\mathit{m}\times {\mathit{\gamma }}_{\mathit{R}\mathit{G}}+\mathit{c}$ m ∈ [2.04, 2.53], c ∈ [−2.65, −1.84]
R2	R2 ∈ [0.61, 0.71]

Table 4. RG-based linear model equations ($m$ is the slope and c is the intercept of the linear equation) and R² acquired from 15 randomly split training sets’ data.

Linear Model Equations	$\mathit{R}\mathit{i}\mathit{p}\mathit{e}\mathit{n}\mathit{e}\mathit{s}\mathit{s}\mathit{I}\mathit{n}\mathit{d}\mathit{e}\mathit{x}=\mathit{m}\times {\mathit{\gamma }}_{\mathit{R}\mathit{G}}+\mathit{c}$ m ∈ [2.04, 2.53], c ∈ [−2.65, −1.84]
R2	R2 ∈ [0.61, 0.71]

3. Results and Discussion

3.1. The Impact of Physics Loss Weighting Coefficient

The performance of the PINN is critically sensitive to the physics loss weighting coefficient, λ, which is used in the composite loss function (Equation (1)). This function creates dual-objective optimization, balancing the physics-based loss function, L_phys, against the data-driven one, L_cls (see Figure 5).

Optimal performance is observed at low λ values (0.1–1.5), yielding accuracies of 0.70–0.73 compared to the baseline CNN’s 0.68. The best result of 0.73 ± 0.05 is achieved at λ = 0.6, representing a synergistic balance between the competing losses. L_phys stabilizes the model above the robust 0.69 baseline, while L_cls continues to guide the extraction of discriminative visual features, producing a hybrid effect that outperforms either component alone.

At higher λ values (e.g., λ ≥ 5), accuracy begins to decline as the imbalance between the two loss components disrupts the optimization process. For instance, at λ = 5 and λ = 50, the recorded accuracy drops to 0.69 and 0.53, respectively. In these cases, the network fails to converge on a solution that simultaneously satisfies the image data and the linear physical model, resulting in a sub-optimal equilibrium where neither objective is sufficiently minimized.

3.2. CNN vs. PINN Per-Class Performance

The per-class metrics further highlight the superior performance of the PINN (λ = 0.6) over the conventional CNN. As shown in

Table 5. Precision, recall, F1-score, and support (mean ± standard deviation) for CNN and PINN (λ = 0.6, 5, and 50).

Model	Class	Precision	Recall	F1-Score	Support
CNN	Class 1	0.41 ± 0.11	0.76 ± 0.22	0.51 ± 0.12	5.00 ± 0.00
	Class 2	0.69 ± 0.08	0.68 ± 0.08	0.68 ± 0.05	23.60 ± 2.12
	Class 3	0.87 ± 0.07	0.67 ± 0.12	0.75 ± 0.07	21.40 ± 2.12
PINN (λ = 0.6)	Class 1	0.59 ± 0.21	0.64 ± 0.26	0.56 ± 0.15	5.00 ± 0.00
	Class 2	0.69 ± 0.07	0.79 ± 0.10	0.73 ± 0.06	23.60 ± 2.12
	Class 3	0.88 ± 0.07	0.68 ± 0.12	0.76 ± 0.07	21.40 ± 2.12
PINN (λ = 5)	Class 1	0.52 ± 0.21	0.61 ± 0.20	0.53 ± 0.16	5.00 ± 0.00
	Class 2	0.66 ± 0.06	0.73 ± 0.12	0.69 ± 0.07	23.60 ± 2.12
	Class 3	0.85 ± 0.09	0.67 ± 0.12	0.74 ± 0.06	21.40 ± 2.12
PINN (λ = 50)	Class 1	0.26 ± 0.07	0.93 ± 0.12	0.40 ± 0.07	5.00 ± 0.00
	Class 2	0.33 ± 0.37	0.12 ± 0.15	0.17 ± 0.21	23.60 ± 2.12
	Class 3	0.70 ± 0.06	0.87 ± 0.06	0.77 ± 0.04	21.40 ± 2.12

, the impact of the PINN is most noticeable in Class 1 (Unripe). The CNN struggled significantly with this class, with a lower precision of 0.41 ± 0.11. In contrast, the PINN recorded a precision of 0.59 ± 0.21. This indicates that the PINN is more robust at identifying minority classes or features than the standard CNN, likely indicating that the training data in this class is limited.

For Class 2 (Underripe) and Class 3 (Ripe), both the PINN (λ = 0.6) and CNN achieved similar precision, but the PINN (λ = 0.6) demonstrated superior reliability, with higher recall and F1-scores, suggesting that the “physics-informed” aspect of the network helps reduce false negatives, ensuring that more actual instances of Class 2 are correctly identified. The PINN model can overcome this limitation by incorporating γ_RG into the loss function, introducing stronger wavelength-dependent constraints that provide a more robust physical basis for classification.

In general, increasing the λ values beyond 5 leads to a decline in per-class precision for the PINN model. For instance (Table 5), at λ = 5, the precisions for Classes 1, 2, and 3 drop to 0.50, 0.66, and 0.86, respectively. At λ = 50, the model’s performance collapses for all classes, recording precision values of only 0.26 for Class 1, 0.33 for Class 2, and 0.70 for Class 3.

The findings demonstrate the strengths of the PINN model. The integration of physical parameters and a physics-based loss function can improve classification accuracy. Notably, the PINN exhibits greater robustness in handling minority Class 1 despite the limited and imbalanced training dataset. This capability is particularly valuable for applications where access to large training datasets is constrained.

The model’s optimal performance within a specific λ range suggests the importance of balancing conventional color-based feature extraction with physics-informed constraints. This balance allows the PINN to leverage both visual information and underlying physical relationships, providing a significant advantage in scenarios where data imbalance is a persistent challenge in real-world classification tasks.

3.3. CNN vs. PINN Under Different Lighting Conditions

We compared PINN (λ = 0.6) and CNN performance under different lighting environments, with specific results for halogen (

Table 6. Precision, recall, and F1-score (mean ± standard deviation) for CNN and PINN (λ = 0.6) under halogen lamp (original).

Halogen Lamp (Original)
	PINN (λ = 0.6)			CNN
Class	Precision	Recall	F1-Score	Precision	Recall	F1-Score
1	0.36 ± 0.47	0.33 ± 0.45	0.33 ± 0.43	0.46 ± 0.47	0.47 ± 0.48	0.44 ± 0.44
2	0.69 ± 0.10	0.92 ± 0.08	0.78 ± 0.08	0.68 ± 0.13	0.84 ± 0.12	0.75 ± 0.12
3	0.93 ± 0.09	0.78 ± 0.15	0.84 ± 0.09	0.89 ± 0.11	0.78 ± 0.16	0.82 ± 0.11

), blue-filter (

Table 7. Precision, recall, and F1-score (mean ± standard deviation) for CNN and PINN (λ = 0.6) under halogen lamp with blue filter.

Halogen Lamp with Blue Filter
	PINN (λ = 0.6)			CNN
Class	Precision	Recall	F1-Score	Precision	Recall	F1-Score
1	0.47 ± 0.29	0.83 ± 0.31	0.54 ± 0.25	0.32 ± 0.15	0.98 ± 0.09	0.46 ± 0.17
2	0.63 ± 0.12	0.71 ± 0.18	0.65 ± 0.12	0.60 ± 0.16	0.52 ± 0.18	0.53 ± 0.13
3	0.98 ± 0.06	0.46 ± 0.21	0.59 ± 0.19	0.99 ± 0.04	0.47 ± 0.18	0.61 ± 0.16

), and red-filter (

Table 8. Precision, recall, and F1-score (mean ± standard deviation) for CNN and PINN (λ = 0.6) under halogen lamp with red filter.

Halogen Lamp with Red Filter
	PINN (λ = 0.6)			CNN
Class	Precision	Recall	F1-Score	Precision	Recall	F1-Score
1	0.52 ± 0.43	0.57 ± 0.46	0.52 ± 0.40	0.49 ± 0.39	0.67 ± 0.45	0.52 ± 0.36
2	0.76 ± 0.13	0.77 ± 0.13	0.75 ± 0.08	0.80 ± 0.15	0.74 ± 0.11	0.76 ± 0.07
3	0.77 ± 0.13	0.77 ± 0.21	0.75 ± 0.14	0.79 ± 0.14	0.74 ± 0.21	0.73 ± 0.13

) conditions.

Under illumination using the original halogen lamp, the results indicate that the PINN model achieves superior performance in Classes 2 and 3 across all evaluated metrics, including precision, recall, and F1-score. Specifically, for Class 3, the PINN reached an F1-score of 0.84 ± 0.09, outperforming the CNN’s 0.82 ± 0.11. Conversely, the CNN exhibited better performance in Class 1, which corresponds to unripe fruit. This phenomenon can be attributed to the spectral profile of the halogen lamp, where intensity increases steadily with wavelength. The high-absorption characteristics of unripe fruit bunches in this range result in a muted spectral response, creating a lack of contrast that hinders the PINN’s ability to distinguish specific color features and wavelength signatures. In this data-sparse environment, the CNN’s standard feature extraction proves more resilient than the physics-constrained approach.

The application of a blue filter shifted the performance advantage toward the PINN, particularly in Classes 1 and 2. In Class 1, the PINN achieved a precision of 0.47 ± 0.29 compared to the CNN’s 0.32 ± 0.15, representing a significant gain in reliability. While the CNN maintained a marginal lead in Class 3 F1-score (0.61 versus 0.59), the PINN’s general improvement stems from the blue filter’s ability to enhance spectral distinctions within the 550–650 nm and 650–780 nm ranges. This increased contrast facilitates superior feature extraction, allowing the physics-informed architecture to better model the underlying reflectance properties of the fruit and leading to higher overall classification accuracy.

In the case of the red filter, the CNN demonstrated superior performance in Classes 2 and 3, whereas the PINN maintained greater precision only in Class 1. The red filter effectively blocks shorter wavelengths, permitting only the “red” region of the spectrum to pass. This isolation enhances the characteristic color features of ripeness, providing the CNN with a highly focused and clear set of visual indicators to facilitate accurate decision-making. Under these specific conditions, the PINN’s additional physical constraints may become redundant or even introduce spectral noise, resulting in lower performance relative to the CNN in the ripened classes. However, the PINN’s slight advantage in Class 1 (0.52 ± 0.43 precision) suggests that it remains more effective than the CNN when dealing with the high-absorption, low-feature profile of unripe fruit, even under filtered-light conditions.

4. Conclusions

This study successfully demonstrates the effectiveness of a Physics-Informed Neural Network for the classification of oil palm FFB ripeness. By fusing visual data from images with a physics-based colorimetric index (γ_RG), the proposed PINN model achieved a classification accuracy of 0.72, demonstrating superior performance compared to the conventional data-driven CNN (0.68), with an improvement of 5% even when using different light sources (different spectral compositions). Note that this improvement is achieved without any additional instruments, just with raw image data. The results underscore the value of integrating physical knowledge into deep learning frameworks to create more robust and accurate predictive models. This hybrid approach represents a significant step forward for developing automated, non-destructive quality control systems in the agricultural sector, promising to enhance efficiency and maximize economic returns.