Growth Stages Discrimination of Multi-Cultivar Navel Oranges Using the Fusion of Near-Infrared Hyperspectral Imaging and Machine Vision with Deep Learning

Abstract

To noninvasively and precisely discriminate among the growth stages of multiple cultivars of navel oranges simultaneously, the fusion of the technologies of near-infrared (NIR) hyperspectral imaging (HSI) combined with machine vision (MV) and deep learning is employed. NIR reflectance spectra and hyperspectral and RGB images for 740 Gannan navel oranges of five cultivars are collected. Based on preprocessed spectra, optimally selected hyperspectral images, and registered RGB images, a dual-branch multi-modal feature fusion convolutional neural network (CNN) model is established. In this model, a spectral branch is designed to extract spectral features reflecting internal compositional variations, while the image branch is utilized to extract external color and texture features from the integration of hyperspectral and RGB images. Finally, growth stages are determined via the fusion of features. To validate the availability of the proposed method, various machine-learning and deep-learning models are compared for single-modal and multi-modal data. The results demonstrate that multi-modal feature fusion of HSI and MV combined with the constructed dual-branch CNN deep-learning model yields excellent growth stage discrimination in navel oranges, achieving an accuracy, recall rate, precision, F1 score, and kappa coefficient on the testing set are 95.95%, 96.66%, 96.76%, 96.69%, and 0.9481, respectively, providing a prominent way to precisely monitor the growth stages of fruits.

1. Introduction

The navel orange (Citrus sinensis Osb.), a citrus fruit, is extensively farmed globally and is a significant cash crop in nations including China, Spain, and Brazil [1]. In China, Gannan navel oranges are abundant in Ganzhou City, Jiangxi Province, and are particularly favored by consumers due to their distinctive flavor and rich nutritional value [2]. As a seasonal fruit, identifying the ideal harvest timeframe for navel oranges is essential. Unripe navel oranges typically exhibit a pronounced sour flavor, inadequate sweetness, and diminished nutritional content. Conversely, overripe oranges tend to rapidly deplete internal nutrients and experience softened peeling, rendering them susceptible to damage and deterioration during transit. This diminishes their quality, compromises food safety, and influences market pricing [3]. Consequently, meticulous oversight regarding the growth stages of navel oranges and comprehending the attributes of each stage are crucial.

Conventional approaches for determining the growth stage of navel oranges predominantly rely on the experiential and visual assessments of fruit farmers, which are significantly affected by human subjectivity and environmental conditions, leading to inefficiency and inaccuracy [4]. MV technology necessitates the comprehensive annotation of sample photos and relies on the external attributes of navel orange samples, disregarding data on internal quality variations, resulting in an incomplete and erroneous evaluation of growth stages. Near-infrared spectroscopy (NIRS) technology relies on the molecular overtone and combination tone vibration properties of O-H, C-H, and C-O covalent bonds in sample components, and can provide essential information regarding the internal compositions of fruits and vegetables. Through the collection and analysis of these spectra, it can help in ascertaining various growth stages [5]. Nevertheless, NIRS is limited to differentiating growth stages through internal quality alterations and fails to offer direct visual insights regarding the appearance, morphology, and color variations of fruits and vegetables. Consequently, the exploration of a more comprehensive and precise methodology for assessing the growth stages of fruits and vegetables via the integrated analysis of external vision and internal quality metrics has become essential for the meticulous management of navel orange quality.

HSI technology is an advanced technique that integrates spectroscopy with digital imaging and encompasses extensive image and spectral information, resulting in a substantial data cube of spectral and image data [6,7]. Owing to its robust data representation characteristics, it is extensively utilized across several fields, including agriculture, remote sensing, food processing, and environmental monitoring. HSI technology is mostly employed in agricultural research to differentiate the growth stages of fruits and vegetables, in order to ascertain the ideal harvest period [8]. For instance, Benelli et al. utilized HSI technology in situ to monitor variations in internal soluble solid content (SSC) and assess the maturity condition of grapes [9]. Zhao et al. employed NIR HSI technology to categorize tomatoes at three distinct stages [10]. Shao et al. utilized HSI to differentiate tomatoes at various growth stages (green mature, color changing, semi-mature, and fully mature) [11]. To date, assessments of navel orange quality based on HSI technology have primarily targeted surface flaws [12] and pesticide residues [13]. Although distinguishing the ripeness of navel oranges based on HSI has been studied by some scholars, these efforts have lacked a comprehensive classification of the growth stages and depend exclusively on traditional machine-learning models for predictions [14].

As deep-learning technologies have advanced, researchers have progressively employed these methods for the examination of fruit and vegetable quality through HSI. In contrast to conventional machine-learning modeling techniques, deep-learning models significantly streamline the modeling process and exhibit superior performance in classification and regression tasks [15,16]. Varga et al. effectively differentiated the various ripeness stages of avocados and kiwifruit by integrating HSI technology with deep learning models [17]. Benmouna et al. utilized hyperspectral photography combined with a CNN to distinguish the various ripeness stages of apples and compared this approach with traditional machine-learning models, such as artificial neural networks (ANNs), support vector machines (SVMs), and K-nearest neighbors (KNNs) [18]. Zhou et al. utilized stacked autoencoders (SAEs) and CNN to differentiate the various ripeness stages of Cabernet Sauvignon grapes by analyzing their reflectance spectra in the range of 400–1029 nm, achieving classification accuracies of 100% for the calibration set and 94% for the testing set [19]. Han et al. acquired hyperspectral images of two kinds of avocados and employed partial least squares regression (PLSR) and CNN to forecast their ripening duration [20]. Gao and Su employed deep learning to assess strawberry maturity [21,22]. Zhang et al. determined the growth years of Puerariae Thomsonii Radix utilizing HSI in conjunction with CNN technologies [23]. To date, most of the research regarding the ripeness discrimination of fruits based on HSI techniques has depended on spectral information, and only single-branch CNN models have been employed. However, the discrimination accuracy for fruit ripeness based on the single-branch CNN model is limited by the use of single-modality spectral data. For a comprehensive analysis to accurately discriminate the growth stages of fruits, important internal and external features should be fully considered through the use of multi-modal approaches.

Although deep learning has been utilized to combine spectra with RGB images [24], spectra with hyperspectral images [25], and hyperspectral images with RGB images [26] for fruit and vegetable quality detection, no research has been reported that integrates spectra, hyperspectral images, and RGB images alongside deep-learning models to differentiate the growth stages of navel oranges. Moreover, research on fruit growth monitoring based on spectral or hyperspectral techniques has been conducted on the same cultivar [10,11]. As such, there are currently no relevant reports on the simultaneous discrimination of growth stages for multiple cultivars of navel oranges.

Precisely managing the growth stages and accurately controlling the harvest time of navel oranges are crucial to ensuring quality (flavor, color, and nutrients), production yield, and economic benefits, especially near the ripening period. This study aims to achieve a highly accurate and simultaneous discrimination of various navel orange growth stages with multiple cultivars by employing a fusion technique using HSI (i.e., spectra and hyperspectral images) and MV (i.e., RGB images), coupled with a deep-learning model with dual-branch multi-modal feature fusion that extracts and integrates features from various modalities, specifically NIR reflectance spectra, hyperspectral images, and RGB images. The main contents of this work are presented as follows: (1) We first develop an information acquisition system for the growth stages of navel oranges using NIR HSI combined with MV and collect reflectance spectra, hyperspectral images, and RGB images of Gannan navel oranges at distinct growth stages. (2) To ensure high discrimination accuracy, the preprocessing of reflection spectra, the selection of hyperspectral images with optimal wavelengths, and the registration of RGB images are carried out. (3) We develop a dual-branch multi-modal feature fusion deep-learning model by integrating a one-dimensional CNN with a two-dimensional VGG11 (1DCNN-VGG11) to discriminate the various growth stages of Gannan navel oranges. The performance of the model is assessed via five indices. (4) We compare the developed deep-learning model with various other machine-learning and deep-learning models based on single-modal and dual-modal data feature fusion, emphasizing the advantages of the established dual-branch model.

2. Materials and Methods

2.1. Samples

In this study, the growth stage discrimination of navel oranges mainly focuses on the fruit-swelling period, the color transformation period, and the ripening period. The time span of data collection was from 23 August 2024 to 10 December 2024. To achieve this objective, 795 navel oranges were collected across 10 batches with intervals of 11 days from the Gannan Navel Orange Planting Demonstration Base in Xinfeng County, Ganzhou City, Jiangxi Province. Each time, five cultivars of navel oranges were gathered—i.e., Newhall, Navelina, CaraCara, Gannan No. 1, and Gannan No. 5—with 15–18 fruits per cultivar. Due to substantial variations in the growth stages of various navel orange cultivars, we categorized the oranges into five distinct growth stages according to the length of fruit swelling, color transformation, and ripening periods, as well as the peel coloration of each cultivar, ranging from entirely green to entirely orange (i.e., fully green, predominantly green, predominantly yellow, predominantly orange, and fully orange), designated as Growth_1, Growth_2, Growth_3, Growth_4, and Growth_5, respectively. The definition of these five growth stages is as follows: Growth_1 is the fruit swelling period of navel oranges; its time span is from approximately August to early October. The navel orange peels are fully green, but the size gradually swells with time. Growth_2 is the color transformation period; its time span is from approximately early October to early November. The navel oranges gradually transition to the mature stage. At this stage, yellow coloring will appear on the navel orange peel. However, this stage cannot be considered the mature stage due to insufficient nutritional components, such as sugar content and nutrition. As time passes, the navel orange will gradually enter the ripening period, which starts around early November to middle December. This ripening period can be classified into three different stages, i.e., the early maturity stage (i.e., Growth_3), the middle maturity stage (i.e., Growth_4), and the fully mature stage (i.e., Growth_5).

Table 1. Five different growth stages for five cultivars of Gannan navel oranges and their numbers.

Cultivar	Growth_1	Growth_2	Growth_3	Growth_4	Growth_5	Numbers
Newhall	23 August– 5 October	6 October– 7 November	8 November– 18 November	19 November– 29 November	30 November– 10 December	159
Navelina	23 August– 16 October	17 October– 27 October	28 October– 8 November	9 November– 19 November	20 November– 10 December	159
CaraCara	23 August– 16 October	17 October– 7 November	8 November– 18 November	19 November– 29 November	30 November– 10 December	159
Gannan No.1	23 August– 13 September	14 September– 16 October	17 October– 27 October	28 October– 19 November	20 November– 10 December	159
Gannan No.5	23 August– 13 September	14 September– 16 October	17 October– 27 October	28 October– 19 November	20 November– 10 December	159

illustrates the five distinct growth stages of Gannan navel oranges and the quantities of each cultivar. In Table 1, each navel orange represents a batch due to content limitations.

2.2. Devices

The information acquisition system was constructed to collect hyperspectral data and RGB images of navel oranges, as presented in Figure 1a. The system comprises a NIR hyperspectral camera (Gaia Field, Dualix Spectral Imaging Co., Suzhou, China), eight halogen lights (50 W each), a translation stage, a dark box, and a computer with the SpecView software (Version: 2.9.3.18). The NIR hyperspectral camera can acquire hyperspectral data, comprising hyperspectral images and reflectance spectra at 860–1730 nm with a spectral resolution of approximately 1.7 nm. In addition, to fully take advantage of the color variations in navel oranges at various growth stages, a digital color camera was equipped for the synchronous acquisition of RGB images. Figure 1b shows a photograph of the system. The acquired HSI cube data of a typical navel orange is presented in Figure 1c. In the HSI cube data, the x and y coordinates form the hyperspectral image of the navel oranges, and the z coordinate indicates the spectral channels.

2.3. Hyperspectral Data Correction

To eliminate the noise interference caused by uneven light source intensity and dark current in the HSI device, with the aim of obtaining relatively stable spectral reflectance and hyperspectral images, it is necessary to perform black and white calibration on the collected HSI data. The black and white calibration formula is provided in Equation (1):

$$I={\displaystyle \frac{I_{\mathrm{r}\mathrm{a}\mathrm{w}}-D}{W-D}}$$

(1)

where I_raw is the original hyperspectral image, D is the dark background image captured with the lens covered, W is the standard white reference image, and I is the corrected hyperspectral image.

2.4. ROI Extraction and Sample Partitioning

This study utilized the ENVI 5.6 remote-sensing image analysis software to process the HSI data of navel oranges for data extraction. In this study, the volumes and sizes of navel oranges are different due to the different growth stages and cultivars. Moreover, during experiments, the HSI and RGB images of some navel oranges were not fully acquired due to their placement beyond the field of view of the hyperspectral camera during acquisition, which results in partial image loss in HSI and RGB regions. At the same time, during experiments on acquiring HSI and RGB images, all navel oranges come with their own leaves, which results in some regions of the navel orange’s surface being covered by their own leaves. To eliminate the impact of leaves and incomplete regions, and unify the sizes of the HSI and RGB images of growth stage discrimination modeling, a 60 × 60-pixel square region that can cover key color features of each navel orange’s hyperspectral image was selected as the ROI rather than the entire navel orange surface, as shown in Figure 1c, and the average spectrum of the ROI was provided as the spectral data for each sample. Furthermore, due to the potential external influences on the HSI data acquisition process, which may result in data abnormalities within some samples, we utilized the Mahalanobis distance approach [27] to identify spectral outliers from 775 valid samples. Ultimately, 740 valid samples were retained. As with the HSI data, corresponding RGB images of these 740 navel oranges were utilized. Moreover, a 60 × 60-pixel region corresponding to that of the hyperspectral image was utilized to extract the ROI data of each RGB image.

To ensure the generalization ability of the discrimination model, a random division method was used to divide the 740 samples into a training set and a testing set at a ratio of 4:1. The specific division is detailed in

Table 2. Number of raw navel oranges, before removal, and division sets after removal for valid samples.

Growth Period	Raw Samples	Before Removal for Valid Samples	After Removal for Valid Samples
			Training Set	Testing Set	Total Number
Growth_1	270	255	205	42	247
Growth_2	180	178	131	37	168
Growth_3	81	79	57	17	74
Growth_4	120	119	91	24	115
Growth_5	144	144	108	28	136
Total number	795	775	592	148	740

. During cross-validation, the samples were partitioned based on five growth stage stratifications. A 10-fold cross-validation strategy was utilized.

2.5. Image Registration

Although hyperspectral and RGB images can be collected from the same viewpoint, those of the navel oranges were acquired using two cameras, resulting in dimensional differences. To solve this problem, image registration is crucial to ensure the accurate alignment of the specified regions of interest (ROIs) in the subsequent analysis. This study utilized pseudo-color images derived from the hyperspectral images as the reference standard, employing the OpenCV library to spatially align the RGB images to the hyperspectral images. Furthermore, to assess the quality of image registration, the structural similarity index (SSIM) [28] was chosen as the evaluation metric. An SSIM value approaching 1 represents better registration performance. The SSIM calculation formula is as follows:

$$\mathrm{S}\mathrm{S}\mathrm{I}\mathrm{M}(x,y)={\displaystyle \frac{[2{\mu }_x{\mu }_y+{\left(k_{1}L\right)}^2][2{\sigma }_{xy}+{\left(k_{2}L\right)}^2]}{[{\mu }_x^2+{\mu }_y^2+{\left(k_{1}L\right)}^2][{\sigma }_x^2+{\sigma }_y^2+{\left(k_{2}L\right)}^2]}}$$

(2)

where μ_x and μ_y are the means of images x and y, respectively; σ_x² and σ_y² are the variances of images x and y, respectively; σ_xy is the covariance between images x and y; L is the dynamic range of the pixel values; the default value of k₁ is 0.01; and the default value of k₂ is 0.03.

2.6. Data Processing

2.6.1. Spectral Preprocessing

Considering that the collected spectra are susceptible to interference from various factors, such as noise and baseline drift, to minimize the influence of uncertainties and irrelevant information variables, preprocessing the raw spectra is necessary before modeling [29]. In this study, several spectral pre-processing methods, including multiplicative scatter correction (MSC) [30], Savitzky–Golay smoothing (SG) [31], first derivative (FD) [31], and detrending (DT) [32], were employed to preprocess the reflectance spectra from the raw HSI data. To obtain the best spectral preprocessing method, partial least squares–discrimination analysis (PLS-DA) models were then established, and the accuracy of growth stage discrimination for navel orange samples in the testing set was compared between the spectral preprocessing methods.

2.6.2. Dimensionality Reduction of Hyperspectral Images

An HSI data cube comprises numerous wavebands, but only a portion of these wavebands accurately reflect the characteristics of different navel orange growth stages. Using hyperspectral images at all wavelengths for modeling would significantly increase the computational complexity of the model and consume substantial computing resources, leading to prolonged training times and low computational efficiency. Moreover, this could affect the accuracy of the growth stage discrimination. Therefore, dimensionality reduction for the HSI is essential to resolve these challenges. In this context, principal component analysis (PCA) [33] is a common dimensionality reduction algorithm for HSI.

2.7. Discrimination Modeling

2.7.1. CNN

CNN is one of the most prominent neural networks in deep learning, extensively utilized in pattern recognition, smart healthcare, precision agriculture, etc. [34]. As shown in Figure 2a, a typical CNN architecture consists of five main components: the input layer, the convolutional layer, the pooling layer, the fully connected layer, and the output layer. Specifically, the input layer receives raw data (such as images and signals) and passes it to the convolutional layer. The convolutional layer uses sliding convolution kernels (filters) to perform local perception on the input data, extracting features. These kernels can recognize various local patterns within the input data, such as edges, textures, and shapes. For one-dimensional data, one-dimensional convolutional neural networks (1DCNNs) employ one-dimensional convolution kernels to capture local features. For two-dimensional data, two-dimensional convolutional neural networks (2DCNNs) utilize two-dimensional convolution kernels to capture spatial local features. The pooling layer is used to downsample the output from the convolutional layer, reducing the dimensionality of the data and, thus, lowering the computational complexity. Pooling operations can be either max-pooling or average-pooling, aggregating local features to enhance model robustness and mitigate the risk of overfitting. Subsequently, the local features extracted by the convolutional and pooling layers are flattened and passed into the fully connected layer. The output layer produces the conclusive categorization findings derived from the activation function.

2.7.2. Proposed Novel Deep-Learning Model

To achieve high-accuracy discrimination for the growth stages of Gannan navel oranges, a lightweight dual-branch multi-modal feature fusion CNN model is proposed in this study. The model comprises two independent CNN branches. One branch is the spectral branch, which extracts NIR spectral features reflecting the different growth stages of Gannan navel oranges. The other is the image branch, which extracts visual features, including color and texture, from NIR hyperspectral and RGB images that reflect the growth stages of Gannan navel oranges. Subsequently, the feature information extracted from both branches is concatenated and processed through three fully connected layers, ultimately outputting the categories of different growth stages via the Softmax operation. The specific structure of the model is shown in Figure 2b.

In Figure 2b, the spectral branch takes one-dimensional NIR spectral data as the input and undergoes feature extraction through five convolutional and pooling layers. Each convolutional layer employs a convolution kernel size of 3 × 1, with the number of kernel channels successively set to 64, 128, 256, 512, and 512. The stride for the convolution operation is 1, and the activation function used in the activation layers is the ReLU function. The pooling layers perform max-pooling operations. The design of the convolutional layers in the spectral branch captures key features indicative of different navel orange growth stages through convolution operations while ensuring that important information is not lost, thereby enhancing feature extraction capabilities. In the image branch, the input consists of an NIR hyperspectral image and an RGB image after channel superposition. The convolutional and pooling layers in the image branch adopt the VGG network structure, a classic deep CNN architecture suitable for image feature extraction. Through this dual-branch CNN structure, the model can effectively capture critical feature information from multi-modal data (NIR spectral, NIR hyperspectral, and RGB images), aiming to achieve higher accuracy in discriminating among Gannan navel orange growth stages.

To validate the viability of the established model, we compared several classical machine learning models, i.e., PLA-DA [35], K-nearest neighbor (KNN) [36], support vector machine (SVM) [37], random forest (RF) [38], and back-propagation neural network (BPNN) [39]. All data processing and qualitative discriminant model construction for spectra, hyperspectral, and RGB images were implemented using the Python 3.9.12 programming language and the PyTorch 1.13.1 framework. Both model training and testing were conducted on a high-performance computing server (Brand: Inspur) equipped with a dual Intel(R) Xeon(R) Silver 4210R CPU @ 2.40 GHz, dual NVIDIA GeForce RTX 3090 GPUs, Visual Studio Code, and the Windows 10 operating system.

2.7.3. Model Evaluation

The performance of the navel orange growth stage discrimination models was evaluated using five macro average indices, i.e., accuracy (AC), recall (RC), precision (PC), F1 score (F1) [40], and kappa coefficient (kappa) [41]. Higher values indicate better model performance.

3. Results and Analysis

3.1. Results of Abnormal Sample Elimination

During the acquisition of HSI data, anomalies may arise due to instrumentation and environmental factors. To improve the discrimination performance of a model, it is imperative to remove these abnormal samples from the raw data. In this study, 35 anomalous samples were removed using the Mahalanobis distance method. Figure 3a depicts the raw spectra of all 775 samples, and Figure 3b shows the spectra after removing the abnormal samples. Comparing Figure 3a,b, it can be seen that the abnormal samples excluded via the Mahalanobis distance method exhibited systematic reflectance shift characteristics, with their spectra demonstrating persistently lower or higher intensities across multiple bands. These anomalies originated from environmental factors rather than variations in fruit quality. Specifically, during the hyperspectral imaging of multiple navel oranges within a dark box-controlled environment, uneven illumination caused certain samples to receive excessive lighting (resulting in elevated reflectance) while others appeared relatively darker (with reduced reflectance). By detecting multivariate deviations from the normal spectral cluster, the Mahalanobis distance effectively identified such systemic biases. Comparison of Figure 3a,b reveals that the spectral data via the operation of abnormal sample elimination becomes more concentrated and regularly distributed after removing the abnormal samples, reducing potential noise interference. This lays a solid foundation for constructing a more robust and precise discrimination model.

3.2. Results of Spectral Preprocessing

To mitigate the impact of external factors on the spectral data, four preprocessing methods, namely SG, FD, MSC, and DT, and their combinations, were applied. Notably, spectral results after the MSC treatment, in conjunction with other approaches, are shown in Figure 3c–f.

The growth stage discrimination results for the training and testing sets based on the PLS-DA model for different spectral preprocessing methods are shown in

Table 3. Discrimination results of PLS-DA models with different spectral preprocessing methods.

Preprocessing Method	PCs 1	Training Set	Testing Set				10-Fold CV
		AC (%)	AC (%)	RC (%)	PC (%)	F1 (%)	AC (%)
RAW	20	81.59	79.73	73.47	83.83	73.26	76.04
FD	19	91.55	78.38	77.06	78.66	77.58	79.06
DT	18	79.05	77.03	72.36	81.8	72.47	75.53
MSC	19	79.9	77.7	72.77	82.73	72.83	75.68
SG	19	79.39	77.7	72.2	82.75	71.94	75.53
FD + DT	20	90.2	79.73	77.59	80.93	78.41	78.21
FD + SG	19	82.94	78.38	73.79	82.85	74.45	76.69
DT + SG	16	77.87	75	70.51	80.68	70.9	75.19
MSC + FD	14	86.99	79.05	76.64	80.87	77.89	76.01
MSC + SG	17	78.55	78.38	73.67	82.81	73.66	74.68
MSC + DT	20	82.43	80.41	75.34	83.9	75.86	78.05

¹ PCs: Principal component numbers.

. Compared with the results of the model established with raw spectra (RAW), most were not improved after spectral preprocessing. Notably, the best performance was observed when the spectrum was preprocessed with a combination of MSC and DT, resulting in a testing set accuracy of 80.41%. Therefore, the combination of MSC and DT was identified as the optimal spectral preprocessing method in this study.

3.3. Results of Feature Wavelength Selection for Spectral Data

Following the preprocessing of navel orange reflectance spectra utilizing MSC combined with DT, feature wavelength selection was performed using CARS [42], LAR [43], UVE [44], and GA [45] to extract important features from the spectra. The results of different feature wavelength selection methods are shown in Figure 4 (marked with red dots). CARS selected 119 wavelengths. LAR selected 175 wavelengths. UVE selected 418 wavelengths, and GA selected 230 wavelengths.

The results of the navel orange growth stage discrimination utilizing several feature wavelength selection algorithms based on the PLS-DA model are presented in

Table 4. Discrimination results of different feature wavelength selection algorithms based on PLS-DA models.

Feature Wavelength Selection Method	PCs	Training Set	Testing Set				10-Fold CV
		AC (%)	AC (%)	RC (%)	PC (%)	F1 (%)	AC (%)
None	20	82.43	80.41	75.34	83.90	75.86	78.05
CARS	16	77.53	75.68	70.70	80.18	70.69	74.34
LAR	18	80.41	76.35	71.06	81.42	70.30	75.68
UVE	19	78.89	79.73	73.76	84.43	73.54	76.53
GA	17	79.90	79.05	73.51	83.81	72.80	76.71

. As indicated by Table 4, the best performance was achieved with no feature wavelength selection (i.e., None), with the testing set’s accuracy, recall, precision, and F1 score being 80.41%, 75.34%, 83.90%, and 75.86%, respectively. Therefore, this study did not employ feature wavelength selection methods but instead used the spectra preprocessed via MSC combined with DT directly for subsequent modeling analysis.

3.4. Results of Image Registration

To ensure that the selected ROI in RGB and hyperspectral images of the same oranges were consistent, we utilized the contour detection function (CDF) in the OpenCV library to first determine the position and size of the navel orange in the hyperspectral pseudo-color image. After that, the navel orange in the RGB image was adjusted to be the same size as that in the hyperspectral image, and it was placed on a white background image with the same size as the hyperspectral image, as shown in Figure 5. Subsequently, to evaluate the image registration accuracy, the hyperspectral pseudo-color image and the registered RGB image were transformed into grayscale images, and we computed their SSIM. The results showed that the SSIM was 0.9558, indicating good registration performance, which laid a foundation for subsequent ROI extraction from the same location in the hyperspectral and RGB images.

3.5. Result of Optimal Selection of Feature Wavelengths for Hyperspectral Images

This study takes a spectral perspective, selecting several optimal wavelengths based on the NIR reflectance differences in the average spectra of Gannan navel oranges at various growth stages. These wavelengths are then utilized to extract corresponding hyperspectral images as characteristic images of navel oranges. However, the average reflectance spectra in Figure 6a show that the wavelengths with significant reflectance differences are concentrated at the ends of the spectra. Despite potentially showing significant differences in reflectance, the data at both ends of the spectra are prone to fluctuations and deviations due to the dark current and noise interference from the hyperspectral camera detector. To avoid such errors, it is necessary to select other wavelengths that can reflect spectral differences with the various growth stages. Furthermore, as illustrated in Figure 6a, the disparities between neighboring wavelengths are insufficiently severe to adequately convey distinctive information about navel oranges. This study employed a strategy called average spectral reflectance interval maximum difference (ASRIMD) to tackle these issues. That is, we divided the entire spectral range into five intervals and identified the wavelength exhibiting the greatest reflectance disparity within each interval as the ideal characteristic wavelength for that interval. Figure 6b shows the ASRIMD with five intervals for Gannan navel oranges, where each color region represents the spectral reflectance difference for each interval. Based on this method, five characteristic wavelengths were selected, i.e., 901.3 nm, 1100.6 nm, 1237.8 nm, 1580.6 nm, and 1699.5 nm. The positions marked by dashed lines in Figure 6a indicate these five selected optimal characteristic wavelengths. The hyperspectral images of navel oranges at these optimal wavelengths and the corresponding sample’s RGB images are shown in Figure 6c–h.

3.6. Results of Modeling Analysis

In this work, a dual-branch multi-modal feature fusion deep-learning model (Figure 2b), i.e., the 1DCNN-VGG11 model, was established to determine the growth stages of navel oranges of five different cultivars. To achieve satisfactory discrimination performance, the structures and hyperparameters of the 1DCNN-VGG11 model were optimized and adjusted. The loss function of the 1DCNN-VGG11 model was the cross-entropy function [46]. To optimize the spectral branch, models with varying numbers of layers and kernel sizes within the spectral branch were compared while keeping the graph branch unchanged. Figure S1a–d illustrates the iterative process of 1DCNN-VGG11 models with different spectral branch configurations using the testing set, where curves of distinct colors represent accuracy or loss under varying layer depths or kernel sizes. Figure S1a–d are provided in the Supplementary Materials. As depicted in Figure S1a, when varying only the number of layers (with other spectral branch structures fixed), all models incorporating three, four, five, and six convolutional layers exhibited an initial accuracy increase followed by convergence to a stable level. Concurrently, Figure S1b shows their losses decreased and subsequently stabilized. The best accuracies achieved were 93.24%, 94.59%, 95.95%, and 93.24%, respectively. Figure S1c shows that models with kernel sizes of one, three, five, and seven exhibit increasing accuracy followed by stabilization, while Figure S1d demonstrates decreasing loss converging to stability. The corresponding best accuracies were 91.22%, 95.95%, 93.92%, and 94.59%. The results indicate that the spectral branch configuration with a kernel size of three and five convolutional layers delivered the optimal performance. Consequently, we selected this architecture as the final spectral branch model to identify the growth stages of navel oranges.

While maintaining an identical spectral branch structure, we evaluated model performance based on variations in the number of convolutional layers and kernel sizes within the image branch. Figure S2a–d illustrate the iterative training process for models with different image branch architectures using the testing set, where accuracy and loss curves are represented by distinct colors. Figure S2a–d are provided in the Supplementary Materials. As depicted in Figure S2a, when modifying only the number of convolutional layers (8, 9, 10, and 11 layers) while keeping other image branch components fixed, the respective peak accuracies were 94.59%, 91.22%, 95.27%, and 95.95%. The accuracy curves indicate that models with layer counts other than 11 exhibited significant performance fluctuations (indicating unstable convergence). As shown in Figure S2b, other models exhibit consistently higher loss values than the 11-layer architecture. For experiments solely adjusting kernel size, models with kernel sizes of one, three, five, and seven were initially planned. However, due to the limited input image dimensions, employing kernel sizes of five or seven resulted in feature maps becoming prohibitively small after repeated convolution and pooling operations, rendering effective feature extraction unfeasible. Consequently, performance comparisons were ultimately restricted to models with kernel sizes of one and three. Figure S2c demonstrates that the model utilizing a kernel size of three achieved higher accuracy, and Figure S2d demonstrates lower loss values than those of its counterpart with a kernel size of one. Based on these comparative results, the original VGG11 architecture was selected as the final image branch to distinguish navel orange growth stages.

Regarding overfitting prevention in the 1DCNN-VGG11 model, dropout regularization [47] was implemented after the first two fully connected layers in our model architecture, with optimal dropout rates automatically selected via Bayesian optimization [48]: spectral branch: 0.1066; image branch: 0.2261. Additionally, a ReduceLROnPlateau scheduler [49] based on Pytorch 1.13.1 was employed to dynamically reduce the learning rate when test loss stagnation occurred, thereby enhancing generalization capability. The optimal learning rate was 0.00025.

To highlight the performance advantages of the dual-branch model constructed in this study, various machine-learning and deep-learning models for navel orange growth stage discrimination were established using three types of single-modal data (i.e., NIR reflection spectra, NIR hyperspectral images, and RGB images), as well as dual-source data fusion (spectra combined with hyperspectral images and hyperspectral images combined with RGB images). In addition, this study also utilized three data sources, i.e., the fusion of spectra, hyperspectral images, and RGB images, to establish different deep-learning models to comprehensively evaluate the performance of various features and modeling methods in navel orange growth stage discrimination. The comparative results of the different modal features and the models are presented in

Table 5. Comparison of discrimination results of navel orange growth stages based on different data modals and models.

Data Modal	Model	Training Set	Testing Set
		AC (%)	AC (%)	RC (%)	PC (%)	F1 (%)	Kappa
Spectra	PLS-DA	82.43	80.41	75.34	83.9	75.86	0.7459
	SVM	97.97	87.16	88.49	89.27	88.82	0.8356
	RF	99.16	76.35	76.64	77.47	76.37	0.6963
	KNN	100	70.27	68.29	69.03	68.09	0.6175
	BPNN	77.42	76.35	79.72	78.9	78.31	0.6995
	1DCNN	83.33	77.7	76.18	82.75	77.38	0.7111
Hyperspectral images	ResNet18	99.33	68.24	64.91	75.96	66.49	0.5849
	AlexNet	77.37	60.81	61.09	61.25	60.45	0.4953
	VGG11	100	68.92	67.78	70.43	68.1	0.5983
RGB images	ResNet18	97.54	83.78	83.97	85.82	84.35	0.7916
	AlexNet	85.27	87.16	87.83	88.44	87.99	0.8355
	VGG11	86.85	83.11	82.5	82.58	82.43	0.7838
Hyperspectral images + RGB images	ResNet18	99.03	91.22	91.78	91.6	91.63	0.8877
	AlexNet	89.08	89.86	89.72	90.77	90.17	0.8701
	VGG11	91.88	91.22	91.72	91.12	91.32	0.8878
Spectra + Hyperspectral images	1DCNN-ResNet18	100	85.14	85.86	86.59	85.9	0.8095
	1DCNN-AlexNet	99.83	87.84	88.89	90.25	89.38	0.8438
	1DCNN-VGG11	100	86.49	87.68	88.99	87.56	0.8265
Spectra + Hyperspectral images + RGB images	LSTM-LSTM	84.81	86.49	87.53	88.21	87.54	0.8269
	LSTM-VGG11	95.96	91.22	91.72	91.63	91.61	0.8876
	LSTM-ResNet18	97.24	90.54	91.47	91.8	91.57	0.8789
	LSTM-AlexNet	92.6	86.49	86.04	89.09	86.97	0.8263
	1DCNN-LSTM	100	86.49	87.65	87.36	87.46	0.8271
	1DCNN-ResNet18	100	91.22	91.76	92.12	91.72	0.8875
	1DCNN-AlexNet	99.84	91.22	92.18	91.48	91.73	0.8878
	1DCNN-VGG11	100	95.95	96.66	96.76	96.69	0.9481

.

Table 5 shows the discrimination results for different growth stages of various navel orange cultivars based on spectra, using MSC combined with DT to preprocess the original spectra, followed by establishing the PLA-DA, SVM, RF, KNN, BPNN, and 1DCNN models with the processed spectra. The discriminative performance based on SVM was better than other models, with testing set accuracy, recall rate, precision, F1 score, and kappa coefficient values of 87.16%, 88.49%, 89.27%, 88.82%, and 0.8356, respectively. The discrimination performance based on the PLS-DA, BPNN, and 1DCNN models was good, all with accuracies above 70%. Notably, both the RF and KNN models performed well only on the training set but showed signs of overfitting, leading to reduced performance on the testing set.

To discriminate between the different growth stages of various navel orange cultivars based on image features, we utilized traditional deep-learning network models, such as ResNet18, AlexNet, and VGG11, to establish discrimination models for different image features. In Table 5, for RGB images, AlexNet demonstrated superior performance, with the accuracy, recall rate, precision, F1 score, and kappa coefficient values reaching 87.16%, 87.83%, 88.44%, 87.99%, and 0.8355, respectively. This indicates that AlexNet exhibits stronger adaptability and robustness in feature extraction and classification tasks for RGB images. For hyperspectral images, all three models exhibited a certain degree of overfitting. However, after fusing the hyperspectral images with the RGB images, the discriminative performance of all three models was enhanced. Among them, the ResNet18 model and VGG11 model demonstrated comparable performance, with all evaluation indices exceeding 90%, except for the kappa coefficient. This demonstrates that the feature fusion of hyperspectral images and RGB images can fully exploit the multi-modal features of different navel orange growth stages, thereby improving the precision and generalization of the discrimination process.

To discriminate between the different growth stages of various navel orange cultivars based on a combination of spectral data and hyperspectral images, three deep-learning models with dual-branch networks were constructed, i.e., 1DCNN-ResNet18, 1DCNN-AlexNet, and 1DCNN-VGG11. Table 5 shows that all three models achieved an accuracy of over 80% on the testing set, indicating their effectiveness in distinguishing different growth stages. The 1DCNN-AlexNet model performed the best, with an accuracy of 87.84%. However, it is important to note that all three models exhibited signs of overfitting. Therefore, further improvement in model generalization may require the fusion of additional features or the optimization of the model structure.

Subsequently, this study integrated three data sources, i.e., spectra, hyperspectral images, and RGB images, to discriminate the different growth stages of various navel orange cultivars. To obtain satisfactory results, we established several dual-branch multi-modal fusion deep-learning models, including long short-term memory (LSTM) networks and CNNs, such as LSTM-LSTM, LSTM-VGG11, LSTM-ResNet18, LSTM-AlexNet, 1DCNN-LSTM, 1DCNN-ResNet18, 1DCNN-AlexNet, and 1DCNN-VGG11. The performance of these models was compared, as shown in Table 5. All LSTM models involved in this study had three layers. Table 5 indicates that the 1DCNN-VGG11 model exhibited superior performance for the feature data amalgamated from the spectra, hyperspectral images, and RGB images, achieving a testing set discrimination accuracy of 95.95%, a recall rate of 96.66%, a precision of 96.76%, an F1 score of 96.69%, and a kappa coefficient of 0.9481. These five evaluation indices were the highest among all of the models constructed in this study, further proving the superiority of the 1DCNN-VGG11 model in handling complex data and discriminating different growth stages of various navel orange cultivars.

To further compare the discrimination results based on the different feature data and models, the confusion matrices of the different models are shown in Figure 7a–h. The elements on the diagonal indicate that the predicted category matches the actual category. The darker the color of the diagonal squares, the better the discrimination effect of the model. In Figure 7a–h, G1, G2, G3, G4, and G5 represent Growth_1, Growth_2, Growth_3, Growth_4, and Growth_5 due to spatial limitations. Figure 7 shows that the VGG11 model, based on hyperspectral images, performed the worst (Figure 7c), with 46 navel oranges mis-discriminated into other growth stages. For the 1DCNN model based on spectral data (Figure 7a), 33 navel oranges were mis-discriminated. The VGG11 model based on RGB images (Figure 7b), the 1DCNN-VGG11 model combining spectral and hyperspectral images (Figure 7d), and the 1DCNN-LSTM model based on three data sources (Figure 7f) had moderate discriminative effects; 25, 20, and 20 navel oranges were mis-discriminated, respectively. The VGG11 model, combining hyperspectral and RGB images (Figure 7e), and the dual-branch multi-modal feature fusion LSTM-VGG11 (Figure 7g) and 1DCNN-VGG11 models (Figure 7h) based on three data sources all achieve an overall discrimination accuracy of over 90%. Notably, compared with the other models, the 1DCNN-VGG11 model based on three data sources exhibited the best discriminative performance, achieving 100% discrimination accuracy for both the Growth_3 and Growth_5 stages, with the discrimination effect for the remaining stages exceeding 90% and only six navel oranges mis-discriminated into other stages.

To more intuitively compare the discrimination effects of different modalities, we plotted the radar charts (Figure 8a–f), where each axis represents the training set accuracy (Train-AC), test set accuracy (Test-AC), recall (RC), precision (PC), F1 score (F1), and kappa coefficient (kappa). Usually, the larger the coverage area with color lines in the radar chart, the better the overall performance of the model in the consideration of multiple indices.

Figure 8a–f show that the models based on hyperspectral images had the worst effects (Figure 8b), with the smallest coverage area for each index under different models, followed by the models based on spectra (Figure 8a). The models based on RGB images (Figure 8c) and the models based on the combination of spectra and hyperspectral images (Figure 8e) had roughly equivalent effects, while the models based on the combination of hyperspectral images and RGB images (Figure 8d) and the models based on the combination of spectra, hyperspectral images, and RGB images (Figure 8f) had significantly improved effects. In Figure 8f, the 1DCNN-VGG11 model based on the combination of spectra, hyperspectral images, and RGB images had the largest coverage area, indicating its optimal performance. Whether from the perspective of the loss curve of the test set, the confusion matrix, or the radar chart, the 1DCNN-VGG11 model based on the combination of spectra, hyperspectral images, and RGB images performed well in many regards and is worthy of further development and in-depth research in follow-up practical applications.

3.7. Interpretability Analysis of Growth Stage Discrimination Based on 1DCNN-VGG11 Model

Deep-learning models are often regarded as “black boxes” due to the inherent difficulty in directly observing their internal workings and learned representations, which limits model interpretability. To enhance interpretability, gradient-weighted class activation mapping (Grad-CAM) [50] was applied to the established dual-branch multi-modal feature fusion 1DCNN-VGG11 model based on three data sources. The resulting Grad-CAM heatmaps intuitively reveal the importance of distinct input regions for the model’s decisions: for the spectral branch, the Grad-CAM heatmap indicates the significance of individual wavelengths; for the image branch, it highlights the importance of specific pixels. Regions depicted in red denote the strongest influence on the model’s decision, while progressively bluer regions indicate weak influence. Figure 9 presents the original hyperspectral image (Figure 9a), the RGB image (Figure 9b), and the corresponding Grad-CAM heatmaps for the Growth_3 stage. An analysis of the image branch Grad-CAM heatmap (Figure 9c) reveals that the regions of highest importance (red/yellow) are predominantly concentrated on the central area of the fruit. This localization indicates that features extracted from this region are critical for the model’s recognition of this specific growth stage. The associated textural and color characteristics—primarily yellow pigmentation with localized residual green patches—represent distinctive markers of the Growth_3 stage. The spectral branch Grad-CAM heatmap (Figure 9d) identifies wavelengths contributing most significantly to the decision, primarily located at spectral curve peaks, troughs, and the extremities (beginning and end) of the spectral range. Peaks and troughs typically correspond to key spectral absorption/reflection features. Additionally, analysis of the mean spectra revealed that wavelengths at both extremities of the spectral range exhibit notable variations across different growth stages. Collectively, these Grad-CAM analyses confirm that the 1DCNN-VGG11 model effectively learns and focuses on discriminative features within both spectral and image data, thereby successfully distinguishing between the distinct growth stages of navel oranges.

4. Discussion

4.1. Comparison Between Hyperspectral Images Selected via Proposed Feature Selection and PCA

To deeply evaluate the effectiveness of the proposed feature selection model for hyperspectral images based on spectral reflectance differences, this study compared its performance with models built using hyperspectral images selected using the PCA method. We selected principal components (PCs) corresponding to a cumulative contribution rate of over 95%, presented in Figure 10a. The reconstructed hyperspectral images, when selecting the principal components with a cumulative contribution rate over 95%, are presented in Figure 10b.

Figure 10a,b show that the contribution rate for the PC1 figure is the largest (i.e., 88.77%). That is, the main features of navel orange peels were focused on in the PC1 image for the PCA method, and the effective features in the PC3, PC4, and PC5 images were minimal. Comparing the five images, it can be seen that the texture and pigment information for peels at five different optimal feature wavelengths in Figure 6c–g are obviously richer than those of PC1~PC5 in Figure 10b, although the contribution rate of the PC1 image is about 88.77%. The richer texture and pigment information alone in Figure 6c–g can better reflect the growth stage differences between navel oranges.

The discrimination results of the 1DCNN-VGG11 model established using hyperspectral images selected based on the spectral reflectance differences method and PCA method are shown in

Table 6. Comparison of discrimination results for 1DCNN-VGG11 model between hyperspectral images selected using the spectral reflectance differences method and the PCA method.

Data Features	Model	Training Set	Testing Set
		AC (%)	AC (%)	RC (%)	PC (%)	F1 (%)	Kappa
Spectra + RGB images + Hyperspectral images with proposed method	1DCNN-VGG11	100	95.95	96.66	96.76	96.69	0.9481
Spectra + RGB images + Hyperspectral images with PCA	1DCNN-VGG11	100	93.92	94.93	94.88	94.9	0.9222

. These modeling results are superior to those established with the hyperspectral images selected using the PCA method, further validating the effectiveness of the method adopted in this study.

4.2. Comparison of Hyperspectral Images with Different Optimal Feature Wavelengths

To obtain the optimal feature wavelengths of the hyperspectral images of navel oranges, we implemented the ASRIMD strategy with five intervals in Section 3.5; a characteristic wavelength was selected from each interval.

To validate the viability of using ASRIMD with five intervals, experimental comparisons were conducted using partition schemes with three, five, seven, and nine intervals. For each configuration, a 1DCNN-VGG11 model was developed by integrating one-dimensional spectral data, hyperspectral images, and RGB images. The performance comparison between these different intervals is listed in

Table 7. Performance comparison for ASRIMD with 3, 5, 7, and 9 intervals.

Data Features	Model	Training Set	Testing Set
		AC (%)	AC (%)	RC (%)	PC (%)	F1 (%)	Kappa
Spectra + RGB images + Hyperspectral images with 3 bands	1DCNN-VGG11	100	93.92	94.81	94.63	94.65	0.9222
Spectra + RGB images + Hyperspectral images with 5 bands	1DCNN-VGG11	100	95.95	96.66	96.76	96.69	0.9481
Spectra + RGB images + Hyperspectral images with 7 bands	1DCNN-VGG11	100	95.27	96.25	96.35	96.19	0.9395
Spectra + RGB images + Hyperspectral images with 9 bands	1DCNN-VGG11	100	94.59	95.35	95.15	95.17	0.9309

.

Table 7 shows that growth stage discrimination using the fusion of spectra, RGB images, and hyperspectral images optimally selected via ASRIMD with five intervals based on the 1DCNN-VGG11 model is better than that of the ASRIMD method with three, seven, and nine intervals. However, the AC values of all of the intervals were 100%. The AC, RC, PC, F1, and kappa coefficient values for the testing set are 95.95%, 96.66%, 96.76%, 96.69%, and 0.9481, respectively. Therefore, for the optimal feature wavelength selection of hyperspectral images of navel oranges, the ASRIMD strategy with five intervals is optimal.

4.3. Comparison with Previous Studies

To further validate the viability and feasibility of the proposed method, the results were compared with those of previous studies in the field of fruits, as listed in

Table 8. Comparison between this study and previous studies.

Fruit	Number of Cultivars	Objective	Method	Model	AC (%)	Ref.
Grape	one	Maturity	HSI	PLS-DA	91	[9]
Tomato	one	Growth stage	HSI	Linear DA (LDA)	93.1	[11]
Avocado/Kiwi	one	Ripeness	HSI	CNN	93.3/66.7	[17]
Grape	one	Maturity	Vis-NIR spectroscopy	Stacked autoencoders (SAE)	94	[19]
Strawberry	one	Maturity	HSI	1D ResNet	86.03	[22]
Papaya	one	Maturity	Vis imaging and HSI	Multi-modal VGG16	88.64	[26]
Korla Pear	one	Maturity	HSI	BPNN	93.5	[51]
Pineapple	one	Maturity	Vis/NIR transmittance spectroscopy	PLS-DA	90.8	[52]
Navel orange	five	Growth stage	NIR spectroscopy, HSI, and RGB imaging	Multi-modal dual-branch model (1DCNN-VGG11)	95.95	ours

.

Although the maturity and growth stage discriminations of fruits have been investigated in previous research [9,11,17,19,22,26,51,52] using HSI and Vis–NIR spectroscopy, there have been no studies on the fusion of NIR spectroscopy, HSI, and RGB imaging in this field. Moreover, the discrimination models of the maturity and growth stages have mainly focused on traditional machine-learning methods, such as LDA, PLS-DA, and BPNN. However, in our work, the growth stage discrimination of navel oranges was deeply studied using the fusion of NIR spectroscopy, HSI, and RGB imaging. Moreover, the research samples were navel oranges from five different cultivars instead of the same one. Table 8 shows that the discrimination accuracy (AC) for the growth stages (95.95%) is higher than that of previous studies. The reasons for the better performance are as follows: (1) Multi-modal information, including NIR reflection spectra, hyperspectral images, and RGB images, was fully utilized, enriching the information reflecting growth stage variations, not only from internal component changes in spectral features [19,22] but also from external pigment and texture features in spatial hyperspectral images [9,11,17] and RGB images [26]. (2) Spectral–spatial complementarity was considered via the fusion of HSI-RGB because HSI has richer spectral information and higher-resolution spatial information [22], and RGB images are suited to morphological changes [26]. (3) A multi-modal dual-branch deep neural network (i.e., 1DCNN-VGG11) was established that can fully extract the key spectral information from the spectral branch of the 1DCNN-VGG11 model and obtain key spatial information from the image branch of the model. Moreover, the extracted spectral and spatial information were simultaneously employed using a concatenation fusion operation [26], as the input data of the fully connected layers can discriminate the growth stages of navel oranges. However, discrimination performance using different cultivars may be improved using more advanced deep-learning models, such as the HSI-RGB fusion strategy [53] and the cross-modal attention module [54]. At the same time, the influences of lighting conditions, different years, and different sensors on navel orange growth stage discrimination will be studied in our future work.

5. Conclusions

To accurately discriminate the growth stages of navel oranges, we adopted NIR HSI and MV technology to acquire NIR HSI data (reflectance spectra and hyperspectral images) and RGB images for five Gannan navel orange cultivars at different growth stages. Moreover, we developed a dual-branch multi-modal fusion model integrating hyperspectral data and RGB images. The proposed model was compared with multiple machine-learning and deep-learning models based on single-modal or multi-modal data. Notably, the 1DCNN-VGG11 architecture combined with three data sources (spectra, hyperspectral images, and RGB images) achieved a testing accuracy of 95.95%, a recall rate of 96.66%, a precision of 96.76%, an F1 score of 96.69%, and a kappa coefficient of 0.9481, demonstrating superior performance in growth stage discrimination. In addition, compared with the traditional PCA technique, using spectral differences as a feature selection method for hyperspectral images had a better effect in terms of improving model performance.

Therefore, the fusion of HSI and MV technology combined with a dual-branch multi-modal deep CNN model can effectively and precisely distinguish different growth stages in navel oranges. This study provides a significant scientific foundation for identifying the optimal harvest time for agricultural crops with different cultivars and their quality control. At the same time, the method proposed in this study can be deployed in field operations, such as those involving the use of sensors mounted on tractors or drones, providing strong support for the practical application of fruit growth management.