Detecting Surface Defects of Achacha Fruit (Garcinia humilis) with Hyperspectral Images

Abstract

Hyperspectral imaging data within the wavelength range of 400–1000 nm were used to classify the common skin conditions (i.e., normal, scar, decay, and insect bite) of achacha fruits. The band ratio (BR) and spectral angle mapper (SAM) algorithms were used in a binary classification. Furthermore, SAM, support vector machine (SVM), and artificial neural network (ANN) models were used in a multiclass classification. The performances of the binary and multiclass classification models were assessed. For the binary-classification approach, the three defective classes were merged into one, and the accuracies of the BR (990 nm/600 nm) and SAM were 78.70% and 75.02%, respectively. Furthermore, the SAM, SVM, and ANN accuracies in the four class problems were 58.36%, 83.59%, and 99.88%, respectively. A principal component analysis (PCA) was used for the data reduction. Nine characteristic wavelengths were extracted from the weighting-coefficient curves of the first four principal components. Using only the nine selected bands, the accuracies of the SAM, SVM, and ANN models were 51.49%, 80.76%, and 96.85%, respectively. Compared with the models using full bands, the classification accuracies of the models using only nine characteristic bands decreased slightly; however, the gain in classification speed and the potential data-acquisition speed can expedite the classification of achacha fruits.

1. Introduction

The achacha fruit (Garcinia humilis) originates in Bolivia [1,2] and is increasingly popular in Taiwan due to its flavor and nutrition. Therefore, the development of sorting and grading methods and criteria is an urgent need in the postharvest process of achacha fruits. Fruits with defects can cause severe consequences if left undetected during postharvest processing. Fruit defects in a specific fruit not only degrade the quality of the fruit but can also affect the surrounding fruits [3,4]. For example, fungi on the surface of a fruit can spread to surrounding fruits during storage and transportation.

Traditionally, fruit-defect assessment was performed manually by experienced workers. However, it is subjective, inconsistent, slow, and affected by eye fatigue during continuous work over a long time period [3,5]. Moreover, manual fruit-defect evaluation proved unsuitable when productivity increased and labor became increasingly expensive. With improvements in hardware and software, machine vision has become valued in fruit evaluation due to its non-invasiveness, fast speed, and accessibility [6]. Researchers commonly use machine vision based on RGB images for surface-defect detection, such as classifying healthy and defective tomatoes [7], mangos, and guavas [8]. However, due to the limitation of the use of three broadband channels, different skin statuses can be similar in RGB images [9]. For example, thrips scarring and oleocellosis in citruses were misclassified as normal skin due to the similarities in color [10]. Furthermore, when performing defect classification on green plums, rain spots can be misclassified as scars due to the stem end, or as normal skin due to low light intensity and shadow [11].

Hyperspectral imaging (HSI) systems can acquire spatial and spectral data and, therefore, provide much more information than RGB cameras. Moreover, HSI systems have recently become increasingly popular for investigating food and agricultural products [12,13]. For example, HSI data have been used to estimate the ripeness of avocados [14] and to detect the quality parameters in the apples [15]. Moreover, this system spectral data within infrared wavelength ranges can provide more information than RGB images. For example, researchers used near-infrared HSI data to establish the differences between the normal and defective skins of jujubes [16] and peaches [17]. Furthermore, the HSI data in the visible and near-infrared were used to detect common skin defects in citrus [18], early downy mildew in grapevines [19], and damage in the apples [20].

The band ratio (BR) method and spectral angle mapper (SAM) are typical signature-based methods for HSI data analysis. The BR is an algorithm calculating the ratio between two spectral bands to increase the contrast between two classes. Global thresholding is commonly applied to BR images to separate the two types after contrast enhancement. Based on spectral characteristics, three ratios, including 987 nm/1464 nm, 1160 nm/1464 nm, and 1285 nm/1464 nm, were treated as the potential to classify insect-infested and normal skin in jujubes [16]. Next, the ratio of 1160 nm/1464 nm was chosen based on visual observation, after which the global threshold T = 0.7 was used to classify the defects. Li et al. considered three wavelengths, 781 nm, 815 nm, and 848 nm, to classify normal skin and defects in the bi-colored peaches [17]. Of the three combination pairs, 781 nm/848 nm was selected as the best option, through visual observation, for increasing contrast before global thresholding (T = 0.7). Zhang et al. [21] used the 680 nm/715 nm ratio to detect the stem end in a fruit mask. Spectral angle mapper (SAM) classification uses an n-dimension (n equals the number of bands) angle to match the HSI data to reference spectra. Smaller angles represent closer matches to the reference spectrum. This method is relatively insensitive to illumination and albedo effects. The SAM method was used for surface-defect detection in citrus [22] and pears [23].

Machine learning (ML), using data and algorithms to imitate human learning, has been successfully used for classification problems. For example, the artificial neural network (ANN) and support vector machine (SVM) are two popular ML algorithms used for the surface-defect classification of fruits. The ANN was developed based on the electrical activity of the nervous system. For classification problems, ANN uses weights and biases for modeling the relationship between the data and labels. The ANN was used to detect chilling injury in Red Delicious apples [24], as well as bruising in the apples, chikoos, and guavas [25]. The SVM, as introduced, is used as a binary classifier that can separate two classes using a hyperplane. A multiclass SVM was extended using a one-versus-one (one-vs-one) or one-versus-all (one-vs-all) approach with a decision function. Using the kernel trick, the SVM can increase flexibility in dealing with nonlinearly separable problems. The SVM approach is widely used in surface-defect detection in fruits with the HSI data. Furthermore, the SVM was used to detect defective jujubes [26] and cracked tomatoes [27].

This study aimed to detect common surface defects in achacha fruit using the HSI data. The conventional HSI data analysis methods (BR and SAM) and ML-based classification models (SVM and ANN) were used, and the accuracies of these approaches were compared. Furthermore, a procedure for detecting the stem ends of achacha fruit to prevent misclassification during defect classification was developed.

2. Materials and Methods

2.1. Achacha Samples and HSI-Data Acquisition

Achacha fruits with three defect types were collected in Pingtung, Taiwan, and kept at room temperature before the experiment without pretreatment. A custom-made hyperspectral image acquisition system was used to obtain the image data. A description of the system can be found in the previous work [28]. The HSI data were obtained within two days after the harvest. The spectral data within the 400–1000 nm wavelength range were collected and used to classify the normal skin and the three surface defects of the achacha fruits. Figure 1 shows the pseudo RGB image of the typical samples. In this study, the fruit samples’ ripeness levels are within levels 4 to 7, as defined in the previous study [28].

2.2. Defect Classification Methods with HSI Data

The classification methods based on the hyperspectral image data, including the convention signature-based detection algorithms such as the BR and SAM, and the ML algorithms including the ANN and SVM, were used for surface defect detection of the achacha fruits. Both binary and multiple-class classification approaches were evaluated. For the binary classification approach, the BR and SAM methods were used to differentiate the normal and defective surfaces; furthermore, the four different surface conditions (normal and three defects) were classified by the SAM, SVM, and ANN.

The BR method and thresholding were applied to classify the normal and defective skin. The two optimized bands were determined by the maximizing correlation squared, using the method proposed by Lee et al. [29]. Dummy variables 0 and 1 represented normal and defective skin conditions, respectively. The band ratios were calculated for all possible combinations. Next, the correlation between the band ratio and dummy variable was evaluated. The 2-band combination with the highest squared correlation value (R²) was selected. Next, the optimized threshold was applied to separate normal and defective classes. The referenced spectra of the SAM classification were obtained by taking the average of the normalized spectra of the training data points of each corresponding class. A pixel would be classified as the class with the smallest spectral angle between the vector of the pixel and the vector of the referenced spectrum of each class in the n-dimensional spectral space. The SVM model used radial basis function (RBF) as the kernel, and the regularization parameter was 1. Moreover, the ANN model consists of four layers, including one input layer (121 nodes), two hidden layers (200 nodes each layer, ReLU activation function), and one output layer (four nodes, softmax activation function). Furthermore, a binary ANN model was trained to create a fruit mask within the HSI image. The labeled data for training and test were selected from the pixels inside the fruit mask areas of the 225 samples. The pixels of the normal, scar, decay, and insect skin types were selected from 88, 85, 80, and 62 of the 225 samples, respectively. Please note that a fruit sample could contain more than one skin type. For each skin type, 7000 data points were selected and labeled; 5000 and 2000 data points were used as training and testing sets, respectively. For binary classification performed by the BR and SAM, all data points of the defects were merged into one data set of the defective class.

2.3. Data Reduction Using Principal Component Analysis

The multidimensional data set of the hyperspectral images consists of valuable and redundant data. Therefore, the data reduction methods are usually used when analyzing the HSI data. The principal component analysis (PCA) is one of the unsupervised data reduction algorithms for the HSI data [5,9]. The PCA can be efficiently used for band selection to reduce the number of spectral bands while retaining the most helpful information to reduce the storage requirements and computation time for the data processing. Based on the PCA results, the bands were selected to classify strawberries into three ripeness levels [30], to detect the skin defects of the bi-colored peaches [17], and to identify the typical defects on oranges [18]. This study used PCA to select useful characteristic wavelengths to classify the four skin types. The weighting coefficients of the PC components were obtained based on the training data. Peaks and valleys of PCs, which can explain most of the variance of skin types, were selected as the optimal wavelengths. The optimal wavelengths were used by the ANN and SVM skin defect classification models, and the accuracies were assessed.

2.4. Stem End Detection

The stem end, a component connecting the fruit to the tree branch, can be misclassified as a defect, so it should usually be identified and excluded when performing the fruit defect classification. When performing defect detection with the RGB image, for the apple which has only one defect candidate region, features of the region should be extracted, and a pattern recognition algorithm should be applied to determine whether the apple is sound or defective [31,32]. When the skin defect detection of the fruits with HSI data is performed, a method involving the size and shape of the defective candidate regions was used to differentiate the stems from the actual defects of the bi-colored peaches [17]. The corrected single-wavelength image at 666 nm was used to distinguish the stem end from the decayed region of the tomato [33]. The achacha fruit is nearly egg-shaped, and the stem end is generally located at the sharp end. Based on this fact, a stem-end detection algorithm was developed. The gravity center location of the fruit was identified first. Next, the pixel in the fruit, which is furthest from the gravity center of the fruit, was considered the tip of the stem end. Finally, the pixels of the fruit, within a search range ($R_{search})$ to the stem end tip, were considered belonging to the stem end. In this study, the search range was proportional to the square root of the fruit area, and the scale factor was determined experimentally.

3. Results and Discussion

3.1. Spectral Characteristics

Figure 2 shows the averaged spectra of the four skin types. The reflectance spectra of all skin types are low and similar within the wavelength range of 400–485 nm, and the reflectance of the normal skin is slightly higher than those of all defective skin types. There is a turn-over point at about 485 nm for the reflectance of all skin types. The increases in reflectance for wavelengths higher than the turn-over point are faster than the increases in reflectance Within the wavelength range of 500–850 nm, the reflectance of normal skin is much higher than those of the other skin types; the insect bite skin has the lowest reflectance. Moreover, rusty and decaying skins have a relatively similar spectrum.

Although the average reflectance intensity of normal skin is higher than that of all the defective skin types within most of the wavelength range (400–920 nm), using reflectance intensity to differentiate between normal and defective skin is problematic. As the curvature of the achacha fruit leads to non-uniform illumination when using point light sources, the reflectance intensity of the defective skin located in the center of the fruit could be higher than that of the normal skin near the fruit border in the image. One thing that can be noticed is that within the wavelength ranges of 715–745 nm and 845–915 nm, the slopes of the reflectance intensity of the normal skin are negative; however, those of all defective skin types are positive. This finding can potentially be used for the derivative analysis of the HSI data. For example, the second derivative of the spectra was used to detect fungal infection in strawberries [34] and in the pits of cherries [35].

3.2. The BR and SAM Algorithms for Binary Classification

The binary defect classification (normal or defective) was performed using the BR and SAM algorithms. Figure 3 shows the contour plot of the R² values of the BR pairs. Two high R² regions centered at 840 nm/600 nm and 980 nm/600 nm were found, and the BR with the highest correlation coefficient is 990 nm/600 nm (R² = 0.6490). Furthermore, when applied to the training data, the optimized threshold for this BR is 1.7726 with a 0.9375 accuracy value.

The accuracies of the BR and binary SAM were 78.70% and 75.02% for the test data set, respectively. Figure 4 shows the classification results of the typical samples predicted by the BR and binary SAM methods. In Figure 4, the sample results are as follows: (a) results show that the BR and SAM algorithms can appropriately classify the scar and insect bite spot correctly; furthermore, it can be seen from the results of sample (b) that the BR can predict decay better than the SAM. When an achacha fruit ripens, its color changes from green to yellow to dark orange. Therefore, the spectra of the achacha fruits with different ripeness levels vary and could lead to the difficulty of using a single band ratio to classify the defects on achacha fruits of all ripeness levels. For example, samples (c) and (d) are the achacha fruits with high and low ripeness, respectively. Therefore, the dark red normal skin of sample (c) and the less ripening region in sample (d) were misclassified as defects.

The gray regions shown in Figure 4 were the stem end regions defined by the stem end detection algorithm proposed in this study. Through visual examination, the optimized scale factor was 1/4; therefore, the search radius was defined as:

$${\mathrm{R}}_{\mathrm{search}}=\frac{\sqrt{\mathrm{Mask}\mathrm{area}}}{4}$$

(1)

3.3. The SAM and ML Models for Multiple-Class Classification

The confusion matrices and accuracies of the SAM, SVM, and ANN models are shown in

Table 1. Confusion matrices of ANN, SVM, and SAM models using full spectral data.

Model	Actual	Classified as (%)				Accuracy
		Normal	Scar	Decay	Insect
SAM	Normal	85.2	11.3	3.4	0.1	58.36%
	Scar	6.2	36.1	17.8	39.9
	Decay	13.4	23.9	39.5	23.2
	Insect	2.5	10.4	14.5	72.6
SVM	Normal	92.8	3.2	4.0	0	83.59%
	Scar	1.6	78.5	7.8	12.0
	Decay	4.8	11.7	73.7	9.9
	Insect	0.2	4.5	6.0	89.3
ANN	Normal	99.9	0.1	0	0	99.88%
	Scar	0	99.9	0.1	0.1
	Decay	0.1	0.1	99.8	0
	Insect	0	0	0.1	100.0

. Among the three models, the SAM model has the lowest accuracy (58.36%), and the ANN model has the highest accuracy (99.88%). For the SAM model, the misclassification rate between the two classes (i.e., normal and insect bite) with the most different spectral characteristics is the lowest. Furthermore, these two classes have higher accuracy (85.2% and 72.6%) than the other two classes (36.1% and 39.3%).

Figure 2 shows that the spectral profiles of the scar and decay skin types are the most similar if the shape and reflectance magnitude are considered simultaneously. Therefore, compared with other skin type pairs, the chances of misclassification are high between these two classes. The misclassification rates of the scar to decay and the decay to scar are 17.8% and 23.9%, respectively. However, there is an exception for the scar skin type in that the misclassification rate of a scar to the insect bite is the highest (39.9%). The reason could be that the SAM model uses only the n-dimension angle of the pixel to match the angle of the reference spectra without considering the magnitude of the n-dimension spectrum vector. Therefore, if the angle between the vector of the scar skin pixel and the reference spectrum vector of the insect bite is smaller than that between the decay reference spectrum vector, the scar pixel could be misclassified as the insect bite class, rather than the decay classes by the SAM algorithm.

Instead of only using the angle information of the n-dimension spectral data, both the SVM and ANN models can use the angle and magnitude of the spectral vectors simultaneously. The accuracy of the SVM model is 83.5%. Similar to the SAM model, the normal (92.8%) and insect bite (89.3%) classes have higher accuracies. Moreover, compared to the SAM model, the misclassification rate of a scar to the insect bite decreased to 12.0%. For the ANN model, the misclassification rates for all classes were no greater than 0.2%. This result implied that the ANN model could be trained appropriately and used to validate the data set used in this study. Although the accuracy of the ANN model is high and the computation speed is fast, the use of the ANN model in practice, such as the online sorting of the achacha fruits during the postharvest process, should be further investigated. Due to the high model complexity, the ANN model is prone to overfitting [36].

Figure 5 shows the typical classification results of the samples using the SAM, SVM, and ANN models. The skin color changes from green to yellow to dark orange when the achacha fruits ripen, and sample (d) has the highest ripeness stage. Moreover, samples (a)–(c) are under uniform illumination. However, the illumination for sample (d) is relatively non-uniform; the upper part of this sample is under lower illumination.

It can be seen, from the first row of classification results, that the SAM can correctly identify normal skin with proper illumination; however, the SAM cannot identify the normal skin of the sample with a high ripeness stage under low illumination. Therefore, under relatively low illumination, the upper part of sample (d) (high ripeness stage sample) was misclassified as either the decay or scar regions. Compared to the SVM and ANN models, the misclassification rate between defect classes is high for the SAM model. For example, a significant portion of the fully developed decayed area in sample (b) was misclassified as an insect bite area. Furthermore, the newly developed decay regions on samples (b) and (c) were misclassified as a scar class.

The models (i.e., SVM and ANN) based on the machine learning algorithms can classify the defect classes better than the SAM model. For example, the sample (b) classification results by the SVM and ANN models show that both models can classify the fully developed decay correctly. Furthermore, the newly developed decay regions located on the border of normal skin and fully developed decay regions in sample (b) and the newly onset decay in sample (c) can also be identified. One thing that should be pointed out is that there are external contamination spots on the fully developed decay region of sample (b). The contamination did not affect the classification results of the ANN model; however, the SVM model classified the decay regions with the contamination as insect bite spots. Therefore, this finding should be considered when applying the above-mentioned models to the sorting line.

Sample (d) classification results can show how low illumination affects the classification results of the high ripeness stage achacha fruits. In addition, the results show that the SAM model, among the three models, is most sensitive to non-uniform illumination. Under relatively low illumination, the SAM model classified the upper part of sample (d) as either the decay or scar regions. Moreover, the ANN is the model least sensitive to low illumination. It can be seen that the misclassification of normal to decayed regions of the ANN model is smaller than that of the SVM model. Except for the high ripeness level (level 7) samples, the ripeness level does not affect the classification result of the classification models used in this study.

At the end of the ripening stage, the dark orange achacha fruits either decay or dehydrate. The ripening and decaying of fruits are continuous processes. Therefore, compared to a classification model, a regression model would be more appropriate to evaluate the fruit’s level of ripeness or decay. However, defective identification models are classification models. Therefore, when using a classification model to identify surface defects (including decay) of the achacha fruits, selecting data points with suitable ripeness and decay level for the training data set is critical for adequately separating the normal skin types from other low reflectance skin types. Moreover, the data selecting and labeling task of the normal data set for model training would involve an iterative trial and error procedure. The classification results of sample (d) show that the SVM model could correctly identify the normal skin type in the properly illuminated area of this high-ripeness stage sample; however, the ANN model misclassified a large portion of normal skin as scars. Therefore, to reduce the ANN model's misclassification of normal skin at the high ripeness stage of achacha fruit, sufficient representative data points should be selected and added to the training and test sets of the ANN model.

3.4. Principal Component Analysis

Figure 6 shows the first four PC images of eight typical samples (two samples per class). The first four PCs can explain 98.75% of the total variance (PC1: 81.77%, PC2: 13.08%, PC3: 2.80%, and PC4: 1.10%). The PC1 images are similar to the grayscale image of the samples, and these images do not provide extra information other than the original spectral images for surface defect detection. However, by investigating the images of PC2~4, a distinguishable contrast of different skin types can be found in the PC image(s). Among the four PCs, PC2 could be the most valuable for detecting defects, especially decay and insect bite. Due to the contrast between decay and normal, insect bite and normal is high. In addition, the contrast between scar and normal is distinguishable. Furthermore, in PC3 and PC4 images, the defects and normal skin can also be differentiated with low contrast.

Figure 7 shows the weighting-coefficient curves of the first four PCs and the wavelengths selected for the training of the classification models. The local minimums and maximums of the weighting-coefficient curves correspond to the effective wavelengths that significantly contribute to the PC images. Therefore, the wavelengths corresponding to the local minimums and maximums of the weighting-coefficient curves were selected to train the SVM and ANN models for a further data reduction analysis [16].

Table 2. Confusion matrices of SAM, SVM, and ANN models (using characteristic wavelengths).

Model	Actual	Classified as (%)				Accuracy
		Normal	Scar	Decay	Insect
SAM	Normal	76.8	18.9	3.7	0.6	51.49%
	Scar	10.6	28.1	16.8	44.5
	Decay	18.3	19.7	30.2	31.8
	Insect	3.7	8.3	17.2	70.9
SVM	Normal	90.8	5.0	4.0	0.1	80.76%
	Scar	1.1	74.2	9.8	14.8
	Decay	4.5	14.1	69.4	12.0
	Insect	0.5	3.5	7.4	88.5
ANN	Normal	98.7	0.8	0.4	0.1	96.85%
	Scar	0.1	96.5	0.9	2.5
	Decay	0.7	4.0	93.8	1.5
	Insect	0	1.1	0.5	98.4

shows the performance of the SAM, SVM, and ANN models using the characteristic-wavelength data selected from the PCA analysis. The accuracies of the SAM, SVM, and ANN models trained with the selected nine bands are 51.49%, 80.76%, and 96.85%, respectively. For the ANN model, the accuracy decreased by 3.13%; however, the number of spectral bands reduced from 121 to 9. Using only nine characteristic wavelengths for surface defect detection can improve the training and classification speeds and provide more options for image data acquisition. In addition to using a push-broom system to obtain the full-range bands and to select the nine characteristic bands, a snapshot multispectral camera with nine-channel Bayer-like mosaic filters could be custom-made to obtain the nine wavelength bands in one shot. Reducing the classification and image acquisition times is essential for the vision-based online sorting systems of agricultural products.

In Taiwan, achacha is an emerging fruit with a long juvenile period. The achacha fruit samples are relatively difficult to obtain at the current stage. The small number of fruit samples used in this preliminary study were obtained from the same farm in the same season. The significant difference in the accuracy of ANN vs. SAM pointed out the possibility of overfitting in the ANN model, and this issue will be investigated by using a larger dataset of the independent samples to train the models in future work. Furthermore, different cultivar and environmental factors could affect the physicochemical properties of the achacha fruits; therefore, the reflection spectra of the four skin types investigated in this study could have a variance for the achacha fruits obtained from the different farms or obtained in other years. Therefore, collecting the hyperspectral image data of the achacha fruits from different farms in other seasons to diversify the training and test data set will also be included in future work; therefore, the robustness of the classification models can be improved.

4. Conclusions

This study showed that the HSI data within the wavelength range of 400–1000 nm could be used to accurately classify the normal and three common defective skin types of the achacha fruit. Both the conventional hyperspectral image data analysis algorithms and the machine learning algorithms were used as classifiers for the skin defect detection of the achacha fruits. However, due to the complexity of the skin defect detection problem of the achacha fruits, the conventional HSI data analysis methods, such as the BR and SAM, cannot classify the skin statuses well, even in binary classification problems. The ML algorithms like the SVM and ANN can give significantly better prediction results than the BR and SAM. In terms of accuracy in multiple-class problems with full spectra, the accuracies of the ANN (99.88%) and SVM (83.59%) are significantly higher than that of the SAM (58.36%). Therefore, it is worth spending more on computation costs for the ML algorithms and using them for sorting achacha fruits during postharvest processing, especially with the recent fast development of computer hardware. The ANN model has the highest accuracy of all the models used in this study and the fastest computation speed. Therefore, the ANN model is a good candidate for the postharvest processing of the achacha fruits, such as online sorting. However, due to the high model complexity, the ANN model is prone to overfitting. To improve the robustness of the ANN model, collecting the hyperspectral image data of the achacha fruits from different farms in other seasons to diversify the training and test data set would be a good option. The accuracy of the models using only characteristic wavelengths selected based on PCA decreased; however, the reduction of the bands can improve training and classification speeds and provide more options for image data acquisition. For example, a snapshot multispectral camera could potentially expedite the data acquisition for only the selected nine characteristic bands. Classification speed is critical for the online sorting system of fruits. Additionally, the stem end detection algorithm developed based on the fruit shape can correctly identify the stem end location, and it can be used to avoid misclassifying the stem end as a defect during the online sorting of the achacha fruits.