Hyperspectral Imaging System with Rotation Platform for Investigation of Jujube Skin Defects

Abstract

A novel object rotation hyperspectral imaging system with the wavelength range of 468–950 nm for investigating round-shaped fruits was developed. This system was used to obtain the reflection spectra of jujubes for the application of surface defect detection. Compared to the traditional linear scan system, which can scan about 49% of jujube surface in one scan pass, this novel object rotation scan system can scan 95% of jujube surface in one scan pass. Six types of jujube skin condition, including rusty spots, decay, white fungus, black fungus, cracks, and glare, were classified by using hyperspectral data. Support vector machine (SVM) and artificial neural network (ANN) models were used to differentiate the six jujube skin conditions. Classification effectiveness of models was evaluated based on confusion matrices. The percentage of classification accuracy of SVM and ANN models were 97.3% and 97.4%, respectively. The object rotation scan method developed for this study could be used for other round-shaped fruits and integrated into online hyperspectral investigation systems.

1. Introduction

Appearance is one of the important attributes of fruits which affect their price and consumers’ preferences. Fruits which are unsuitable for sale or storage due to surface defects caused by mechanical injuries, insects, diseases, over-ripeness, under-ripeness, etc. should be sorted out during postharvest processing since the consumers normally prefer to buy fruits with no apparent defects. Furthermore, removing fruits with surface defects from the sound ones can reduce losses by preventing secondary contamination. Sorting out fruits with surface defects is usually carried out by manual operation, which is time-consuming, laborious, and subjective [1,2,3].

Scientific and technological advances have contributed to improve post-harvest processing. For example, computer vision techniques based on red, green, and blue (RGB) images have been developed and applied in fruit sorting and grading systems over the past decades. By using RGB values, Golden Delicious and Jonagold apples were graded into four categories according to European standards and the overall accuracy are 78% and 72% respectively [4]. The color index of citrus fruits was measured by using the average of the CCI value and the average RGB values. And the optimized algorithms for in-line color estimating of citrus were developed for mobile platform [5]. A computer vision model based on RGB image was used to estimate the size and weight of yellow melon [6]. Color moments and color histogram extracted from color images of tomato were used to classify ripeness stages of tomato [7]. A computer vision system with three color cameras was used to separate normal and defective apples with 11% classification error. However, this system could not distinguish different defect types [8].

Although the traditional computer vision systems based on RGB color video cameras are effectively used to measure or detect external quality characteristics such as color, texture, size, shape, and some obvious defects of fruits, it is impossible or difficult to detect or differentiate some other less obvious defects whose colors and texture are similar to the normal skin [9]. Unlike the conventional color camera capturing three monochromatic images centered at 700.0 nm (red, R), 546.1 nm (green, G), and 435.8 nm (blue, B) to imitate the vision of humans, the multispectral computer vision system can capture two or more different waveband monochromatic images in the spectrum as we want by utilizing optical filters. A common multispectral camera approach utilizes a filter-wheel consisting of optical bandpass filters. By shifting the filters sequentially into the optical path, the spectrum image is acquired through the channels [10]. A multi-spectral vision system including four wavelength bands in the visible/NIR range was developed to investigate the defects of Jonagold apples [11]. The used filters were centred at 450, 500, 750, and 800 nm and had, respectively, a bandwidth (full width at half maximum or FWHM) of 80, 40, 80, and 50 nm. No error was made on rejected fruits and high classification rates were reached for apples presenting serious defects and recent bruises. Huang et al. [12] developed a multispectral imaging system with two dichroic beamsplitters, two band-pass filters and three charge-coupled device (CCD) cameras for online detection of bruises on apples. The online image acquisition speed is about three apples per second and the classification accuracy of bruised apples in online tests is 74.6%. Hyperspectral imaging, combining conventional two-dimensional imaging and spectroscopic techniques, produces a three-dimensional image data cube, and records the spatial and spectral information simultaneously [13]. In recent years, many researchers have demonstrated the feasibility of hyperspectral imaging systems in detecting skin defects of fruit [14,15]: bruise on apple and pear [4,16], chilling injury in peaches [17,18], decay in citrus [5,19], fungal infections in strawberry [20], and green mold pathogens on lemons [21] to name a few.

Whole surface inspection is a challenging problem until now that affects the comprehensive and reliability of the grading results of fruits, since information leakage and repetition of fruit surface images or spectral data during color and defect detection could affect successive classification. Although many efforts have been made by worldwide grading engineers for decades, a thorough and practical solution is still not available so far [1]. Capturing images from different views with multiple cameras or with the assistance of mirrors is a common method used in the inspection of fruits to obtain much or whole surface of fruits [1,8,9,22]. Li et al. [1] developed an imaging system with two mirrors on opposite sides of the apple to capture four sides of an apple sitting in a shallow cup by using two cameras mounted above and below the conveyor. This method allows most of the apple surface to be taken, but portions of the apple surface are blocked by the supporting cup. Imou et al. [22] built a vision system with mirrors to measure the three dimensional (3D) shape of the strawberries. Nine mirrors around the object at an angle were used to reflect side views of the object, and the side views were taken simultaneously with a digital camera. The 3D shape was reconstructed from the nine-sided views by the volume intersection method. Reese et al. [9] used computer vision system with mirrors for whole surface imaging of apple suspended by two thin wires. Results revealed that taking one single image using the four or six mirror configurations could catch the image of almost 100% of the surface of apples. However, for testing, an individual apple had to be suspended by two thin wires, which restricts its application in commercial in-line sorting lines. Zou et al. [8] developed a computer controlled system with color cameras for in-line detection of apple defects. In this system, the apples placed on rollers were rotating while moving and three color cameras were placed on the line. Each camera captured three images from each apple. A total of nine images obtained for each apple allowed the entire surface to be scanned. The defects, including the stem-ends and calyxes, called regions of interest (ROIs), were segmented and counted in each of the nine images. Since a calyx and stem-end cannot appear at the same image, an apple is defective if any one of the nine images has two or more ROIs. Good separation between normal and defective apples was obtained; however, different defect types could not be distinguished.

The above-mentioned studies about how to achieve whole surface inspection of fruits are suitable for computer vision with snapshot camera(s), while the techniques for whole surface scan or increasing scannable surface area in one scan pass about fruit are not examined thoroughly for line-scan machine vision system such as push-broom hyperspectral imaging (HSI) system. Sun et al. [23] built a hyperspectral imaging system with rotation unit employed two motor driven rollers to obtain all-round hyperspectral image of the whole peach for the study of decayed on peach surface. Baek et al. [22] developed an optical system for push-broom hyperspectral camera to obtain the whole surface spectral data of a round object. The system uses four mirrors to view a rotating round object from two opposite sides and project a combined two-view image onto the aperture of a push-bloom HSI camera. The image of the whole surface of the round object can be used to detect defects located on any part of that surface. In order to obtain the two side views that include the areas around the poles, the distance from the inside mirrors to the outside mirrors and the inclination angles of the outside mirrors need thoughtful design. In their study, calibration tests showed that the system could detect six simulated defects dispersed evenly over the surface of an apple, without missing or repeating a defect, for 78% of the apple fruits tested. In their study, the rotation of spheroidal or round object was driven by four truncated-cone rollers rotating at a steady speed. However, due to the difference in fruit size and shape, the rotation speed of fruit on the rollers may not be well controlled [1]. Furthermore, bounding or wobbling of the fruit could happen during rotation [22]. Thus, the scan lines might be overlapped or missed in some regions due to the un-uniform rotation of fruit.

The objectives of this study were to (1) investigate the capability of using a rotation stage to increase scannable surface area in one scan pass about a jujube for push-broom HSI camera, and (2) develop efficient data processing procedure and classification models for the surface defect detection of jujubes. The increasing of scannable surface area of this object-rotation scan method, compared with the scannable area of using a linear stage for moving the jujube in one scan pass, was assessed.

2. Materials and Methods

2.1. Jujube Samples

The jujubes used in this study were selected from the jujubes, with clear typical defect(s), screened out during postharvest process in jujube orchard located in Pingtung, Taiwan. The defects of jujube were identified by visual inspection with the assistance from jujube farmers. The samples with normal surfaces and five types of common skin defects (i.e., rusty spots, decay, white fungus, black fungus, and cracks) of Kaohsiung No.11 jujube were collected and stored in inspection laboratory at 25 °C. The hyperspectral images of specimens were acquired within two days after samples obtained. Typical images for each skin condition are shown in Figure 1.

2.2. Hyperspectral Imaging System

A prototype hyperspectral imaging system capable of scanning jujube fruit under linear movement or rotation was developed (Figure 2). This line-scan vision system consists of a monochrome camera (acA1920-155 um, Basler AG, Germany), an imaging spectrograph (Inspector V10, Specim Ltd., Oulu, Finland) and a 50 mm focal length VIS-NIR lenses (50 mm/F2.0 67,717 VIS-NIR, Edmund Optics Inc, USA). The complementary metal oxide semiconductor (CMOS) sensor of camera had 1920 × 1200 pixels and the binning function (2 × 2) was used to reduce the resolution and noise of image. The resultant image has resolution of 960 and 600 pixels in special and spectral axes respectively, and the spectral range of this imaging system spanned from 359 nm to 1024 nm. Two 50 W halogen lamps were used to illuminate the sample under investigation. A single axis actuator (KP26, TBI motion, Taiwan) with step motor (ST17-1, National Instruments, Austin, TX, USA) was used for linear motion, and a rotation stage (SGSP-120Yaw, SIGMA KOKI Ltd., Tokyo, Japan) was used for rotation of sample for HSI scan. The rotion stage was attached on top of the stage of linear actuator and jujube sample was placed on the center of rotation stage for scan. The step motors of both stages were driven by a PWM signal generated by counters of general purpose data acquisition devices (NI USB-6210, National Instruments, Austin, TX, USA).

As shown in Figure 2, the direction of scan line of camera and rotation axis of stage was perpendicular to direction of linear stage movement. The linear scan was performed by driving the linear stage without actuating the rotation stage while scanning. On the other hand, when performing a rotation scan, the rotation stage was actuated while the linear stage was held still. Before the rotation scan started, the axis of rotation stage was moved into the plane of line scan by adjusting the position of the linear stage so that the panoramic image of the jujube would be like the cylindrical projection image of a globe-like object. The setup of camera parameters, step motor velocity and operation of system are coordinated through a computer program that was developed in Labview (National Instruments, Austin, TX, USA) and the spectral images were collected in a darkroom. The spatial-spectral data of each scan line was saved as an image file in tagged image file format (TIFF). All TIFF images belonged to the same scan pass were combined to form a three-dimensional data cube when performing data pre-processing.

2.3. Data Collection

Hyperspectral data of 80 samples were collected for this study. Most samples had more than one defect. The numbers of samples with rusty spots, decay, white fungus, black fungus, and cracks are 34, 25, 16, 30, and 38 respectively. For each jujube sample, the sample was manually put on the center of the rotation stage with the stem-calyx axis being aligned vertically and went through the linear and rotation scan. Then, the linear and rotation scan was repeated with the sample’s stem-calyx axis being aligned horizontally. A total of 50 samples were randomly selected as training set to evaluate scanning methods and developing the classification models. The remaining samples were used for evaluating the performance of the object rotation HSI system proposed in this work. The white and dark reference images were taken at a regular time interval for the calculation of relative reflectance [11]. A white diffuse reflectance board with 99% reflection efficiency was used to obtain white reference images, and dark reference images were collected by turning off the lamps and covering the lens with a black cap.

2.4. Data Pre-Process

A program with graphic user interface (GUI) developed in Matlab (The MathWorks, Inc.) environment was used to process spectral images (Figure 3). For spectral data of a scan pass, the TIFF images corresponding to each scan line were combined as a three-dimensional data cube (x, y, λ) where x, y, and λ represent spatial and spectral components respectively. Due to the low light output of lamps in the VIS < 468 nm and low quantum efficiency of the CMOS in the NIR > 950 nm, only hyperspectral image data in the wavelength range between 468 nm and 950 nm were used for analysis in this work. A region of interest (ROI) was applied to the data cube to select the relevant subset of data for further process. The locally estimated scatterplot smoothing (LOESS) filter was applied to selected data in the spectral direction, and then the relative reflectance was calculated using white and dark references.

Pseudo RGB images constructed from calibrated data by combining three wavelengths were used for feature labeling of jujube surface. Due to the smooth and waxy skin of the sphere-like jujube surface, the mirror-like reflection or glare of incident light cannot be completely removed by adjusting the angle of incident light. In HSI, glare alters the spectral fingerprint, which could introduce artifacts and hence deteriorate classification results. Therefore, glare pixels need to be detected and removed before feature extraction and classification [24]. In this study, glare was considered as one type of surface feature and thus pixels of seven types of jujube skin conditions (i.e., normal, rusty spots, decay, white fungus, black fungus, crack, and glare) were visually identified and labeled on the pseudo RGB images for the use of supervised machine learning methods.

2.5. Classification Methods

Support vector machine (SVM) and artificial neural network (ANN) are effective methods for spectral data analysis. SVM, based on the statistical learning theory and structural risk minimization, is a supervised classification method which can deal with both linear and nonlinear data efficiently with good generalization ability [25,26]. Compared with other methods, SVM has such advantages as it does not need a large number of training samples for developing a model and is not affected by the presence of outliers [27] and has been proven as a reliable and efficient method for the spectral data analysis of agricultural products [28,29,30,31,32]. Artificial neural networks have been widely used in agriculture areas, including surface inspection of fruits [17,33,34,35,36,37,38], since regaining their popularity in the early 1980s. By using a 3-layer feedworward backpropagation ANN model, Pan et al. demonstrated the feasibility of hyperspectral reflectance imaging technique for detecting cold injury of peaches [17], and ElMasry et al. differentiated injured apples from normal ones [36].

In this work, SVM and ANN were used for classification experiments performed on the hyperspectral data of the jujubes. The classification accuracy of SVM and ANN models trained by using HSI data obtained by linear and rotation scan were compared respectively. The Radial Basis Function (RBF) was chosen as the kernel function of SVM, and grid search was used to select the optimal penalty parameter C and the kernel function parameter g. Moreover, the three-layer feedworward backpropagation model was adopted as ANN model in this study. The input layer had 435 nodes, representing the spectral responses of each jujube sample point with 435 spectral bands. The hidden layer had 100 nodes and the output layer had 7 nodes defined primarily by number of skin features. A hyperbolic tangent activation function was used as a transfer function between the input and hidden layers, while the transfer function between the hidden and output layers was a softmax function. The training of SVM and ANN models and prediction of skin defects of jujube samples were performed in the Python environment with the use of scikit-learn and Keras packages. Pixels from the linear scan dataset and the rotation scan dataset were selected respectively from the 50 randomly chosen jujube samples. Each skin feature had the same number of data. The classification accuracy of models trained using data obtained from linear scan and rotation scan were compared. The selected data were divided into a training set (70%) and a validation set (30%). The training set was used to establish classification models, while the validation set was used for evaluating the performance of models. Classification models were evaluated based on the confusion matrices of the classification performed on the validation sets.

3. Results and Discussion

3.1. Linear and Rotation Scan Images

Line scan is a convenient way for imaging cylindrical object because an unwrapped image of the entire surface without distortion can be obtained by rotating the cylinder when scanning. Furthermore, line-scan allows the application to adjust the image pixel aspect ratio by controlling the scan line pitch. If the image pixel aspect ratio was greater than one when imaging a cylindrical object by rotation scan, the distance between scan lines would be greater than the width of scan line, and thus the surface between could not be interrogated. In this work, the rotation scan was set up to let the image of a cylinder with a diameter of 60 mm under rotation scan had a 1:1 image pixel aspect ratio. By using this setting, all of the image pixel aspect ratios of jujube specimens (diameter < 60 mm) under rotation scan were less than one. Therefore, the surface feature of specimens would not be missed.

The pseudo color images of typical jujube specimens obtained by using linear and rotation scan are shown in Figure 4. Because the specimen was put on the rotation stage during scan, a part of the jujube surface was blocked by the supporting plate when scanning. The viewable surfaces of specimens using linear and rotation scan methods are about 49% and 95% of the whole fruit surface respectively. By using linear scan, the intensity distortion of the peripheral areas of fruit was severe due to the curved shape of fruit [14], and thus the pixels in peripheral areas were difficult to use for surface defect detection because the intensity of these pixels was too low. To cover the whole surface of fruit without serious distortion by linear scan, the images taken from the six views shown in Figure 4b were required. Figure 4c shows the rectangular images of specimen obtained using rotation scan. These images were similar to the two-dimensional flat surface of Earth’s curved surface obtained by a cylindrical projection. With the experimental setup used for the rotation scan in this work, the distortion of image is minimal in equatorial regions and maximal at the poles. Because surface defect detection is a qualitative analysis, the shape distortion in the images and replicated sampling of some surface regions are not problems. In the fruit images obtained by using rotation scan, the areas with severe intensity distortion were only located near the upper and lower boundaries of fruit images. Thus the percentage of area with severe intensity distortion of images obtained by rotation scan was less than that of images obtained by linear scan. Furthermore, it can be seen from Figure 4c that the areas occupied by the rectangle and circles could cover the whole surface of specimen without severe distortion. Defect detection can be performed on the pixels in these areas only.

3.2. Spectral Characteristics

Every pixel of HSI data set used in this study has spectral information. The curves of mean reflectance spectra of different types of jujube skin conditions were shown in Figure 5. The spectral characteristics of normal, decay, cracks and rusty spots were relatively similar in the wavelength range of 680–950 nm. The rapid change in reflectance of vegetation, also known as red edge, in the range of 680–730 nm could be clearly observed. The reflectance of the spectral of glare was higher than those of all other skin types in the whole wavelength range. Moreover, the curve shape of spectral of glare is similar to that of normal surface. The possible reason could be most of the glare pixels in the data set of this study were selected from the glossy normal surface. Spectral characteristics of white and black fungus were similar, and were different from those of other skin conditions. The reflectance curves of the spectral of black and white fungi were lower than those of other skin types. Additionally, there was no rapid change in reflectance in the range of 680–730 nm. This could be due to the spectral signatures of pathogenic fungi of jujubes being different from those of green plant.

It was found that there was no reflectance of single band in the wavelength region between 468 nm and 700 nm could be used to segment defect surface from normal using simple thresholding. It was also noticed that the reflectance of the spectra of normal skin was higher than those of other skin types in the region between 710 nm and 840 nm. However, the spectra shown in Figure 5 did not account for the non-uniform light intensity distribution on the spherical jujube surface. The reflectance of the spectra, in the above mentioned wavelength range, of normal surface near the edge could be lower than those of defect surface near the center due to intensity distortion. Thus, using simple thresholding for a single band image in this region might be inadequate for discriminating defects from normal skin. It was further noted that the slope of the curve of reflectance of normal surface spectral was negative, and those of defect surface were positive in the range of 550–630 nm. Therefore, the two-band ratio image based on wavelength images at 550 nm and 630 nm, or the first derivative of the curve of reflectance in the range of 550–630 nm could be considered for developing the detection algorithm to segment the skin defects from normal surface of jujubes.

3.3. Classification Models

Table 1. Classification results of support vector machine (SVM) and artificial neural network (ANN) models with the use of data set obtained by linear and rotation scan.

Classification Model	Precision
Model 1 (SVM trained with linear scan dataset)	96.3%
Model 2 (SVM trained with rotation scan dataset)	97.3%
Model 3 (ANN trained with linear scan dataset)	96.5%
Model 4 (ANN trained with rotation scan dataset)	97.4%

shows the classification accuracy, estimated on the validation set, of SVM and ANN models trained by using HSI data obtained by linear and rotation scan respectively. The classification accuracy was high for all four models and ranged from 96.3% to 97.4%. It can be noted that the accuracy of ANN models was higher than that of SVM models, and the models trained with the use of rotation scan data had higher accuracy than those trained with the use of linear scan data.

To analyze and evaluate the performance of the four tested models for discriminating skin features of the jujube and to show how the pixels with spectral data belonging to different classes were classified by the specific model, the confusion matrices of these models were created and shown in

Table 2. Confusion matrices of the classification models with the use of linear scan dataset.

Actual	Classified as
	Normal	Rusty Spot	Decay	White Fungus	Black Fungus	Crack	Glare
	(%)	(%)	(%)	(%)	(%)	(%)	(%)
Normal	94.32	0.16	0.21	0	0	0.03	5.28
	95.36	0.32	0.21	0.03	0	0	4.08
Rusty spots	4.48	91.23	2.39	0	0.03	0.16	1.71
	4.45	90.41	2.72	0.03	0.11	0.41	1.87
Decay	0.75	11.06	83.49	0.05	3.26	0.05	1.34
	0.5	9.92	84.81	0.1	3.68	0.05	0.94
White fungus	0	0.12	0.09	95.47	1.29	3.03	0
	0	0.12	0.09	94.48	1.56	3.75	0
Black fungus	0.22	2.06	3.64	0.29	93.79	0	0
	0.29	1.54	2.94	0.99	94.24	0	0
Crack	0.27	4.74	0.37	2.69	0.13	81.73	10.07
	0.13	4	0.35	4.21	0.08	86.25	4.98
Glare	0.14	0	0.08	0.59	0	0.03	99.16
	0.19	0.03	0.16	0.03	0	0.18	99.41
Model 1	Model 3

and

Table 3. Confusion matrices of the classification models with the use of linear scan dataset.

Actual	Classified as
	Normal	Rusty Spot	Decay	White Fungus	Black Fungus	Crack	Glare
	(%)	(%)	(%)	(%)	(%)	(%)	(%)
Normal	98.67	0.8	0.29	0	0	0.03	0.21
	98.51	0.53	0.35	0	0	0.05	0.56
Rusty spots	1.3	95.95	2.01	0.03	0.05	0.63	0.03
	1.44	93.83	3.78	0.05	0.08	0.71	0.11
Decay	0.05	4.45	94.02	0.02	1.29	0.17	0
	0.07	2.41	95.94	0.02	1.39	0.12	0.05
White fungus	0	0	0	98.68	0.48	0.84	0
	0	0	0.03	98.38	0.84	0.75	0
Black fungus	0	0.48	2.35	0.18	96.95	0.04	0
	0	0.29	2.17	0.33	97.21	0	0
Crack	0	1.15	0.19	1.27	0	97.39	0
	0	0.4	0.4	0.77	0.03	98.35	0.05
Glare	0.38	0	0	0	0	0	99.59
	0.24	0.05	0	0	0	0.05	99.66
Model 2	Model 4

. The rows of matrix represent actual classes and the columns denote predicted classes. The diagonal entries of the matrixes denote the percentage of pixels of the properly-recognized cases, while each entry outside the diagonal means an error (the percentage of misclassification). The confusion matrixes show that glare was the class from which most correct classification was obtained (higher than 99.16%) for all models. This can be easily explained by the fact that the reflectance of the spectra, in the full spectral range, of glare was much higher than those of other classes (Figure 5). Confusion between decay and rusty spot was higher than that of any other pair of classes, and the second high confusion was between decay and black fungus. One reason of confusion between decay and rusty spot was probably due to the spectral characteristics of decay and rusty spots being relatively similar in the full wavelength range used in this study. Another reason for the confusion between these classes was because decay and rusty were not mutually exclusive. The development of decay could occur on not only normal surface but also rusty surface, thus the pixels selected from decayed rusty surface could pertain to class decay and class rusty. However, in this work, single-output classification models were used so only one label could be assigned to pixels on decayed rusty surface. Therefore, confusion between decay class and rusty surface class could occur. To overcome this problem, multi-label classification models would be required, but this was beyond the scope of this work.

The possible reason of confusion between decay and black fungus could be due to the large variation of reflectance of decay surface. Fungal infections that occurred on jujube surface typically caused a rapid decay in the area surrounding the fungal infection spot. As a jujube ripen, the color of skin changes from green to yellowish. The colors of decay or rot were highly dependent on the maturity stages of the jujube. In RGB image, the color of decay on mature green jujubes was much darker than that of ripen jujubes and was close to the color of black fungus. In this work, data labeling was based on visual assessment of surface features in the RGB image. Thus, it was challenging to differentiate dark decay and black fungus on green mature jujube surface by visual assessment of the surface features of the jujube in the RGB images. Therefore, labeling error about decay and black fungus could occur and lead to the confusion between classes of decay and black fungus. To minimize the labeling effort caused by above mentioned reason, spectral information of pixel can be taken into consideration when labeling. Furthermore, although the averaged reflectance values of the spectra of decay and black fungus were distinctly different (Figure 5), the reflectance of the spectra of dark decay on mature green jujube surface was closer to that of black fungus, and might lead to confusion.

3.4. Spatial Region of Interest for Classification

Figure 6 shows the procedure to create the mask of jujube specimen for pixel based defect detection. As the difference between the spectra for jujubes and background could be easily distinguished, a threshold of the reflectance at a single wavelength (760 nm) was used to segment jujube pixels and background pixels and to create a binary image. In a second step, a jujube mask was created by using morphological dilation and erosion to fill the holes inside the mask and remove isolated jujube pixels outside the mask. Due to the intensity distortion [14] caused by the curvature of the jujube, the acquired intensities of jujube pixels near the edge of the mask was too low and these pixels were highly possible to be misclassified as low reflectance classes such as black fungus and decay (Figure 6b). Although commonly used pre-processing technique such as normalization, dividing each spectrum by maximum value, could be used to minimize the intensity variation due to fruit curvature. However, the signal to noise ratio could not be improved by normalization. In this work, a new mask created by eroding approximately 5% of the original mask was used as the spatial region of interest. Defect classification was only performed on jujube pixels in new mask (Figure 6c).

3.5. Classification Results of Jujubes

Figure 7 shows the prediction results of four example jujubes using the proposed classification models. These examples included all surface features studied in this work. For each sample, RGB image was shown in the first column, and the classification results of SVM and ANN models were shown in the second and third columns respectively. By using linear scan method, only the image of surface facing camera could be obtained. The surface area, defined by the new mask, suitable for pixel based defect detection was about 43% of the whole jujube surface. By using the rotation scan method, about 91% of pixels of the whole jujube surface area could be used for defect classification.

It can be seen from Figure 7 that most of the fruit pixels can be properly classified. However, in Figure 7a, it can be noted that the pixels near the right boundary of the fourth specimen were misclassified as black fungus due to low intensity. Similarly, in the classification results of the third sample shown in Figure 7b, rusty spots near the top of the image misclassified as black fungus could be due to the same reason. It can be noticed that a small region of rusty spots, shown in the circles of Figure 7b, was incorrectly predicted as crack of the fourth jujube sample. By inspecting the color image of this specimen, it can be seen that the rusty area misclassified as crack was brighter than the typical rusty area. The brighter rusty surface could be due to the glare on this rusty area and could be the reason that results in the misclassification. One thing should be pointed out is that jujube surface could have many small rusty spots thus many clusters of pixels classified as rusty spots could be small (only several pixels). Therefore, if image post processing were used to improve accuracy, a particle filter should be used with care to avoid removing small rusty spots by mistake. The Kaohsiung No.11 jujube has smooth calyx; however, the concave feature of stem-end of the Kaohsiung No.11 jujube is similar to that of apple. The color of calyx is similar to that of rusty spot. The color of normal stem-end is the same as that of normal jujube surface; however, the surface of stem-end, compared with other normal surface, could present a relative low intensity due to the curvature of stem-end. Furthermore, during harvesting, the stem of the jujube could remain on the tree and the skin and flesh surrounding the stem could be pulled out of the jujube (Figure 7b (4)). Therefore, it would be necessary to consider methods to recognize stem-end and calyx and to differentiate the normal stem-end and calyx from the real defects in the future work.

The current hyperspectral imaging system had a stable scanning frame rate of 33 frames per second (FPS) and it took about 40 s for rotation scan of one jujube sample. The use of higher frame rate could not guarantee the image data can be taken at stable frame rate with the use of current computer hardware and software. To increase the stable scan rate of this system, a real time vision and motion control system with synchronization features for camera and motors could be used. In this study, we used whole spectral data to train the classification models. In order to improve the classification speed, dimension reduction techniques should be used to reduce data redundancy of the models while retaining important features of the image. Thus, the classification time can be improved and the classification accuracy can be retained.

4. Conclusions

To ensure the inspected fruits are free from defects, images including the whole surface of fruit are necessary for defect detection. Although many efforts have been made by investigators, current solutions provide advantages as well as disadvantages, and a thoroughgoing solution is still not available. Furthermore, most of the whole surface detection methods were developed for area scan imaging systems. The studies about how to inspect the whole surface of the fruits by using line scan imaging system such as a push broom hyperspectral camera are limited. In this study, without using mirrors or multiple cameras, a simple and reliable hyperspectral imaging system for scanning rotating globe-like object was developed. The rotating speed of object could be precisely controlled by using a rotation stage thus the bounding or wobbling of the fruit during rotation on roller could be eliminated. Stable rotation of object is important for line scan imaging systems to obtain high quality images of rotating object. By using this system, about 95% of the whole jujube surface area could be observed in one scan pass. A robotic arm with suction pads could be easily integrated into this imaging system and the second image of the jujube under inspection could be captured after rotating the rotation axis of the jujube about horizontal line 90 degree. The area of whole surface can be covered by these two images and used for defect classification. The comparison of classification accuracy of SVM and ANN models trained by using HSI data obtained by linear and rotation scan respectively showed that the models trained with the use of rotation scan data had higher accuracy than those trained with the use of linear scan data. This line scan HSI system for rotating objects and the defect classification models can be integrated into current automatic jujube grading systems which grades jujubes by weight, and has the potential to be used for the surface defect detection of other fruits, such as peach, pear, and apple. In order to promote the use of hyperspectral imaging for the defect detection of fruits with a glossy surface, further illumination optimization is recommended to reduce the glare.