The Surface Defects Detection of Citrus on Trees Based on a Support Vector Machine

Abstract

Machine learning and image processing have been combined to identify and detect defects in mature citrus fruit at night, which has great research and development significance. First, a multi-light vision system was used to collect citrus UV images, and from these, 1500 samples were obtained, 80% of which were training and 20% were experimental sets. For a support vector machine (SVM) model with “2*Cb-Cr”, “4*a-b-l”, and “H” as the training features, the accuracy of the final training model in the experimental set is 99.67%. Then, the SVM model was used to identify mature citrus regions, detect defects, and output the defective citrus regions label. The average running time of the detection algorithm was 0.84097 s, the accuracy of citrus region detection was 95.32%, the accuracy of citrus defect detection was 96.32%, the precision was 95.24%, and the recall rate was 87.91%. The results show that the algorithm had suitable accuracy and real-time performance in recognition and defect detection in citrus in a natural environment at night.

1. Introduction

China is an agricultural country with a huge market. As the largest citrus-producing country, China accounts for a third of the world’s citrus production [1,2], during which time defect detection and classification are needed. Traditional screening by hand is laborious and time-consuming, but modern developments include combining agricultural machinery with information technology to build a new operating system. Machine vision can realize automatic target judgment through image acquisition, processing and analysis, and the recognition of physical characteristics [3]. The use of machine vision for visual positioning and fruit defect detection on citrus trees provides technical support for citrus picking and yield estimation, leading to precision agriculture and production cost control [4,5].

At home and abroad, current research into citrus surface defect detection focuses mainly on quality detection using spectrum and citrus surface defect detection based on machine vision. Li et al. [6,7] achieved quality detection using spectrum to extract citrus surface features combined with image processing. Rong [8] built a visual testing system platform where the fast-edge and local gray threshold segmentation were used to detect eight kinds of common defects on a navel orange with an accuracy reaching 92%. Abroad, Abdelsalam et al. [9] combined multispectral imaging and adaptive pretreatment for accuracy above 95%. Kondo et al. [10,11] proposed citrus fluorescence defect detection based on a machine vision system. Ketones, a product of citrus decay, fluoresced under UV irradiation and revealed minor surface defects [12]. Chenglin et al. [13] combined color-based k-means and the circle hough transform (CHT) to conduct a preliminary screening of the fruit zone before constructing the feature-based AdaBoost classifier to detect and count unripe citrus with a precision of up to 85.6%. Goel et al. [14] used the fuzzy rule-based classification system (FRBCS) to predict the maturity of unripe tomatoes. Training images, using the difference and ratio of red to green components, sorted the tomatoes into six kinds of maturity, and accuracy reached 94.29%. Momin et al. [15] took citrus unshiu as the test subject, and the fluorescent and white light images were acquired on the assembly line for grading recognition. The system made accurate judgments of the citrus surface, thereby avoiding spoilage during storage.

Citrus quality detection research is still conducted according to size, shape, color, and obvious few defects [16], while research into detecting minor citrus defects is still relatively rare. Md. Abdul Momin et al. [15] pointed out that adding non-obvious decaying citrus to normal citrus would accelerate the decay of the latter. Therefore, studying the non-obvious defects of citrus has a certain significance.

In this paper, recognition counting and defect classification of on-tree citrus at night was investigated. Fluorescence detection of surface defects was proposed based on a support vector machine (SVM) using “2*Cb-Cr”, “4*a-b-L”, and “H” three color features as the image classification training model [17,18]. When combined with image processing, it achieved citrus region recognition and fluorescence defect detection. The output of the region of defect aided growth quality monitoring and yield estimation, as well as technical support for real-time citrus-picking detection [19].

In a word, machine vision detection technology was used to identify and detect the defects of the mature citrus in the natural environment at night so as to realize the yield estimation and quality monitoring of the growing period of citrus. This method has suitable accuracy and real-time for the identification and defect detection of citrus trees in natural environments at night and meets the accuracy requirements of fluorescent defect detection of citrus trees in outdoor environments at night. It can provide technical support for the estimation of fruit and vegetable yield in intelligent agricultural production and has important significance for the intelligent production management of citrus orchards.

2. Materials and Methods

2.1. Selection of Light Source

Unlike most fruit recognition research, UV light is used to excite and recognize targets at night, so the design of the visual system is different [20], especially the light source, which needs to provide both lighting and excitation.

Ketone fluorescence requires UV excitation at certain wavelengths [21] used a fluorescence spectrophotometer to conduct a test on the citrus epidermis and found that emission brightness was the highest when the excitation wavelength was around 380 nm. Experiments were carried out with UV light sources of common specifications, and the fluorescence of the defective parts of citrus mitis under the excitation wavelength of 365, 385, and 395 nm was compared. Analyzing the color images, as shown in Figure 1a–c, revealed that the fluorescence at 365 and 385 nm was relatively obvious, while at 395 nm, it was not obvious because, at that wavelength, a fluorescent lamp looks white. When testing the defects of immature citrus mitis (indigo), the green fluorescence of the defective part was similar to that of the normal indigo skin of immature citrus mitis, which hampered defect identification. Under light at 385 nm, the skin of the immature citrus mitis was dark violet, which contrasted sharply with green fluorescence. The histogram of the citrus image in component a was analyzed, as shown in Figure 1d. There are two peaks at 365 nm: the one on the left is the defective part, and the peak and valley between it and the citrus and background parts were not obvious, so it was difficult to see the defect. As shown in Figure 1e, the histogram at 385 nm had three peaks where the peak and valley between the defective part on the far left and the other peaks were obvious, so the defect was easily separated. As shown in Figure 1f, the histogram at 395 nm had only one peak, which indicated that the defect was not obvious and could not be separated. On the other hand, the UV light at 385 nm was less expensive than other specifications. Therefore, this vision system adopted the Juhong chip UV-LED.

In addition, as the light wavelength emitted by the UV-light-emitting diode (LED) was relatively low, according to the inversely proportional relationship between energy and wavelength, the long-time working UV lamp will cause the ambient temperature to rise and harm humans [22], so it was only turned on during the detection. To find the location of citrus during detection, a JH-3W140P45-T4A model white light LED manufactured by MacroLight (Guangzhou, China) was added to the visual system, the intensity of which was easy to control. It can be turned on for a long period to provide a light environment other than during the detection state.

2.2. Structural Design of the Visual System

The nighttime natural environment was selected for the visual system because it provided stable, even illumination, unlike the daytime when changes in the sun’s position prevented control of the citrus illumination within an appropriate brightness range, so the image brightness was too low or too high.

In this vision system, 12 1 W white light and 18 1 W UV LEDs with at 385 nm wavelength were interlocked and embedded in a 30° converging plane forming a dual light detection source that provided ambient brightness for detecting citrus mitis decay.

The LED drive circuit adopted XL6019 to build a booster circuit and output 20 V of LED power. Constant voltage and limited current were adopted to supply power for both the white and UV lamps. Different resistance values were adopted to adjust the voltage.

Through experiments, the optimal brightness range of this vision research system was determined to be 55–65 Lx. The citrus was placed in a condition without outside light, and the optimum brightness range determined the test (light intensity ≤ 5 Lx). The best brightness of the visual system was determined by starting from 0 and adding 10 Lx increases. To set the best brightness, visual system detection effects under different brightness intensities determined the scope of the best detection of light intensity in mature citrus.

Comparing the detection results of mature citrus under different light conditions, when the light intensity was weaker than 55 Lx or higher than 65 Lx, the defect detection effect will be missed or multiple, so the defective part cannot be well highlighted. The detection of mature citrus was most accurate when the light intensity was within the range of 55–65 Lx, as shown in Figure 2.

Based on the preceding, a self-made multilight-source structure lamp was designed and adopted in this vision system that included white LED and UV lamps at a wavelength of 385 nm and with the structure of a 30° inclined ring panel. Referring to the principle of a shadowless lamp with multi-angle irradiation, combined with the circular shape of citrus mitis, the system light source was designed to be an inclined ring lamp to achieve uniform illumination on the surface of citrus mitis and eliminate the shadow. To increase fluorescence excitation energy, the system light source should be concentrated by tilting it to a certain angle. Comparing the fluorescence of the defective part of citrus mitis under the irradiation of a planar circular and an inclined ring light source with an angle of 30°, the inclined ring light source concentrated UV light on the surface of citrus mitis, which enhanced the fluorescence of the defective part. Because the white and violet LEDs were arranged alternately, they were concentrated and evenly distributed on the surface of the citrus. The structure is shown in Figure 3.

2.3. Image Acquisition

The vision system was composed of a high-definition (HD) camera, white LED lamp, violet LED lamp, STC15 microcontroller, and other hardware. The camera was Logitech’s C525, with autofocus and a resolution of more than 5 megapixels. The white LED light source was JH-3W140P45-T4A, produced by Juhong Light Source Co., Ltd. (Guangzhou, China). Its color temperature was 5000–6000 k, and its luminous flux was 190–210 LM/W. The wavelength of the violet LED light source was 380–385 nm, luminous flux was 20–30 LM, and voltage was 3.4–3.6 V, which provided an effective and stable nighttime light source. The microcomputer was the STC15F2K60S2 (Shanghai, China) of the STC15 series. The AOD4184 MOS (Shenzhen, China) field-effect rectifier tube was used to control the level switch through the power drive chip XL6019 (Shenzhen, China). The MBRD1045 10A/45V patch (Shenzhen, China), TO-252 rectifier (Shenzhen, China), and 45 G Schottky diode were used to build the boost circuit. The acquisition method of citrus images was as follows: the camera at the center of the self-made light source and LED lamp constituted the visual system, which was about 150–200 mm from the citrus fruit. Images were filmed in parallel while the system was controlled by a PC using the GUI interface. Image acquisition was conducted on the evening of 23 January 2019 in an orchard of South China Agricultural University, Guangdong province. The citrus variety was mature citrus mitis, and the image resolution was 1920 × 1080 pixels. The image acquisition scene is shown in Figure 4. The center of the lighting vision system was 15–20 cm from the center of the acquisition object at the same level. To ensure the comprehensiveness of the samples, 154 citrus UV images of different citrus regions from different perspectives were collected. From these, 139 images were selected: 62 non-defective and 77 defective. Of the 139 samples, 50 defective and 50 non-defective images were selected for the collection of training model samples. The remaining 27 defective and 12 non-defective UV images were used to test the accuracy of the algorithm. Two kinds of light sources were used to collect the white light and UV images, as shown in Figure 5.

3. Method

3.1. Training of SVM Model

The SVM [23] first proposed by Vapnik [24] had a machine-learning algorithm based on statistical and Vapnik–Chervonenkis (VC) dimension theory to minimize structural risk [25]. The main idea was to establish a hyperplane as a decision surface to maximize the isolated edges between positive and negative samples [26]. In this paper, the SVM method was used to identify citrus regions and detect defects.

3.2. Analysis of Image Color Features

The color features of a large number of citrus UV images were analyzed, and the three color features “2*Cb-Cr”, “4*a-b-l”, and “H” were selected as sample attributes to train the SVM model.

Figure 6 shows the histogram of each component of a 512 × 288 citrus UV image in the YCbCr, Lab, and HSI color models. This citrus UV image had a fluorescence defect, but it occupied the smallest area, whereas the background occupied the largest; therefore, the histogram with a peak ordinate of less than 500 was the defective part, 500–9000 was the citrus portion, and >9000 was the background. To highlight the histogram of the defective part, the ordinate range of the partial histogram in Figure 6 was adjusted. As shown in Figure 6b, in the YCbCr color model, the grayscale of citrus defects in the histogram of the Cb component was distributed in the range of 60–120, while the grayscale of citrus and background was in the range of 120–180. In Figure 6c, the grayscale of citrus in the Cr component histogram was distributed within the range of 110–130, while the background part was within the range of 130–153. Therefore, the Cb and Cr components can separate citrus, defect, and background well, but classification may still be misjudged due to the close proximity of the gray scale regions of the three parts. As shown for the Lab color model in Figure 6d–f, the grayscale level of the defective part in the three components was in the front or back end of the histogram distribution, so the defective part was more prominent in this model. For the H component of the HSI color model in Figure 6g, there was a large interval between the defective part and the background and citrus parts, so the Lab model and the H component separated the defective part better.

Through the analysis of the YCbCr, Lab, and HSI color model components, the Cb component, the defective part, and the non-defective part hardly overlapped, but the boundary was not obvious. In the Cr component, the characteristic values of citrus and background almost did not overlap; the boundary was not obvious, and the defect was not prominent. In this paper, the among of the three kinds of eigenvalues were increased through the algebraic operation of Cb and Cr components, and the “2*Cb-Cr” eigenvalues were used to separate the defective part, citrus, and background. The eigenvalue boxplot of the three parts is shown in Figure 7. In the “2*Cb-Cr” features of 1500 samples, the citrus eigenvalues in the 25–75 percentile range were 107–123, while those for the defective part were 13–67, and for the background, 131–134. Apparently, the “2*Cb-Cr” eigenvalues separate citrus, defects, and background.

Due to the large span of the characteristic defect values in the “2*Cb-Cr” feature, misjudgment may have occurred. To separate the defective part from the citrus more effectively, the algebraic operation “4*a-b-l” of each component in the Lab color model and the H component of the HSI color model was used. The result was that the defective part, citrus, and background were effectively separated.

The boxplot of the 4*a-b-L feature is shown in Figure 8. The eigenvalues of citrus in the 25–75 percentile range were 307–360; for the defective part, 50–172; and for the background, 358–394. Apparently, the eigenvalue span of the defective part in the “4*a-b-L” features was small. The defects, citrus, and background overlapped little, so citrus, defect, and background separated well. In addition, it can be seen from Figure 9 that for the “H” component, eigenvalues in the 25–75 percentile range were 0.17–0.19 for the defective part and 0.59–0.63 for the background part. These are quite different from each other, so the defective part was well separated.

Therefore, the feature means of citrus, defects, and background in “2*Cb-Cr”, “4*a-b-l”, and “H” features were combined to train the SVM model.

3.3. Selection of SVM Kernel Function

The LIBSVM toolbox was selected for SVM model training. The toolkit has available linear, polynomial, RBF, and sigmoid kernel functions. In this paper, the predicted accuracy for the test set with the same parameters but different kernel functions was compared, and the segmentation accuracies of models with different kernel functions were 99.67%; thus, the commonly used and better performance RBF kernel function was selected for model training to classify each pixel of the orange UV image. The setting of the penalty parameter c and kernel function parameter g were determined through cross-validation, where the best penalty parameter c was 32,768, and the best kernel function parameter g was 9.03 × 10⁻⁸.

4. Fluorescent Detection for Citrus Defects

4.1. Detection Algorithm Flow

The flowchart of the fluorescence detection algorithm for citrus surface defects is shown in Figure 10.

4.2. Segmentation of Citrus UV Image

Based on the SVM model, fluorescent defect detection was used to identify the areas of citrus and surface defects. It was used to divide the citrus UV images into defective part, normal part, and other. To ensure the accuracy of the model, 1500 samples (500 defective citrus, 500 normal citrus, and 500 background) were taken from 100 citrus UV images for model training and accuracy testing. To determine the accuracy of the training model, 80% of the 1500 samples were used as the training set, and 20% were used as the test set. To ensure the completeness of the training set, the 500 defective part images comprised obvious fluorescence and insignificant fluorescence, the 500 non-defective images comprised bright spots and no spots, and the 500 background images were made up of leaves, branches, and others.

The image acquired by the multilight-source vision system had a resolution of 1920 × 1080 and a size of 1280 × 720. To reduce the computational load of the algorithm and speed up operation, the citrus UV images were compressed into 512 × 288 without affecting the image quality. The compressed image was converted into YCbCr, Lab, and HIS color models, from which the three color features of “2*Cb-Cr”, “4*a-b-l”, and “H” were obtained. For each image pixel, these features constituted a feature matrix where the number of image pixels was the number of rows, and the number of features was the number of columns, which were the input variables of the SVM. The trained model classified each pixel according to the feature matrix, which divided the citrus UV image into three parts: defective, non-defective, and background, and output a label matrix. The classification results of the SVM model are shown in Figure 11. Figure 11a–d represent the original image, the normal-part pixels of the citrus, the defective part pixels of the citrus, and the background part, respectively.

4.3. Marking and Defect Detection of Citrus Areas

After classification by the SVM model, the normal-part image revealed gaps and holes in the detection of the citrus, as shown in Figure 11b. This may have been caused by leaves or branches covering citrus on trees or by epidermal defects. To obtain a more complete citrus region, the normal-part binary image of citrus after SVM classification was closed to fill in the gaps in the shaded part and the holes in the defective part. The binary image is shown in Figure 12a, and the result of the closed operation is shown in Figure 12b. Afterward, there were still small areas in the normal part of the citrus because the dim part of the citrus was ignored in the OTSU threshold segmentation [27] (leaving only a small white area), most of the citrus was blocked, and the exposed area was relatively small. Small areas of citrus can be ignored, so they can be removed by deleting the small area algorithm, as shown in Figure 12c. According to the binary image of the normal part of citrus after processing, the citrus area in the original image was marked by a rectangle to label it, as shown in Figure 12d.

The defects were identified and cut out of each marked citrus region in the original images and corresponding label matrix. Binary images of the citrus region were obtained according to the citrus UV image and the label matrix after clipping. Then the average value of the calculated binary images was used to judge whether there were defects in the citrus region. It can be seen from Figure 13 that the pixel value of the defective part of the non-defective citrus region was 0, while that of the defective citrus region was 1. Therefore, if the mean value of the binary image of the defective part in the citrus region is not 0, the citrus region had a defect.

5. Experiment Design and Results Analysis

5.1. Running Time Test of Detection Algorithm

The computer configuration used in this study is Intel Core i5-6300HQ CPU @ 2.30 GHz × 4. The operating system is Windows 10, and the programming language is Matlab 2016a.

As mentioned above, 139 of 154 citrus UV images were selected as samples after screening, and of these, 100 were used to collect the features of the training model, while the remaining images were used to test the feasibility of the algorithm. Therefore, the batch operation was performed on 39 citrus UV images, and each image was used to identify and detect the citrus regions and output the regions with defects. Finally, the average running time of the fluorescence detection algorithm was found to be 0.84097 s.

5.2. Effect Test of Citrus Region Detection

To test the algorithm’s accuracy in citrus region detection, the 39 citrus UV images were identified. Because citrus grows in spring, it is difficult to label them individually. Therefore, the presence of citrus in the marked area was judged to be correct detection, and the number of correct detection rectangles was denoted as correct. When there was no citrus in the marked area, error detection was considered, and an error detection rectangle denoted it. If the citrus in the image with obvious and less shade was not marked, it was considered to have missed detection, and the number of rectangles of missed detection was denoted as misses. Figure 14 shows the detection effect of citrus regions in part of the citrus UV image. In Figure 14a, green rectangle 6 misidentified leaves in the rectangle instead of citrus, which was caused by the misjudgment of the model. With the exception of rectangle 6, the green rectangles correctly identified citrus, as shown in Figure 14a. There were two reasons for missing detection. First, the whole epidermis of the citrus was defective, which led to the failure of the SVM to identify the non-defective part, as shown in the red rectangle in Figure 14b. Second, citrus leakage in the edge of the tree far from the light vision system had a brightness darker than that of the citrus in the center of the visual system; thus, the OSTU threshold segmentation in that region of pixels determined it to be a negative sample, which caused detection to be missed, as shown the red rectangle in Figure 14c. It should be noted that the main task of this study is to detect the normal and defective pixels of citrus. Therefore, the detection result is correct even though there are multiple targets in one prediction box.

Formula (1) was used to calculate the accuracy of citrus region detection: 407 correct, 1 wrong, and 21 missing detection rectangles were finally counted. Therefore, the detection accuracy of citrus regions in the 39 citrus UV images was 94.87%.

$$Accuracy=correct\times 100/\left(correct+miss+error\right)$$

(1)

5.3. Effect Test of Defect Detection

To test the detection effect of citrus defects in the fluorescent detection algorithm, defect detection was carried out on the regional detection results of 39 citrus UV images. Detection was calculated by Formulas (2)–(4) and described with accuracy, precision, and recall [28]. A total positive (TP) result is a condition when a defect is actually present in the citrus region and also judged to be present; a false negative (FN) occurs when there are actual defects, but none is judged to exist; a total negative (TN) is the case where there are no defects, and none is judged to exist, and a false positive (FP) occurs when there are no defects but hey are judged to exist [29].

$$Accuracy=\left(TP+TN\right)\times 100/\left(TP+FN+TN+FP\right)$$

(2)

$$\mathit{Precision}=\mathit{TP}\times 100/\mathit{TP}+\mathit{FP}$$

(3)

$$Recall=TP\times 100/\left(TP+FN\right)$$

(4)

Table 1. Test statistics results of citrus regional defect detection.

Citrus Region	Defective Area	Non-Defective Area
Area number	91	317
TP	80	-
FN	11	-
TN	-	313
FP	-	4
Correct number	393
Recall	87.91%
Precision	95.24%
Accuracy	96.32%

shows the statistical results of the defect detection test in the citrus region, from which it can be seen that the accuracy was 96.32%, precision was 95.24%, and recall was 87.91%. The fluorescence detection algorithm for surface citrus defects had high accuracy and progress; however, the low recall was due to the fact that there were many non-obvious defects in the citrus UV image, and the area was small, so such samples were difficult to sample in the training set. In addition, the less obvious citrus defects were pale yellow, similar to the color of some citrus epidermis; as a result, the SVM model misjudged these small defects as belonging to the non-defective part of citrus.

6. Discussion and Conclusions

In order to improve the cognitive ability of visual monitoring of fruit quality in a natural environment, a fluorescence detection algorithm for citrus surface defects was proposed, and a visual system was designed to collect citrus images under two light sources at night. According to the absorption spectrum of citrus epidermis, the suitable UV wavelength was determined. To obtain a stable lighting environment with less interference, white and UV light sources were set on the visual system under low external light. The switching control of the two light sources was realized, and on-tree citrus images were obtained from the same angle under two kinds of light.

The detection and classification of citrus defects under UV light were carried out. First, according to the circle detection result of the white light image, the background was removed from the UV image, and the fruit area was defined. The peak number of a component gray Lab color model histogram was judged based on whether there was a defective area in the image. Then, the improved k-means clustering algorithm was applied to the images containing defects to segment the citrus defect areas in the images that were stimulated by UV fluorescence. According to the fluorescent area of scratched and crabbed fruit that had been classified and manually counted in advance, the classification criteria were set, and the defects were divided into two types: scratched and crabbed fruit. The two defect regions were compared to the original citrus fruit regions to identify where the defect was located, marked, and counted in the UV image. Finally, the detection and classification of the defective fruit were realized. Compared with manual judgment, the precision of defect detection was 95.24%, recall was 87.91%, and accuracy was 96.32%. In the classification of the defective fruit, only one of the 427 true positive samples was misclassified. The experimental results show that fluorescence under UV light can be used to identify defects in citrus with high accuracy.

Multiple light sources and machine vision were combined to count and detect the quality of the fruit on citrus trees. The accuracy of the whole algorithm was 99.67%, and the average running time was 0.84097 s. The results showed that this algorithm effectively detected the nighttime state of on-tree citrus and provided technical support for the estimation of citrus yield and the night vision intelligence of a picking robot.

The algorithm in this paper had two problems. First, in the detection of citrus fruits and defects, citrus that were far away from the light vision system and had low brightness and defects that were not obvious and had small areas may be missed. Second, when the target was blocked by leaves, the algorithm may not be able to accurately detect citrus and its defects.

In view of the above problems, we adopted a deep learning method in the later stage to optimize the detection performance of the orange UV image so as to achieve a citrus area picking robot based on a single mark and provide picking points with higher recall for defect detection.