# Lychee Fruit Detection Based on Monocular Machine Vision in Orchard Environment

^{1}

^{2}

^{3}

^{*}

^{†}

## Abstract

**:**

_{1}-score was 87.07%.

## 1. Introduction

## 2. Materials

#### 2.1. Datasets

#### 2.2. Application Hardware Architecture Design

## 3. Lychee Fruit Detection Method

#### 3.1. Image Preprocessing and Foreground Segmentation

#### 3.1.1. Image Preprocessing

_{h}represents the segmented region after using the CLAHE algorithm and ${R}_{r}$ represents the manually segmented standard region for reference. As the results show in Table 1, when the number of blocks was 10 × 10 and the contrast enhancement limit was 0.015, the average relative overlap rate reached the highest value, 86%. Therefore, 10 × 10 and 0.015 were selected as the parameters to be used for the CLAHE algorithm in this paper.

#### 3.1.2. Lychee Foreground Segmentation

#### 3.2. Two-Step Potential Lychee Region Extraction

#### 3.2.1. Distinguishing the ROIs between Isolated and Overlapped Lychee Fruits

**Input:**Binary images of lychee fruits foreground after segmentation I

_{f}.

**Step 1.**Select every foreground region C

_{n}from I

_{f}one by one. (n = 1, 2, …, N, N is the number of lychee regions)

**Step 2.**Extract the edge of C

_{n}and operate the Hough transform.

**Step 3.**Calculate the circle center Oh

_{n,i}(X

_{n,i}, Y

_{n,i}) and radius Rh

_{n,i}of each Hough circle. (i = 1, 2, …, p, p represents the number of Hough circles generated in C

_{n})

**Step 4.**Generate the equivalent foreground area circles of C

_{n}by calculating its circle center Oa

_{n,j}(X

_{n,j}, Y

_{n,j}) and radius Ra

_{n,j}. (j = 1, 2, …, q, q represents the number of equivalent foreground area circles in C

_{n})

_{n,j}(X

_{n,j}, Y

_{n,j}) equal the coordinates of the center of gravity in C

_{n}, the radius Ra

_{n,j}is found using Equation (2):

**Step 5**. Calculate lychee fruits status S

_{n}of C

_{n}using Equations (3)–(8), where S

_{n}= 1 means single isolated, S

_{n}= 2 means occluded, S

_{n}= 3 means overlapped status.

_{n}, the green circles represent the Hough circles Oh

_{n,i}of C

_{n}and the blue circles represent the equivalent foreground area circles Oa

_{n,j}of C

_{n}.

_{n,j}, the equivalent foreground area circle radius is R

_{an}, the number of equivalent foreground area circles is $p$, and the number of Hough circles is $q$, then the following expressions can be obtained:

_{n}.

**Output:**Lychee fruits status S

_{n}.

#### 3.2.2. Individual Fruit Extraction from Overlapped Lychee Regions

**Input:**The foreground region C

_{n}contains overlapping lychee fruits determined by ACHC (S

_{n}= 3).

**Step 2.**Take the center of gravity A as the origin of the polar coordinate system.

**Step 3.**Calculate the distance from the origin A to the pixel points (indexed by d) on the edge of the domain using Equation (9) for every degree (360°) in a counter-clockwise direction, as shown in Figure 8B.

_{d}in the XOY coordinate system are $A\left({x}_{A},{y}_{A}\right)$ and ${E}_{d}\left({x}_{Ed},{y}_{Ed}\right)$, respectively. Figure 9 shows an example of a geometric calculation model of two overlapping lychee fruits.

**Step 4.**Calculate every maximal value point p

_{j}and minimal value point q

_{i}on the edge E (i = 1, 2, …, m, m is the number of minimal value points; j = 1, 2, …, m, m is the number of maximal value points).

**Step 5.**Separate the edge E by “q

_{i}-p

_{j}-q

_{i+1}” order, as shown in Figure 8B (“q

_{i}-p

_{j}-q

_{i+1}” represents “local minimal value point - local maximal value point - local minimal value point” order).

**Step 6**. Determine circles by every set of three extreme points “q

_{i}-p

_{j}-q

_{i+1}”, as shown in Figure 8C.

**Output:**Location and size of every single lychee fruit in the foreground region C

_{n}.

#### 3.3. LBP-SVM Recognition of Lychee Fruit

## 4. Results and Discussion

#### 4.1. Performance Evaluation under Well-Illuminated Conditions without Using LBP-SVM Classifier

_{1}-score is used to combine the metrics, including the number of TPs, the number of FPs and the number of FNs. The F

_{1}-score is adopted and defined as follows:

_{1}-score for the different methods are also shown in Table 2. The results also demonstrate that the proposed Method A is more appropriate. Table 2 shows that: (1) The proposed method achieved acceptable detection results in both test datasets, where the detection performance is similar. Among them, their recall rate is around 89%, accuracy rate is around 80% and F

_{1}-score is close to 84%, indicating that proposed Method A is not sensitive to different settings of image acquisition procedure and has a certain generalization ability. (2) The detection speed is the average detection time (seconds per frame) for the test dataset. The time consumption of Method A in detecting images with lychee fruits is around 1 s, which is a little higher than the average time consumption of the other three methods (Method B is 0.745 s, Method C is 0.821 s, Method D is 0.654 s). (3) The test results of Method C indicate that it had the highest in recall rate, 93.02%, in three methods, which is about 4%–5% higher than the other two methods, but the testing precision rate of Method C is only 68.38%, which is caused by the oversegmentation phenomenon of watershed transform algorithm, and thus produces far more numbers of FP results than the other two methods. (4) The test results of Method D indicate that it had the fastest average detection time. The amount of FP (false positive) results is similar to Method A and Method B in dataset A1. The recall rate of Method D is 80.34%.

#### 4.2. Performance Evaluation under Overexposure and Weakly Illuminated Conditions without Using the LBP-SVM Classifier

_{1}-score 82.38%. In contrast, as the sample shows in Figure 12E,F, the recall rate of Method B is 82.48% with a precision rate of 76.23%.

#### 4.3. Performance Evaluation in an Orchard Environment Using the LBP-SVM Classifier

_{1}-score were 87.48%, 86.66% and 87.07%, respectively. The highest TP rate was 88.75%, which was achieved under well-illuminated conditions. The lowest TP rate was 84.70%, which was under weakly illuminated conditions. The time consumed by the proposed Method A mainly occurred during clustered lychee extraction and clustered lychee matching. The time consumed by the isolated lychee detection mainly occurred in the searching of the fruit regions using the LBP-SVM classifiers, and the total average processing time was nearly 0.2 s longer than that without using the LBP-SVM classifier. The average time consumed from the extraction of clustered overlapped lychee fruits to fruit localization was 1.412 s. However, adopting the LBP-SVM classifier greatly reduced the number of FPs—from 126 down to 80—under overexposure illumination, which increased the precision rate from 81.08% to 87.10%. Similarly, the LBP-SVM classifier reduced the number of FPs from 67 down to 38 under weakly illuminated conditions, and the precision rate correspondingly increased from 77.21% to 85.66%. Moreover, the LBP-SVM classifier reduced the number of FPs from 151 down to 81 under well-illuminated conditions, and the precision rate subsequently increased from 80.49% to 88.49%. The LBP-SVM classifier improved the recall rate of detection by approximately 7%.

## 5. Conclusions

_{1}-score is 84.42%.

_{1}-score of 82.38%. However, without using the LBP-SVM classifier, the precision rate is only 79.9% as a result of misjudging background chaff interferences. Furthermore, the results demonstrated that the proposed method can be used for fruits in different levels of maturity, including lychee fruits.

## Author Contributions

## Funding

## Acknowledgments

## Conflicts of Interest

## Abbreviations

ACHC | Equivalent foreground area circles and Hough circles |

CLAHE | Contrast limited adaptive histogram equalization |

HIK-SVM | Histogram intersection kernel based support vector machine |

HSI | Hue/saturation/intensity color space |

HSV | Hue/saturation/value color space |

IPC | Industrial personal computer |

LBP | Local binary pattern |

LDA | Linear discriminant analysis |

PCEVP | Polar coordinate extreme value projection |

RGB | Red/green/blue color space |

ROI | Region of interest |

UAV | Unmanned aerial vehicle |

UGV | Unmanned ground vehicle |

YIQ | YIQ color space, where Y is Luminance, I is in-phase, and Q is quadrature |

YCbCr | Luma/Blue chromaticity component/Red chromaticity component |

## References

- Kong, D.; Xue, Y.; Mao, L.; Wang, K.; Chen, H.; Huang, K. Lychee image segmentation algorithm based on ant colony and space constraints FCM. Comput. Eng. Appl.
**2013**, 49, 1877–1893. [Google Scholar] [CrossRef] - Guo, A.X.; Zou, X.J.; Zou, H.X. Recognition on image of lychee fruits and their main fruit bearing based on twin-threshold method. Comput. Eng. Des.
**2014**, 35, 557–561. [Google Scholar] [CrossRef] - Kurtulmus, F.; Lee, W.S.; Vardar, A. Green citrus detection using ‘eigenfruit’, colour and circular gabor texture features under natural outdoor conditions. Comput. Electron. Agric.
**2011**, 78, 140–149. [Google Scholar] [CrossRef] - Lü, J.; Zhao, D.; Ji, W.; Chen, Y.; Shen, H.; Zhang, Y. Fast positioning method of apple harvesting robot for oscillating fruit. Trans. Chin. Soc. Agric. Eng.
**2012**, 28, 48–53. [Google Scholar] [CrossRef] - Sujay, R.N.; Deka, P.C. Support vector machine applications in the field of hydrology: A review. Appl. Soft Comput.
**2014**, 19, 372–386. [Google Scholar] [CrossRef] - Liu, T.H.; Ehsani, R.; Toudeshki, A.; Zou, X.J.; Wang, H.J. Detection of citrus fruit and tree trunks in natural environments using a multi-elliptical boundary model. Comput. Ind.
**2018**, 99, 9–16. [Google Scholar] [CrossRef] - Wang, D.D.; Song, H.B.; Tie, Z.H.; Zhang, W.Y.; He, D.J. Recognition and localization of occluded apples using K-means clustering algorithm and convex hull theory: A comparison. Multimed. Tools Appl.
**2016**, 75, 3177–3198. [Google Scholar] [CrossRef] - Sa, I.; Ge, Z.; Dayoub, F.; Upcroft, B.; Perez, T.; McCool, C. DeepFruits: A Fruit Detection System Using Deep Neural Networks. Sensors
**2016**, 16, 1222. [Google Scholar] [CrossRef] - Bac, C.W.; Hemming, J.; Van Henten, E.J. Stem localization of sweet-paper plants using the support wire as a visual cue. Comput. Electron. Agric.
**2014**, 105, 111–120. [Google Scholar] [CrossRef] - Font, D.; Tresanchez, M.; Martínez, D.; Moreno, J.; Clotet, E.; Palacín, J. Vineyard yield estimation based on the analysis of high resolution images obtained with artificial illumination at night. Sensors
**2015**, 15, 8284–8301. [Google Scholar] [CrossRef] - Wang, C.L.; Tang, Y.C.; Zou, X.J.; SiTu, W.M.; Feng, W.X. A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot. Optik
**2017**, 131, 626–631. [Google Scholar] [CrossRef] - He, Z.L.; Xiong, J.T.; Lin, R.; Zou, X.; Tang, L.Y.; Yang, Z.G.; Liu, Z.; Song, G. A method of green lychee recognition in natural environment based on improved LDA classifier. Comput. Electron. Agric.
**2017**, 140, 159–167. [Google Scholar] [CrossRef] - Xu, Y.; Li, Y.H.; Song, H.B.; He, J.D. Segmentation method of overlapped double apples based on Snake model and corner detectors. Trans. Chin. Soc. Agric. Eng.
**2015**, 31, 196–203. [Google Scholar] [CrossRef] - Xiang, R.; Jiang, H.Y.; Ying, Y.B. Recognition of clustered tomatoes based on binocular stereo vision. Comput. Electron. Agric.
**2014**, 106, 75–90. [Google Scholar] [CrossRef] - Liu, X.; Zhao, D.; Jia, W.; Ruan, C.; Tang, S.; Shen, T. A method of segmenting apples at night based on colour and position information. Comput. Electron. Agric.
**2016**, 122 (Suppl. C), 118–123. [Google Scholar] [CrossRef] - Nguyen, T.T.; Vandevoorde, K.; Wouters, N.; Kayacan, E.; De Baerdemaeker, J.G.; Saeys, W. Detection of red and bicoloured apples on tree with an RGB-D camera. Biosyst. Eng.
**2016**, 146, 33–44. [Google Scholar] [CrossRef] - Chen, S.W.; Shivakumar, S.S.; Dcunha, S.; Das, J.; Okon, E.; Qu, C.; Taylor, C.J.; Kumar, V. Counting apples and oranges with deep learning: A data-driven approach. IEEE Rotot. Autom. Lett.
**2017**, 2, 781–788. [Google Scholar] [CrossRef] - Zhuang, J.J.; Luo, S.M.; Hou, C.J.; Tang, Y.; He, Y.; Xue, X.Y. Detection of orchard citrus using a monocular machine vision-based method for automatic fruit picking applications. Comput. Electron. Agric.
**2018**, 152, 64–73. [Google Scholar] [CrossRef] - Xiong, J.T.; He, Z.L.; Lin, R.; Liu, Z.; Bu, R.B.; Yang, Z.G.; Peng, H.X. Visual positioning technology of picking robots for dynamic lychee clusters with disturbance. Comput. Electron. Agric.
**2018**, 151, 226–237. [Google Scholar] [CrossRef] - Suchet, B.; James, U. Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchard. J. Field Robot.
**2017**, 1039–1060. [Google Scholar] [CrossRef] - Senthilnath, J.; Dokania, A.; Kandukuri, M.; Ramesh, K.N.; Anand, G.; Omkar, S.N. Detection of tomatoes using spectral-spatial methods in remotely sensed RGB images captured by UAV. Biosyst. Eng.
**2016**, 146, 16–32. [Google Scholar] [CrossRef] - Zuiderveld, K. Contrast Limited Adaptive Histograph Equalization. In Graphic Gems IV; Academic Press Professional: San Diego, CA, USA, 1994; pp. 474–485. [Google Scholar] [CrossRef]
- Ji, W.; Tao, Y.; Zhao, D.; Yang, J.; Ding, S. Iterative Threshold Segmentation of Apple Branch Images Based on CLAHE. Trans. Chin. Soc. Agric. Mach.
**2014**, 45, 69–75. [Google Scholar] [CrossRef] - Xiong, J.T.; Zou, X.; Chen, L.; Guo, A. Recognition of mature lychee in natural environment based on machine vision. Trans. CSAM
**2011**, 42, 162–166. [Google Scholar] [CrossRef] - Otsu, N. A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern.
**1979**, 9, 62–66. [Google Scholar] [CrossRef] - Luo, L.; Zou, X.; Yang, Z.; Li, G.; Song, X.; Zhang, C. Grape image fast segmentation based on improved artificial bee colony and fuzzy clustering. Trans. CSAM
**2015**, 46, 23–28. [Google Scholar] [CrossRef] - Fu, L.S.; Wang, B.; Cui, Y.J.; Su, S.; Gejima, Y.; Kobayashi, T. Kiwifruit recognition at night-time using artificial lighting based on machine vision. Int. J. Agric. Biol. Eng.
**2015**, 8, 52–59. [Google Scholar] [CrossRef] - Ojala, T.; Pietikainen, M.; Maenpaa, T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal.
**2002**, 24, 971–987. [Google Scholar] [CrossRef] - Vapnik, V.N.; Chervonenkis, A.Y. On the Uniform Conver Gence of Relative Frequencies of Events to Their Probabilities. In Measures of Complexity; Springer: Cham, Switzerland, 2015; pp. 11–30. [Google Scholar] [CrossRef]

**Figure 1.**Examples of lychees in an orchard environment with a resolution of 100 × 100 pixels. (

**A1**) Well-illuminated examples of dataset 1. (

**A2**) Weakly illuminated examples of dataset 1. (

**A3**) Overexposure-illuminated examples of dataset 1. (

**B1**) Well-illuminated examples of dataset 2. (

**B2**) Weakly illuminated examples of dataset 2. (

**B3**) Overexposure-illuminated examples of dataset 2. (

**C1**) Non-lychee samples of dataset 1. (

**C2**) Non-lychee samples of dataset 2.

**Figure 4.**Effect of lychee image illumination compensation. (

**A**) Weakly illuminated lychees. (

**B**) Weakly illuminated lychees after illumination compensation. (

**C**) Overexposed lychees. (

**D**) Overexposed lychees after illumination compensation.

**Figure 5.**RGB color component statistics of lychees, immature lychees and background. (

**A**) Example diagrams and sample red line. (

**B**) The R and B component curves corresponding to the sampling red line. (

**C**) Results of R-B chromatic mapping with morphological algorithm.

**Figure 6.**Performance evaluation of lychee foreground segmentation. (

**A**) Original image of a lychee cluster, which is labelled I

_{1}. (

**B**) Foreground segmentation result of I

_{1}. (

**C**) Original image of a lychee cluster, which is labelled I

_{2}. (

**D**) Foreground segmentation result of I

_{2}.

**Figure 7.**Topological diagrams of the lychee location relationships. (

**A**) Single isolated status. (

**B**) Occluded (covered with leaves and branches) status. (

**C**) Overlapped status.

**Figure 8.**Example of image processing using polar coordinate extreme value projection (PCEVP) processing. (

**A**) Determine the center of gravity of the foreground region (

**B**) by calculating all local maximal and minimal value points on the boundary of the foreground region. (

**C**) Individual foreground regions in (

**B**) segmented using PCEVP. (

**D**) The Euclidean distance $\left|A{E}_{d}\right|$ and extreme value points of the foreground region boundary. (

**E**) The Euclidean distance $\left|A{E}_{d}\right|$ in bar graph in a 3D coordinate system.

**Figure 9.**Geometric calculation model of the extreme value points of the boundary of two lychees in the foreground region.

**Figure 11.**Results of lychee fruit cluster detection performance under well-illuminated conditions without using the local binary pattern based support vector machine (LBP-SVM) classifier. (

**A**) Results of the proposed Method A. (

**B**) Results of Method B. (

**C**) Results of Method C. (

**D**) Results of Method D.

**Figure 12.**Results of isolated and lychee fruit cluster detection performance under overexposure and weakly illuminated conditions without using the LBP-SVM classifier. (

**A**,

**B**) Foreground segment results without using image preprocessing. (

**C**,

**D**) Foreground segment results using image preprocessing. (

**E**,

**F**) Results of lychee fruit detection using Method A.

**Figure 13.**Lychee fruit recognition results in orchard environments using the LBP-SVM classifier. (

**A**,

**C**) Results without using the LBP-SVM classifier. (

**B**,

**D**) Results using the LBP-SVM classifier.

**Table 1.**The average relative overlap rate S of 25 sets of parameters used in contrast limited adaptive histogram equalization (CLAHE).

The Contrast Enhancement Limit | The Number of Blocks | ||||
---|---|---|---|---|---|

5 × 5 | 8 × 8 | 10 × 10 | 12 × 12 | 15 × 15 | |

0.005 | 78% | 79% | 80% | 80% | 77% |

0.01 | 78% | 81% | 83% | 80% | 78% |

0.015 | 79% | 84% | 86% | 82% | 80% |

0.02 | 79% | 81% | 85% | 82% | 82% |

0.025 | 78% | 79% | 82% | 80% | 81% |

**Table 2.**Detection results of overlapped lychees under well-illuminated conditions without using the LBP-SVM classifier.

Method | Test Dataset | Average Detection Time (s) | Total Lychee Fruits | TP | FN | FP | Precision (%) | Recall (%) | F_{1}-Score (%) |
---|---|---|---|---|---|---|---|---|---|

A | A1 | 1.081 | 702 | 623 | 79 | 151 | 80.49 | 88.75 | 84.42 |

A | B1 | 0.994 | 213 | 190 | 23 | 50 | 79.17 | 89.20 | 83.89 |

B | A1 | 0.745 | 702 | 611 | 91 | 177 | 77.54 | 87.04 | 82.01 |

C | A1 | 0.821 | 702 | 653 | 49 | 302 | 68.38 | 93.02 | 78.82 |

D | A1 | 0.654 | 702 | 564 | 138 | 162 | 77.69 | 80.34 | 78.99 |

**Table 3.**Detection results of overlapped lychees under weak or overexposure illumination without using the LBP-SVM classifier.

Method | Illumination State | Average Detection Time (s) | Total Lychee Fruits | TP | FN | FP | Precision (%) | Recall (%) | F_{1}-Score (%) |
---|---|---|---|---|---|---|---|---|---|

A | Weak | 1.226 | 634 | 540 | 94 | 126 | 81.08 | 85.17 | 83.08 |

A | Overexposure | 1.261 | 268 | 227 | 41 | 67 | 77.21 | 84.70 | 80.78 |

A | Weak and overexposure | 1.242 | 902 | 767 | 135 | 193 | 79.90 | 85.03 | 82.38 |

Illumination Conditions | Average Detection Time (s) | Lychee Fruits | TP | FN | FP | Precision (%) | Recall (%) | F_{1}-Score (%) |
---|---|---|---|---|---|---|---|---|

Weak | 1.42 | 634 | 540 | 94 | 80 | 87.10 | 85.17 | 86.12 |

Overexposure | 1.42 | 268 | 227 | 41 | 38 | 85.66 | 84.70 | 85.18 |

Well | 1.38 | 702 | 623 | 79 | 81 | 88.49 | 88.75 | 88.62 |

Comprehensive | 1.41 | 1604 | 1390 | 214 | 199 | 87.48 | 86.66 | 87.07 |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Guo, Q.; Chen, Y.; Tang, Y.; Zhuang, J.; He, Y.; Hou, C.; Chu, X.; Zhong, Z.; Luo, S.
Lychee Fruit Detection Based on Monocular Machine Vision in Orchard Environment. *Sensors* **2019**, *19*, 4091.
https://doi.org/10.3390/s19194091

**AMA Style**

Guo Q, Chen Y, Tang Y, Zhuang J, He Y, Hou C, Chu X, Zhong Z, Luo S.
Lychee Fruit Detection Based on Monocular Machine Vision in Orchard Environment. *Sensors*. 2019; 19(19):4091.
https://doi.org/10.3390/s19194091

**Chicago/Turabian Style**

Guo, Qiwei, Yayong Chen, Yu Tang, Jiajun Zhuang, Yong He, Chaojun Hou, Xuan Chu, Zhenyu Zhong, and Shaoming Luo.
2019. "Lychee Fruit Detection Based on Monocular Machine Vision in Orchard Environment" *Sensors* 19, no. 19: 4091.
https://doi.org/10.3390/s19194091