# Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks

^{1}

^{2}

^{3}

^{4}

^{*}

## Abstract

**:**

^{2}) between the estimated and the actual weighted yield, mean forecast error (MFE), mean absolute percentage error (MAPE), and root mean square error (RMSE) were 0.81, −0.05, 10.7%, 2.34 kg/tree, respectively. For the model of the ripening period, these measures were 0.83, −0.03, 8.9%, 2.3 kg/tree, respectively. In 2011, the two previously developed models were used to predict apple yield. The RMSE and R

^{2}values between the estimated and harvested apple yield were 2.6 kg/tree and 0.62 for the early period (small, green fruit) and improved near harvest (red, large fruit) to 2.5 kg/tree and 0.75 for a tree with ca. 18 kg yield per tree. For further method verification, the cv. “Pinova” apple trees were used as another variety in 2012 to develop the BPNN prediction model for the early period after June drop. The model was used in 2013, which gave similar results as those found with cv. “Gala”; (4) Conclusion: Overall, the results showed in this research that the proposed estimation models performed accurately using canopy and fruit features using image analysis algorithms.

## 1. Introduction

^{2}of 0.80 between apples detected by the fruit counting algorithm and those manually counted and R

^{2}of 0.57 between apples detected by the fruit counting algorithm and actual harvested yield.

_{N}, area of fruits F

_{A}, area of fruit clusters F

_{CA}, and foliage leaf area (L

_{A})) in the canopy image into account, besides the apple fruit number. The aims of the present paper are: (1) to describe the processes of extracting canopy features and “learning” the relationship between the features and actual yield per tree by use of a back propagation neural network (BPNN) using the data from 2009–2010; (2) to evaluate the BPNN prediction models to analyse the relation between the estimated and harvested apple yield; and (3) to represent the accuracy of the BPNN models by predicting the yield for 30 samples from 2011.

## 2. Materials and Methods

#### 2.1. Site Description and Image Acquisition

^{−2}·s

^{−1}using an EGM-5 (PPSystems, Amesbury, MA, USA). Images were obtained on these apple trees, in the early afternoon (3 to 5 pm) on days with indirect light to exclude stray or blinding light, and deep shades at a time of low solar angle on the second date (period 2).

#### 2.2. Apple Fruit and Leaf Feature Description

_{N}) and the fruit area (F

_{A}) are the first two essential features for yield prediction. The third feature is the area of the apple clusters (F

_{CA}) in the image, because apple clusters are a conspicuous characteristic of canopy structure, which can be comprised of more than two apples. Compared with the pixel proportion of the bright red calibration sphere (Figure 1a), which was of the size range for an apple fruit in period 1, if the fruit domain exceeded the size of the calibration sphere by 3-fold, it was assumed to be an apple cluster. Since the leaves can impact apple yield estimation by occluding fruit, foliage area (L

_{A}) is the fourth one.

_{A}, F

_{N}, L

_{A}, and F

_{CA}extracted from canopy images as essential parameters for yield prediction [5], we converted them to the ratios F

_{1}, F

_{2}, F

_{3}and F

_{4}(Table 2). These ratios were subsequently employed for modelling and the different steps in the modelling process are visualized in a flowchart (Figure 2).

#### 2.3. Fruit Identification and Feature Extraction (Step 1)

#### 2.4. Leaf Identification and Feature Extraction (Step 2)

_{A}) was computed by summing pixels that belonged to foliage.

#### 2.5. Development of BPNN Yield Prediction Model (Step 3)

_{1}, F

_{2}, F

_{3}, F

_{4}and F

_{5}(Table 3) were computed based on the parameters I

_{A}, F

_{A}, F

_{N}, F

_{CA}, L

_{A}and Y

_{A}(Table 2). One data set of 150 images acquired in period 1 was collected as set 1, and the other set of 150 images acquired in period 2 was collected as set 2. Sixty images were sampled in the summer of 2009 and ninety images were sampled in the summer of 2010 for the two sets, i.e., each set included 150 samples, respectively. Each sample consisted of five parameters. Two BPNN prediction models were built for the two periods using Sets 1 and 2, respectively.

_{A}was adjusted to select the best one based on the minimization of the mean squared errors (MSE), which is a statistical measure showing how well the model predicts the output value and the target value (yield).

- -
- N
_{I}is the number of input neurons, - -
- N
_{O}is the number of output neurons, - -
- N
_{H}is the number of hidden neurons, - -
- N
_{A}is the number of the neurons, which can be added in hidden neurons based on MSE.

#### 2.6. The Measures for Model Evaluation

^{2}, MFE, RMSE, and MAPE. Mean Forecast Error (MFE) is a measure of unbiasedness of the predictions, defined as Equation (2), and MFE is closer to 0, then the model becomes less unbiased. Root mean squared error (RMSE) is an often used measure of the difference between values predicted by a model and those actually observed from the object being modeled, and is defined as Equation (3). It can rule out the possibility that large errors of opposite signs could cancel out in a MFE measure. The Mean Absolute Percentage Error (MAPE) is computed through a term-by-term comparison of the relative error in the prediction with respect to the actual value of the variable, and is defined as Equation (4). Thus, the MAPE is an unbiased statistical approach for measuring the predictive capability of a mode [14,15].

## 3. Results

#### 3.1. Data Analysis

_{1}, F

_{2}, and F

_{3}were much smaller compared with the foliage-related ratio F

_{4}; in period 2, the values of F

_{1}, F

_{2}, and F

_{3}increased with increasing fruit size. Therefore, the fruit detection is strongly dependent on the amount of foliage in the canopy, which makes it almost impossible to detect apple fruit using image analysis in period 1. In period 2, the obvious colour and size changes of apple fruits make the detection easier and the influence of the foliage becomes weak. Overall, F1 appears to be the most important parameter, because it reflects the area of all apples in the tree image. Hence, it includes the overall information on the size and number of all apples.

#### 3.2. BPNN Model Structure and Validation

^{2}(0.02), RMSE (0.15 kg/tree), MFE (0.02), and MAPE (0.45 %), and shows that the results of Model 1 (Table 5) were similar to those of Model 2 (Table 6) with Model 1 being slightly inaccurate.

#### 3.3. Yield Prediction for Subsequent Year

#### 3.4. Yield Prediction for Other Apple Varieties

## 4. Discussion

^{2}values in the calibration data set between apple yields estimated by image processing and actual harvested yield were 0.57 for young cv. “Gala” fruit after June drop, which improved to R

^{2}= 0.70 in the fruit ripening period. By comparison, the presented combined approach (Table 5) improved the coefficient of determination (R

^{2}) for young, small, and light-green cv. “Gala” fruit to 0.81 and for ripening fruit to 0.83. This is also an advancement of the results of Rozman et al. [6] with a correlation (r) between the forecast and actual yield of r = 0.83 for “Golden Delicious” and 0.78 for “Braeburn”, with standard deviations (SD) of 2.83 and 2.55 kg. In our study, R² was 0.81 and SD was 2.28 kg for “Gala” (Table 5).

## 5. Conclusions

_{N}, single fruit size F

_{A}, area of fruit clusters F

_{CA}, and foliage leaf area (L

_{A})) to develop two back propagation neural network (BPNN) models for early yield prediction, i.e., for young, small, green fruitlets and mature red fruits. Apple was used as a model fruit or crop and the algorithms were developed for image acquisition under natural light conditions in the orchard. The results showed that BPNN can be used for apple yield prediction and that those four selected canopy features are suitable for early yield prediction and present an elegant way for predicting fruit yield using machine vision and machine learning for apple and possibly other fruit crops.

## 6. Outlook

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

ANN | Artificial neural network |

BPNN | back propagation neural network |

## References

- Wulfsohn, D.; Zamora, F.A.; Téllez, C.P.; Lagos, I.Z.; Marta, G.F. Multilevel systematic sampling to estimate total fruit number for yield forecasts. Precis. Agric.
**2012**, 13, 256–275. [Google Scholar] [CrossRef] - Stajnko, D.; Rakun, J.; Blanke, M. Modelling apple fruit yield using image analysis for fruit colour, shape and texture. Europ. J. Hortic. Sci.
**2009**, 74, 260–267. [Google Scholar] - Wachs, J.P.; Sturm, H.J.; Burks, F.; Akhanais, V. Low and high-level visual feature-based apple detection from multi-modal images. Precis. Agric.
**2010**, 11, 717–735. [Google Scholar] [CrossRef] - Blanke, M.M. Prediction of apple yields in Europe—Present and new approaches in research. In Proceedings of the 106th Annual Meeting of the Washington State Horticultural Association (WSHA), Yakima, WA, USA, 8–10 December 2011; Smith, L., Ed.; WSHA Publishing: Yakima, WA, USA, 2011; pp. 68–75. [Google Scholar]
- Zhou, R.; Damerow, L.; Sun, Y.; Blanke, M. Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield. Precis. Agric.
**2012**, 13, 568–580. [Google Scholar] [CrossRef] - Rozman, C.; Cvelbar, U.; Tojnko, S.; Stajnko, D.; Karmen, P.; Pavlovie, M.; Vracko, M. Application of Neural Networks and Image Visualization for Early Predicted of Apple Yield. Erwerbs-Obstbau
**2012**, 54, 69–76. [Google Scholar] - Stajnko, D.; Rozmana, Č.; Pavloviča, M.; Beber, M.; Zadravec, P. Modeling of ‘Gala’ apple fruits diameter for improving the accuracy of early yield prediction. Sci. Hortic.
**2013**, 160, 306–312. [Google Scholar] [CrossRef] - Kelman, E.; Linker, R. Vision-based localisation of mature apples in trees images using convexity. Biosyst. Eng.
**2014**, 118, 174–185. [Google Scholar] [CrossRef] - Ye, X.; Sakai, K.; Garciano, L.O.; Asada, S.I.; Sasao, A. Estimation of citrus yield from airborne hyperspectral images using a neural network model. Ecol. Model.
**2006**, 198, 426–432. [Google Scholar] [CrossRef] - Linker, R.; Cohen, O.; Naor, A. Determination of the number of green apples in RGB images recorded in orchards. Comput. Electron. Agric.
**2012**, 81, 45–57. [Google Scholar] [CrossRef] - O'Neal, M.R.; Engel, B.A.; Ess, D.R.; Frankenberger, J.R. Neural network prediction of maize yield using alternative data coding algorithms. Biosyst. Eng.
**2002**, 83, 31–45. [Google Scholar] [CrossRef] - Sezgin, M.; Sankur, B. Survey over image thresholding techniques and quantitative performance evaluation. J. Electron. Imaging
**2004**, 13, 146–165. [Google Scholar] - Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning Representations by Back-Propagating Errors. Nature
**1986**, 323, 533–536. [Google Scholar] [CrossRef] - Esmaeili, A.; Tarazkar, M.H. Prediction of shrimp growth using an artificial neural network and regression models. Aquac. Int.
**2010**, 19, 705–713. [Google Scholar] [CrossRef] - Greene, W.H. Econometric Analysis, 4th ed.; Prentice-Hall: Englewood Cliffs, NJ, USA, 2000; p. 1007. [Google Scholar]
- Annamalai, P.; Lee, W.S.; Burks, T. Color Vision Systems for Estimating Citrus Yield in Real-Time. In Proceedings of the ASAE/CSAE Annual International Meeting, Ottawa, ON, Canada, 1–4 August 2014.
- Lee, W.S.; Chinchuluun, R.; Ehsani, R. Citrus fruit identification using machine vision for a canopy shake and catch harvester. Acta Hortic.
**2009**, 824, 217–222. [Google Scholar] - Zaman, Q.U.; Schumann, A.W.; Percival, D.C.; Gordon, R.J. Estimation of wild blueberry fruit yield using digital color photography. Trans. ASABE
**2008**, 51, 1539–1544. [Google Scholar] [CrossRef] - Cheng, H.; Damerow, L.; Sun, Y.; Blanke, M.M. Detection of apple fruit in an orchard for early yield prediction as dependent on crop load. Acta Hortic.
**2016**, 1137, 59–66. [Google Scholar] [CrossRef] - Aggelopoulou, A.D.; Bochtis, D.; Fountas, S.; Swain, K.C.; Gemtos, T.A.; Nanos, G.D. Yield prediction in apple orchards based on image processing. Precis. Agric.
**2011**, 12, 448–456. [Google Scholar] [CrossRef]

**Figure 1.**Sample apple tree at different times; left picture (

**a**) was acquired in the early period after the June drop (period 1), about 3 months before harvest, right picture (

**b**) was acquired during the ripening period (period 2), about 15 days before harvest.

**Figure 5.**Example of an image of an apple tree with colour-coded mapping of colour differences between G (green) and B (blue) for each pixel, showing the leaves as bright colour dots and the background in deep blue.

**Figure 6.**Yield prediction for 2011 based on (

**a**) “Prediction Model 1” for young apple fruit in July and (

**b**) “Prediction Model 2” for ripe apple fruit in September for the subsequent year (n = 30 trees).

**Figure 7.**Yield prediction for 2013 based on the prediction “Pinova” Model for young apple fruit in July for the subsequent year (n = 34 trees).

**Table 1.**Characteristics of samples. Tree fruit load % refers to the % of trees carrying a high (>mean + SD), low (<mean − SD), and moderate fruit load, respectively.

Season | Numbers of Trees | Number of Images (Period 1, Period 2) | Yield/Tree (mean ± SD) | Tree Fruit Load (%) High, Mod, Low |
---|---|---|---|---|

2009 | 60 | 60; 60 | 20.62 ± 4.90 | 20; 68; 12 |

2010 | 90 | 90; 90 | 16.68 ± 5.43 | 18; 63; 19 |

2009 & 2010 | 150 | 150; 150 | 18.26 ± 5.55 | 15; 67; 18 |

2011 | 30 | 30; 30 | 18.63 ± 4.17 | 10; 70; 20 |

Parameter | Description | Parameter | Description |
---|---|---|---|

I_{A} | Sum of pixels of the whole images | Y_{E} | The estimated yield of apple tree |

F_{A} | Sum of pixels belonging to apple fruits | F_{1} | F_{A}/I_{A} |

F_{N} | Number of fruit | F_{2} | F_{N}/200 |

F_{CA} | Sum of pixels belonging to apple clusters | F_{3} | (F_{A}– F_{CA})/I_{A} |

L_{A} | Sum of pixels belonging to foliage | F_{4} | L_{A}/I_{A} |

Y_{A} | The actual yield of apple tree | F_{5} | Y_{A}/50 |

MAPE | Mean Absolute Percentage Error | SD | Standard deviation of the error |

MFE | Mean Forecast Error | RMSE | Root Mean Square Error |

Sets | Set 1 | Set 2 | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|

Tree | F_{1} | F_{2} | F_{3} | F_{4} | F_{5} | F_{1} | F_{2} | F_{3} | F_{4} | F_{5} | |

1 | 0.0159 | 0.3100 | 0.0159 | 0.5495 | 0.4571 | 0.0552 | 0.5100 | 0.0331 | 0.2278 | 0.4571 | |

2 | 0.0041 | 0.1800 | 0.0041 | 0.4265 | 0.3355 | 0.0279 | 0.3850 | 0.0242 | 0.2100 | 0.3355 | |

3 | 0.0103 | 0.3100 | 0.0103 | 0.4718 | 0.3808 | 0.0348 | 0.4900 | 0.0199 | 0.1900 | 0.3808 | |

4 | 0.0088 | 0.2550 | 0.0088 | 0.7245 | 0.3679 | 0.0347 | 0.6100 | 0.0199 | 0.2052 | 0.3679 | |

5 | 0.0214 | 0.5250 | 0.0193 | 0.7022 | 0.4423 | 0.0590 | 0.5500 | 0.0297 | 0.2066 | 0.4423 | |

6 | 0.0081 | 0.3850 | 0.0081 | 0.5714 | 0.4509 | 0.0318 | 0.5350 | 0.0251 | 0.2372 | 0.4509 | |

7 | 0.0108 | 0.4000 | 0.0060 | 0.5024 | 0.3611 | 0.0320 | 0.4700 | 0.0159 | 0.1893 | 0.3611 | |

8 | 0.0125 | 0.3750 | 0.0125 | 0.7344 | 0.3648 | 0.0440 | 0.5400 | 0.0329 | 0.2084 | 0.3648 | |

9 | 0.0060 | 0.3100 | 0.0060 | 0.4066 | 0.5106 | 0.0549 | 0.5000 | 0.0327 | 0.3360 | 0.5106 | |

10 | 0.0149 | 0.3900 | 0.0149 | 0.7559 | 0.3863 | 0.0537 | 0.5100 | 0.0329 | 0.2101 | 0.3863 | |

11 | 0.0191 | 0.4550 | 0.0170 | 0.7116 | 0.4431 | 0.0596 | 0.4350 | 0.0340 | 0.1902 | 0.4431 | |

12 | 0.0096 | 0.3350 | 0.0096 | 0.4149 | 0.3944 | 0.0445 | 0.5750 | 0.0269 | 0.1753 | 0.3944 | |

13 | 0.0144 | 0.4150 | 0.0144 | 0.7099 | 0.4131 | 0.0546 | 0.3900 | 0.0229 | 0.2593 | 0.4131 | |

14 | 0.0180 | 0.3150 | 0.0148 | 0.6459 | 0.3526 | 0.0585 | 0.3850 | 0.0173 | 0.1663 | 0.3526 | |

15 | 0.0041 | 0.1800 | 0.0041 | 0.4265 | 0.3355 | 0.0279 | 0.3850 | 0.0242 | 0.2100 | 0.3355 |

Parameter | Value | Parameter | Value |
---|---|---|---|

Input | F_{1}, F_{2}, F_{3}, F_{4} | Hidden layer transfer function | Logarithmic sigmoid transfer function |

Target | F_{5} | Output layer transfer function | Linear transfer function |

Output | Forecast value | Learning function | Gradient descent learning function |

Performance function | MSE | Training function | Levenberg-Marquardt back-propagation |

Parameter | Structure | Samples (Trees) of 2009 and 2010 | RMSE in kg/Tree | MAPE (%) | MFE | R^{2} | |
---|---|---|---|---|---|---|---|

Model | |||||||

Model 1 | Train set | 4-12-1 | 135 | 2.34 | 10.67 | −0.05 | 0.81 |

Test set | 15 | 2.53 | 12.40 | 0.16 | 0.80 | ||

Model 2 | Train set | 4-11-1 | 135 | 2.27 | 8.9 | −0.03 | 0.83 |

Test set | 15 | 2.31 | 10.36 | −0.06 | 0.82 |

Model | Actual Yield (A) in kg per 150 Trees | Predicted Yield (P) in kg per 150 Trees | Difference (|A – P|) in kg | Mean Difference in kg per Tree |
---|---|---|---|---|

Model 1 | 2736 | 2744 | 8 | 0.05 |

Model 2 | 2736 | 2740 | 4 | 0.03 |

**Table 7.**The model structure and the evaluation of model performance which was developed based on samples of 2012 for cv. “Pinova”.

Parameter | Structure | Samples (Trees) of 2012 | RMSE in kg/Tree | MAPE (%) | MFE | R^{2} | |
---|---|---|---|---|---|---|---|

Model | |||||||

“Pinova” Model | Train set | 4-10-1 | 80 | 2.24 | 11.45 | −0.14 | 0.89 |

Test set | 10 | 2.53 | 14.19 | 0.06 | 0.88 |

Model | Actual Yield (A) in kg per 100 Trees | Predicted Yield (P) in kg per 100 Trees | Difference (|A–P|) in kg | Mean Difference in kg per Tree |
---|---|---|---|---|

“Pinova” Model | 1817 | 1822 | 5 | 0.06 |

© 2017 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Cheng, H.; Damerow, L.; Sun, Y.; Blanke, M. Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks. *J. Imaging* **2017**, *3*, 6.
https://doi.org/10.3390/jimaging3010006

**AMA Style**

Cheng H, Damerow L, Sun Y, Blanke M. Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks. *Journal of Imaging*. 2017; 3(1):6.
https://doi.org/10.3390/jimaging3010006

**Chicago/Turabian Style**

Cheng, Hong, Lutz Damerow, Yurui Sun, and Michael Blanke. 2017. "Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks" *Journal of Imaging* 3, no. 1: 6.
https://doi.org/10.3390/jimaging3010006