In-Field Estimation of Fruit Quality and Quantity

Every new tool gives humankind a new capability or capabilities, as a new tool finds a range of applications [...]


Contribution of the Special Edition
The Special Issue "In-Field Estimation of Fruit Quality and Quantity" presents a collection of eight articles. The topic is reviewed by Anderson et al. [1] with attention to the use of temperature monitoring and non-invasive NIRS assessment of fruit attributes in the forecasting of harvest timing, the use of in-field machine vision for a direct assessment of the fruit load, and UAV or satellite multispectral imaging, or extrapolation of historical yield data, for an indirect assessment of fruit load. Given that the different technologies measure harvest timing and load in different ways, the various estimates are complimentary.
An example of the use of NIRS technology is provided by Goke et al. [2], in a study involving the assessment of pear fruit dry matter content in the context of the pruning regime. NIRS is a relatively mature technology, established in commercial pack-line use for over two decades and instrumentation advances allowing its use in the orchard environment of variable light and temperature in recent years (Walsh et al. [3]). Attention is therefore moving from the development of the technology, per se, to its application uses.
In contrast, the introduction of deep learning (convolutional neural networks) to imagedetection tasks is relatively recent, and the publication focus remains on the development of the method. For example, several papers in the Special Issue explored the use of machine vision in the estimation of either flowering or fruit load. Koirala et al. [4] reported on the machine-vision-based detection of mango flower panicles at three developmental stages and demonstrated a time course of the quantification of stages, using orchard images collected weekly. This is a step towards automation of the detection of flowering 'events' in an orchard, with heat units tallied from this date in forecast of fruit harvest maturation. Koirala et al. [4] also reported on the use of augmented image sets, although it was noted that the addition of rotated bounding boxes did not improve the training.
The primary limitation in use of in-field machine vision to estimate the fruit load is that the camera does not 'see' all fruit on the trees, especially for dense canopied trees. Anderson et al. [5] and Villacres and Cheein [6] both report issues with fruit occlusion in their respective documentation of the performance of machine-vision-based estimation of the fruit yield of whole cherry and mango orchards (Figure 1), respectively. Anderson et al. [5] documented the improvement in yield estimates made using a muti-view approach, involving multiple images per tree and tracking of fruit between images, over a 'dual-view' approach. The impact of the canopy architecture on yield estimates was also documented.

Contribution of the Special Edition
The Special Issue "In-Field Estimation of Fruit Quality and Quantity" presents a collection of eight articles. The topic is reviewed by Anderson et al. [1] with attention to the use of temperature monitoring and non-invasive NIRS assessment of fruit attributes in the forecasting of harvest timing, the use of in-field machine vision for a direct assessment of the fruit load, and UAV or satellite multispectral imaging, or extrapolation of historical yield data, for an indirect assessment of fruit load. Given that the different technologies measure harvest timing and load in different ways, the various estimates are complimentary.
An example of the use of NIRS technology is provided by Goke et al. [2], in a study involving the assessment of pear fruit dry matter content in the context of the pruning regime. NIRS is a relatively mature technology, established in commercial pack-line use for over two decades and instrumentation advances allowing its use in the orchard environment of variable light and temperature in recent years (Walsh et al. [3]). Attention is therefore moving from the development of the technology, per se, to its application uses.
In contrast, the introduction of deep learning (convolutional neural networks) to image-detection tasks is relatively recent, and the publication focus remains on the development of the method. For example, several papers in the Special Issue explored the use of machine vision in the estimation of either flowering or fruit load. Koirala et al. [4] reported on the machine-vision-based detection of mango flower panicles at three developmental stages and demonstrated a time course of the quantification of stages, using orchard images collected weekly. This is a step towards automation of the detection of flowering 'events' in an orchard, with heat units tallied from this date in forecast of fruit harvest maturation. Koirala et al. [4] also reported on the use of augmented image sets, although it was noted that the addition of rotated bounding boxes did not improve the training.
The primary limitation in use of in-field machine vision to estimate the fruit load is that the camera does not 'see' all fruit on the trees, especially for dense canopied trees. Anderson et al. [5] and Villacres and Cheein [6] both report issues with fruit occlusion in their respective documentation of the performance of machine-vision-based estimation of the fruit yield of whole cherry and mango orchards (Figure 1), respectively. Anderson et al. [5] documented the improvement in yield estimates made using a muti-view approach, involving multiple images per tree and tracking of fruit between images, over a 'dualview' approach. The impact of the canopy architecture on yield estimates was also documented. Figure 1. The technique of in-field machine is now being applied to assessment of fruit and flower load of whole farms. Colours refer to fruit density, from green (low) through yellow to red (high).
Koirala et al. [7] explored the use of techniques to correct for occluded fruit, for example, by the use of a machine-vision-based measure of the fraction of partially occluded to non-occluded fruit. None of the proposed techniques were recommended for adoption, and this topic remains open. Koirala et al. [7] explored the use of techniques to correct for occluded fruit, for example, by the use of a machine-vision-based measure of the fraction of partially occluded to non-occluded fruit. None of the proposed techniques were recommended for adoption, and this topic remains open.
However, the fruit number is only part of the harvest load story, with the fruit size also being critical. The estimation of size requires information on distances, with several relevant candidate technologies being available. Neupane et al. [8] reported on an evaluation of depth cameras in the context of this application, while Mendez et al. [9] report on the use of LiDAR for fruit sizing, fruit counting, and canopy modelling. The application of LiDAR for fruit counting was also compromised by fruit occlusion in dense canopies, and acquisition time was also an issue, given the need for a high-density point cloud.

Conclusions
This collection thus presents a snapshot of the currently available techniques for the in-field estimation of fruit quality (maturity) and quantity. As for the NIRS technology, over the next five years, the machine vision and LiDAR techniques will mature, with the focus shifting from the assessment of the technology to the use of the technology to aid agronomic decision-making. As noted in the review by [1], "this is an exciting period to be involved in the development and application of tools for the forecast of tree fruit load and harvest timing".