Fruit Morphological Measurement Based on Three-Dimensional Reconstruction

: Three-dimensional (3D) shape information is valuable for fruit quality evaluation. Grading of the fruits is one of the important postharvest tasks that the fruit processing agro-industries do. Although the internal quality of the fruit is important, the external quality of the fruit inﬂuences the consumers and the market price signiﬁcantly. To solve the problem of feature size extraction in 3D fruit scanning, this paper proposes an automatic fruit measurement scheme based on a 2.5-dimensional point cloud with a Kinect depth camera. For getting a complete fruit model, not only the surface point cloud is obtained, but also the bottom point cloud is rotated to the same coordinate system, and the whole fruit model is obtained by iterative closest point algorithm. According to the centroid and principal direction of the fruit, the cut plane of the fruit is made in the x -axis, y -axis, and z -axis respectively to obtain the contour line of the fruit. The experiment is divided into two groups, the ﬁrst group is various sizes of pears to get the morphological parameters; the second group is the various colors, shapes, and textures of many fruits to get the morphological parameters. Comparing the predicted value with the actual value shows that the automatic extraction scheme of the size information is effective and the methods are universal and provide a reference for the development of the related application.


Introduction
In recent years, with the development of machine vision technology, the measurement of fruit shape has become more intelligent and standardized. Many researchers have conducted depth research on fruit measurements in the field of machine vision and have achieved fruitful results [1,2].
Machine vision technology mainly obtains the fruit images through specific industrial cameras and then uses advanced technology such as image processing to detect the shape characteristics of the fruit. Rashidi et al. [3] calculated the roundness ratio and volume of cantaloupes and studied the shape classification of cantaloupes. Su et al. [4] developed a novel method of three-dimensional models for estimating potato mass and shape information, and a new image processing algorithm for depth images including length, width, thickness, and volume as mass prediction related factors. Wang et al. [5] applied the RGB-depth sensor to measure the maximum perimeter and volume of sweet onions, estimated the density of onions using measured parameters and the RMSE of onions volume is 18.5 cm 3 . Lu et al. [6] developed phase analysis techniques for reconstruction of the three-dimensional surface of fruit from the pattern images acquired by a structured illumination reflectance imaging system. Fukui et al. [7] based on features extraction from images through a sub-image clustering technique, described images as the number of pixels in various labels in a regression model to estimate of morphological structure and physiological functions into the visualization and digitization stage. Due to the complexity of the internal structure of plants, how to quickly and accurately establish a three-dimensional model of plants has been one of the research focuses for computer graphics and agronomy.
At present, the detection of fruit shape is mainly through the digital camera to obtain the information, but two-dimensional images cannot reflect the structure of the surface. Therefore, a measurement method based on RGB-depth (RGB-D) camera which can collect both color image and depth image simultaneously is proposed. RGB images contain color and appearance information, and depth image contains distance information between RGB-D sensors and objects. The recent success of RGB-D cameras such as the Kinect sensor depicts a broad prospect of three-dimensional data-based computer applications [32]. In recent years, with the development of low-cost sensors such as Microsoft Kinect and Intel RealSense, RGB-D images have been widely used in face database [33][34][35], hand gesture [36][37][38][39][40], head pose [41][42][43], and skeleton information [44][45][46]. Compared with the RGB image, the RGB-D image can provide additional information about the object's three-dimensional geometric structure and provide more effective information for object localization and 3D measurement.

Preprocessing of Fruit Point Cloud
In this experiment, a Kinect v2.0 camera and an electric turntable are used to build a three-dimensional fruit model as shown in Figure 1. During the process of scanning, eight pieces of point clouds (point cloud is collected by rotating every 45 • ) are registered by the iterative closest point (ICP) algorithm to obtain the initial 360 • surface point cloud model. According to the rotation matrix, the subsequent point clouds are rotated around the y-axis in Equation (1). It is specified that the direction parallel to the camera coordinate system is the x-axis, the direction vertical to the camera coordinate system is the y-axis, and the direction parallel between the camera and the object is the z-axis.
where p i orgin is the point cloud from each direction of the surface, p i is the processed point cloud by the rotation matrix, and i ∈ 1, 7.  The basic principle of the ICP [47] algorithm is: Match the target point cloud p (i) from one single direction point cloud P to the source point cloud q (i) from the adjacent point cloud Q, find the nearest neighbor point (p i , q i ) according to certain constraints, and then calculate the optimal matching parameters R and t to minimize the error function in Equation (2), The traditional ICP algorithm can be summarized in two steps: compute correspondences between the two-point cloud; calculate the transformation that minimizes the distance between the corresponding points. For the original point cloud of the model, preprocessing is needed, and the outlier points are removed by the statistic outlier filter [48]. In the process of statistics, the number n of the nearest neighbor point is considered, and the distance d i from the point p i to all neighboring points p is calculated to determine whether it is less than the threshold standard deviation δ of outliers. The point whose average distance is outside the standard range will be removed from the data set. After ICP registration, there are overlapped data in the adjacent patches point cloud, so the normal estimation method based on moving least square (MLS) method is used to resample the point cloud.

Registration the Surface and Bottom Point Cloud of Fruit Model
When the fruit is placed vertically on the turntable, it cannot photograph the bottom point cloud of the fruit in the experiment. The shape of the fruit bottom is different, there are convex ones, such as eggplant and potato, and concave ones, such as tomato and apple. The bottom is a part of the fruit which will affect the accuracy of the result of measuring, so it is necessary to supplement the points from the bottom.
Because the bottom point cloud is collected in the positive direction of the camera, which is different from the coordinate system of the surface point cloud. The collected bottom point cloud p b should be rotated 90 • around the x-axis so that the bottom and the surface point cloud are in the same coordinate system. Equation (3) shows the bottom point cloud in the same coordinate system as the surface point cloud after rotating along the x-axis. Then, move the centroid c b of the bottom point cloud to the centroid c of the surface point cloud. It shows that the bottom point cloud is in the same coordinate system as the surface point cloud after moving the centroid c b to c. Through the rotation matrix and translation vector, the bottom point cloud p b is processed preliminarily. The initial processed point cloud and the surface point cloud are in the same coordinate system. They are all measured with x-axis as the fruit's length, y-axis as the height, and z-axis as the width.
where p b orign is the initial bottom point cloud, p b is the bottom point cloud after preliminary matrix operations, α is 90 • , it means that the angle of bottom point cloud around the x-axis, and T p b is the preliminary translation vector. p b is in the same coordinate system as the surface point cloud, and the direction of the fruit's bottom point cloud is downward. ICP algorithm is still needed to accurately register surface point cloud and bottom point cloud. The rotation matrix R b and translation vector t b are obtained by the ICP method below 30 mm of the surface point cloud registering with the processed bottom point cloud. The obtained transformation matrix is also applicable to the whole surface point cloud. Then the whole point cloud is obtained after the surface point cloud and the bottom point cloud registered by R b and t b . As shown in Equation (4), the whole point cloud of fruit is where p s is the surface point cloud, p b is the bottom point cloud, and p is the whole point cloud needed in this experiment. Through coarse and precise registration of the surface and bottom points of the point cloud, the whole point cloud is finally obtained.
In Figure 2, eight pieces of continuous point clouds are registered together to obtain a complete 360 • frontal model, which contains the surface information of the fruit. At this point, the point cloud of the fruit bottom and the point cloud of the front are respectively obtained, and through preliminary processing, they are in the same world coordinate system, because the fruit surface is the convex body, the surface image and the bottom image taken by the camera can simultaneously acquire the characteristics of the points under the side. Then, through the registration of points collected at the bottom side of the fruit in different coordinate systems, the bottom rotation matrix and the translation vector are obtained, and the bottom model is further processed. Through these steps, a point cloud model with shape-preserving, accurate normal, and small curvature variance is obtained. Finally, the whole fruit model is obtained by the Poisson surface reconstruction method for the preprocessing-smoothed model point cloud p. The fruit's stalk and pulp point cloud is obtained by frontal surface registering, while the fruit's bottom point cloud is obtained by image acquisition and coordinate system transformation. The pulp and the bottom of the fruit are combination called fruit pulp in the next subsequent chapters, which is convenient for the subsequent separation of the fruit stalk.

Segmentation the Fruit Stalk
Based on the 3D model, the stalk cannot be counted as the height of the fruit, so the length of the fruit stalk needs to be removed when calculating the height. In this experiment, the stemmed fruits need further treatment for removing stalks by the locally convex connected patches (LCCP) method [49], which does not depend on the color or texture of point cloud, but on the spatial information and normal vector information. For the segmented model, it is necessary to calculate the convexity-concavity relationship of adjacent patches. The relationship is judged by the extended connectivity criterion and sanity criterion. Extended connectivity criterion uses the angle between the centerline vector x 1 , x 2 and the normal vector n 1 , n 2 of the adjacent two pieces to judge concavity or convexity. If a 1 > a 2 , the relationship of the two patch p i , p j is concave, otherwise is convex as shown in Figure 3. Here, d is the difference between the vector x 1 and x 2 , s is the cross product between the vector n 1 and n 2 as shown in Equation (5), If the relationship between the two pieces is convex, it should be further judged by the sanity criterion. When the angle between d and s is greater than the threshold θ, it can be sure that they are convex relations [49] as in Equation (6), After marking the concavity-convexity relationship of each small region, the region growth algorithm is used to cluster the small region into larger objects. This algorithm restricted by this relationship of small regions can clearly distinguish the boundary between stalk and pulp as shown in Figure 4; the red line represents concavity and the white line represents convexity between adjacent patches. There is an obvious concavity and convexity change at the junction of the fruit's stalk and pulp, which meets the criteria of LCCP segmentation. The LCCP algorithm can correctly separate fruit stalk and pulp, which is useful to calculate the actual height of the fruit without stalk using the bounding box method. The point cloud is automatically segmented by convexity-concavity relationship between pulp and stalk, and the compared models are shown in Figure 5. At present, the point cloud is divided into two parts, namely p pul p and p stalk . The pulp model is retained as the base model for the following measurements.

Measurement Length, Height, Width through Principal Component Analysis Bounding-Box Algorithm
When the three-dimensional model is obtained, the length, height, and width of the fruit are calculated first. The steps are as follows: calculating the central point of fruit, where the world coordinate system is the principal direction of point cloud; using the principal direction and centroid establishing the bounding box of the point cloud. As the direction corresponding to the natural axis of the object, the maximum and minimum positions along the three directions of x, y, and z from the vertex are calculated in Equation (7).
By comparing all points (1, n), the minimum coordinate point x, the maximum x, the minimum y, the maximum y, the minimum z, and the maximum z have been obtained. Then, combine the minimum values as the minimum point (x min , y min , z min ) and the maximum value as the maximum point (x max , y max , z max ). So, the minimum three-dimensional point and the maximum three-dimensional point in the three-dimensional coordinate system of fruit can be obtained. The center point is the midpoint of the two extreme points, representing the centroid of the bounding box, the center point's three-dimensional coordinates (a, b, c) is shown in Equation (8).
According to the matrix coordinate position of the bounding-box extracted the region of fruit. The effective region includes x, y, and z of fruit, which represent the direction of length, height, and width of fruit respectively. Take the minimum point (x min , y min , z min ) as the starting point, and the maximum point (x max , y max , z max ) as the end-point and draw a line from left to right along the main direction x, draw a line from up to down along the main direction y, draw a line from front to back along the main direction z, which represents the length, height, and width of the fruit, respectively. The length of fruit can be obtained by subtracting the left and right limit values of x intersection points, the height can be obtained by subtracting the limit values of y intersection points, and the width can be obtained by subtracting the limit values of z intersection points. The length l, height h, and width w of the bounding box can be obtained by the differences of the three directions (x, y, z) of the maximum three-dimensional point and the minimum three-dimensional point in Equation (9).
Based on the bounding-box method, the three-dimensional fruit plane is displayed in Figure 6. It is specified that the x-axis corresponds to the length l and height h of fruit, the y-axis corresponds to the width w and height h of fruit, and the z-axis corresponds to the width w and length l of fruit. The display of the z-direction surface is divided into top and bottom surfaces to observe the top and bottom details.

Measurement Perimeters of Fruit
Further, analyzing the contour of the model, and taking the centroid of the model as the research object to calculate the perimeter of each section is another task for fruit. The distribution of three-dimensional continuous image f (x, y, z) on any plane P : ax + by + cz + d = 0 is called the section of f along plane P. The outline of multi-layer slices can be represented in three-dimensional coordinates. In this experiment, the segmentation point is obtained according to the symmetry of the fruit. The tangent planes are made parallel to xoy, yoz, and xoz, respectively according to the intersection ( p x , p y , p z ), where section A, B, and C are parallel to plane yoz, plane xoz, and plane xoy, respectively, the points made up of these planes are shown in Equation (10), Based on the three sections parallel to x, y, z-axis through the centroid, the intersection point of the fruit model is the set of the required point cloud, which constitutes the contour of the fruit, i.e., the perimeter. The perimeter is of great significance to identify the shape of the fruit. As shown in Figure 7, the contour obtained by plane A tangent to the model is shown in red, the contour obtained by plane B tangent to the model is shown in blue, and the contour obtained by plane C tangent to the model is shown in green. Each plane centered on the intersection is divided into four quadrants, and the intersection of these lines in the two-dimensional plane represents the outline of the fruit. These points are discrete in two-dimensional space, so the spline interpolation method is used to connect them together. As the surface of the fruit tends to be a sphere or ellipsoid, the obtained points are improved by the cubic Bezier curve spline [50] in Equation (11), where P i is the boundary points of the contour, t is the scale value, the smooth curve is drawn according to the coordinates of four adjacent points. Here, we use the method in [51], which not only retains the characteristics of the cubic Bezier curve but also passes through all the points in the experiment, that is, the control point P 1 , P 2 can be expressed by unit vector and tension parameter. For the continuous curve L on the section, its density distribution function is B(t). Because the function B(t) is a piecewise curve, the perimeter is obtained by the curve integral in Equation (12) on each small curve [a, b].
The surface profile is obtained by the intersection of the tangent plane and the parallel section as shown in Figure 8, which are described as the value points of the contour line.

Experiment
Two groups of fruits were tested in this paper. In the first group, the predicted value is compared with the measured value after modeling the same type of fruits (pears in this experiment) in different sizes; in the second group, the predicted value is compared with the measured value after modeling many types of fruits with different shapes, colors, textures, and stalks. When practically measured, for the length, width, and height of the fruit, the actual value measured by the ruler is compared with the predicted value; for the perimeter of the fruit, the average value obtained by the soft ruler within 3 mm of the centroid is compared with the predicted value.

Measurement of the Morphological Parameters of the Pear
Forty pears are used as experimental objects to verify the validity of the experiment by modeling, measuring and comparing with the actual value. The measured results are compared with predicted values of length, height, width, and perimeter to get the minimum error, maximum error, coefficient of determination (R 2 ), and root mean square error (RMSE). The results are shown in Figure 9. In the coordinate graph, the x-axis represents the predicted value, and the y-axis represents the measured value of pears. The other methods [5,18,52] are compared with the algorithm proposed in this paper, and the results are shown in Table 1. The value obtained by averaging the length and width is taken as vertical diameters. The results show that average of vertical pear diameter RMSE is 1.17 mm and the R 2 is 92.5%; the horizontal pear diameter RMSE is 1.03 mm and the R 2 is 97.7%; the perimeters of the pears are calculated, the smallest error is from the pears' z-axis perimeters, RMSE is 3.72 mm; the largest error is the pears' x-axis perimeters, RMSE is 3.85 mm.

Measurements of the Morphological Parameters of Many Kinds of Fruit
A variety of vegetables and fruits are used as experimental objects to establish a three-dimensional model and calculate the morphological parameters. For onion, pear, kiwifruit, and other fruits with stalks, the LCCP algorithm is used to remove its stalk and calculate the height of the pulp. For fruits without a stalk, like tomato, orange, eggplant, the model calculates the height directly. The measured values and predicted values are shown in Figure 10, the yellow bar represents the predicted value, the green bar represents the measured value. Among various fruit models, the error between the measured value and the predicted value of the green pepper model is larger than other fruits. The shape of the green pepper model is irregular, some details are removed during the point cloud registration process, so the model error of the green pepper is larger than that of other fruits. The details of the model still need to be further studied, which is also the focus of the next work. To show the model more clearly, when collecting the point cloud, the Kinect camera coordinate system is used to calibrate the color and depth images, so the collected point cloud has the attributes of color and coordinate position. The fruit model is displayed from the front, bottom, and side, respectively, in the experiment. In Figure A1, from left to right is the front image of fruit, the registering model of the fruit point cloud, the front of 3D model, the bottom image of fruit, the bottom of the 3D model, and the overall appearance of the 3D model. Through the observation of the model by the bounding-box method, it can be seen that the method provided in this experiment can correctly fit the 3D model of fruit. It can not only measure the morphological parameters, such as length, height, width, perimeter but also reflect the color and texture characteristics.

Conclusions
The three-dimensional fruit model is obtained by point cloud modeling from different coordinate systems. Eight surface point clouds are registered by the ICP algorithm, the bottom point cloud coordinate system and the surface point cloud coordinate system are unified to get the whole point cloud of the fruit. The fruit model is classified according to its stalk, the height of the fruit without stalk is calculated directly, while the height of fruit with stalk is calculated after further processing. The parameters of length, height, width, and centroid are obtained by the bounding-box method. Through the tangent planes to the fruit model of the centroid and main direction x, y, and z, the contours of the fruit is obtained.
The innovation of this experiment is to obtain a complete model, not just a front surface model of the fruit. Secondly, for the measurement of fruit with stalk, the concavity-convexity relationship between adjacent patches is used as the segmentation judgment standard to remove the fruit stalk to improve the accuracy of height measurement. Finally, the effectiveness of the algorithm is verified by comparing the measured values of the length of the bounding box method and the perimeter of the fruit with the actual values.
In this paper, the length, height, width and three-dimensional perimeters of the fruit are measured and compared with the actual values. In the first group of the experiment, pears with stalks are taken as the research subjects, the concavity and convexity of the LCCP method are used to remove the stalk, then the phenotype parameters of the model are calculated. In the second group, a variety of fruits are taken as the research objects, and the phenotype parameters of fruits with different shapes, color, and texture are measured by our method.
Through the experimental results, it can be seen that the experimental method in this paper is universal and can complete the phenotype parameters measurement of many kinds of fruits. Funding: This project is supported by the national key science and technology infrastructure project, "National Research Facility for Phenotypic and Genotypic Analysis of Model Animals", grant number "4444-10099609".

Conflicts of Interest:
The authors also thank the editor and anonymous reviewers for providing helpful suggestions for improving the quality of this manuscript.

Abbreviations
The following abbreviations are used in this manuscript: