1. Introduction
Automation is necessary in the agricultural industry to help accelerate the rate of increased crop productivity through genetic improvement techniques, in order to help cope with the rapid increase in human population and future demands on worldwide food security. Phenotyping of new and old varieties under varying environmental conditions to assess their suitability presents a challenge [
1,
2]. There is a need for developing novel, field-deployable systems with semi- or fully-automatic processing of plant phenotypes for a suite of vegetative traits that can aid in our understanding of the relationships between genetic information and food productivity. The critical elements of such systems are the sensors that help to automate phenotyping and contribute knowledge to the final understanding of this complex relationship. Most of the sensors used in agriculture have limited resolution or dimensionality and are not able to acquire the full scope of available information about plants, such as their structure and leaf texture. This leads to a limitation in distinguishing different types of deficiencies [
3]. Advanced sensors, like cameras, that can characterize spatial and color information from natural objects will play a crucial role in the future development of agricultural automation [
4,
5].
Recent methods for sensor-based 3D reconstruction have been developed for a wide range of applications. A method for 3D shape scanning with a time-of-flight (ToF) camera has been described in [
6]. The ToF camera technique can measure depth information in real-time, and when the method [
6] is used to align depth scans in combination with a super-resolution approach, some mitigation of the sensor’s typically high signal noise level and systematic bias can be achieved. In [
7], a “visual structure-from-motion system” for 3D reconstruction is implemented via feature extraction, image matching and dense reconstruction algorithms. This structure-from-motion implementation works successfully on rigid, enclosed (
i.e., non-porous) objects, with the use of a single camera capturing images from multiple viewpoints. A stereo vision technique applied to 3D reconstruction is introduced in [
8]. This work provides a framework to create visually realistic 3D reconstructions of solid objects, including the steps of stereo image acquisition, feature detection, feature matching, camera calibration matrix calculation, point cloud generation and surface reconstruction. A stereo vision-based 3D reconstruction system for underwater scenes is proposed in [
9]. This system yields promising depth map results in exploring underwater environments. There is a great deal of work that utilizes consumer-grade range camera technology (e.g., the Microsoft Kinect) [
10] for scene reconstruction. The Kinect, which works in a similar way to a stereo camera, was originally designed for indoor video games. Due to its robustness and popularity, it is being studied for use in many research and industrial applications. In [
11], the Kinect device was utilized to reconstruct dense indoor scenes.
A number of stereo vision-based methods have been developed for 3D modeling of plants and leaves. The research in [
12] presents a combination of binocular stereo vision and structure-from-motion techniques [
7] for reconstruction of small plants from multiple views. The 3D reconstruction results for the plant canopy include height, width and volume of the plant, as well as leaf cover area. In [
13], color and ToF cameras were used to model leaves through a combination of hierarchical color segmentation and quadratic surface fitting using ToF depth information. A moving robot arm holding a camera was additionally employed to inspect the quality of leaf segmentation. The study [
14] shows how the stereo and ToF images can be combined for non-destructive automatic leaf area measurements. In [
15], the Microsoft Kinect was utilized in combination with the Point Cloud Library [
16] to provide measures of plant height and base diameter in “tower mode”. In [
17], Kinect-based visualization of potted greenhouse tomato plants in indoor environments is presented with a method of automatic plant stem detection. The work in [
18] is mostly similar to [
15] with the addition of depth information-based leaf segmentation. A novel 3D mesh-based technique was presented in [
19] for temporal high-throughput plant phenomics. Based on the plant meshes reconstructed using commercial software for 3D scanning [
20], this study provided mesh segmentation, phenotypic parameter estimation and plant organ tracking over time to yield promising measurement accuracies of stem height and leaf size. In [
21], the Kinect sensor was assessed to determine its best viewing angles to estimate the plant biomass based on poplar seedling geometry. The purpose of [
22] is to analyze the accuracy of a structure-from-motion combined with the multiview stereo method for tomato plant phenotyping at the organ level, based on the reference data provided by a close-up laser scanner. The extracted 3D features herein include leaf area, main stem height and convex hull of the complete plant. In [
23], the first steps of utilizing 3D light field cameras were introduced for phenotyping large tomato plants in a greenhouse environment. Promising results of 3D reconstructed greenhouse scenes were shown in this work with several aspects related to camera calibration, lens aperture, flash illumination and the limitation of the field of view. There are recent studies that use 3D laser scanning for plant phenotyping. In [
24], a surface feature histogram-based approach is introduced to adapt to laser scans of plants for the purpose of plant organ classification. Local geometric point features are used to describe the characteristics of plant organ classes. Classification results of grapevine and wheat organs are shown with very high reliability in this study. High throughput phenotyping of barley organs based on 3D laser scanning is presented in [
25]. By combining the advantages of a surface feature histogram-based approach with a parametric modeling of plant organs, this work shows automatic parameter tracking of the leaves and stem of a barley plant over time.
This paper describes a novel 3D reconstruction system for plants that incorporates a number of unique hardware-based stereo features: multiple pairs of high-resolution color digital cameras, visible structured lights, ease of configuration adjustment and the ability to work indoors and outdoors. There are three principal contributions of this research. First, a custom mechanical structure for multi-view 3D reconstruction of plants was designed that consists of an arc to hold ten high-resolution digital color cameras, a plant-stationary mount for two visible structured lights and a target turn-table for a 360-degree view of the plant. Second, a custom design for structured lights was created that projects random-dot patterns onto the target to enhance the uniqueness of the visual texture on the object surface, so that significantly better stereo matching results can be obtained. Computer control of illumination synchronization and intensity allows the system to adapt to indoor and outdoor scenes. Third, a complete hardware and software solution (including camera calibration, structured light control, stereo matching, the proposed 3D point cloud generation and registration, point cloud noise removal and segmentation and the proposed 3D leaf detection and 3D plant feature measurement) is created for both 3D reconstruction and non-destructive phenotyping measurement (plant height, number of leaves, leaf height and width and internode distances) of plants.
2. System Design
The 3D image data were based on digital color images of individual plants taken by ten electronically-controlled, high-resolution, digital single-lens reflex cameras (Model EOS Rebel T3, Canon Inc., Tokyo, Japan). The ten cameras, each equipped with a zoom lens (Model EF-S 18–55 mm 1:3.5–5.6 IS II, Canon Inc., Tokyo, Japan), were organized into five stereo camera pairs aimed at the target object and held fixed relative to one another on an arc at 25, 40, 55, 70 and 85 degrees relative to the ground plane.
Figure 1 shows the mechanical structure of our 3D reconstruction system, where
Figure 1b shows the arc and cameras. One camera in each stereo pair was mounted “upside down” relative to the other in order to place the lens centerlines on exactly parallel optical axes with their pairwise baseline set to 89 mm. A Universal Serial Bus (USB) hub was used to connect all cameras to a computer and to allow complete control of the cameras by an open-source software application
digiCamControl [
26]. All ten color images, with a resolution 1920 × 1280 pixels, were captured and transferred to the computer at once via the USB. Because a difference in image resolution might dramatically affect the processing speed of the system from image transfer, segmentation and stereo matching to point cloud processing, we selected the resolution of 1920 × 1280 as the best trade-off between speed and quality, so that highly-accurate results can be obtained in an acceptable processing time. Camera parameters, including focal length, aperture, shutter speed, ISO and white balance, were manually set to achieve the best quality images for camera calibration and stereo matching.
Figure 1.
Mechanical structure (a) of the 3D reconstruction system: (b) the arc holding ten Canon EOS Rebel T3 cameras; (c) a pair of cameras where the second camera is upside down relative to the first one; (d) structured light devices and their power adapters with a relay controlled by a digital I/O control NI USB-6008; (e) a turn-table that rotates the plant 360 degrees; and (f) the target plant.
Figure 1.
Mechanical structure (a) of the 3D reconstruction system: (b) the arc holding ten Canon EOS Rebel T3 cameras; (c) a pair of cameras where the second camera is upside down relative to the first one; (d) structured light devices and their power adapters with a relay controlled by a digital I/O control NI USB-6008; (e) a turn-table that rotates the plant 360 degrees; and (f) the target plant.
A structured illumination system was designed to provide plant-stationary active visual texture enhancement of plant foliage from all camera viewpoints. The system utilized two telephoto 105-mm focal length lenses (Model NIKKOR-P f/2.5, Nikon Co., Tokyo, Japan), each equipped with a high-power, 29 mm-diameter LED array (Model BXRC-40E10K0-L-03, Bridgelux Inc., Livermore, CA, USA, white color, maximum 10,000 luminous flux and correlated color temperature of 4000 K), designed to project two grayscale, random-dot patterns printed in high resolution on clear film onto the scene. The brightness and strobe synchronization of the LED arrays could be controlled by a dimmable LED power supply (Model HLG-120H-42B, MEAN WELL Enterprises Co., Guangzhou, China) configured with a solid state relay connected to the computer via USB through a digital I/O device (Model USB 6008, National Instruments Co., Austin, TX, USA).
Figure 1d shows the structure light device and its power adapter.
For enhanced visual texture generation, a random-dot pattern was created using a 3000 × 3000 pixel grayscale image with a printed resolution of 1200 dots-per-inch, where a 30% Hurl noise was applied to a transparent background. This projected pattern is hence able to support the segmentation and matching algorithms at a three pixel-wide resolution of approximately 1 mm on the leaf surface of a plant.
Figure 2 shows the random-dot pattern projected onto white and black surfaces (a piece of paper and a cloth curtain) and plants. The dot pattern, printed on a 38 mm-diameter transparency film (
Figure 2a), was placed between two transparent glass windows for support and inserted into the focal plane of the structured light cylinder (
Figure 1d). The pattern projection was done with an object distance of approximately 1.5 m to provide good depth of focus.
Figure 2b,c shows that even when the object is completely white or black, the structured light can still create added visual texture. In
Figure 2d–f, dot textures are created effectively on the leaf surface of the cabbage and cucumber plants and tomato leaves. The two structured light devices were mounted on two arms at a 90-degree top view and a 45-degree side view, respectively. Both of these arms were mounted in a fixed position relative to the plant (
Figure 1d,e).
A turn-table was utilized to account for the target plant not being simple, i.e., its leaves are curved or overlapping, causing portions of the plant to be occluded when using a single view angle of the arc relative to the plant. The current choice of optics in the illumination system supports object sizes up to 30 cm × 30 cm × 50 cm.
4. Experimental Results
A computer (CPU Model Core i7 at 3.4 GHz, Intel Co., Santa Clara, CA, USA, with 12-GB DDR3 random-access-memory) was used for all processing steps, except that a 1152-core GPU (Model GeForce GTX 760, NVidia Co., Santa Clara, CA, USA) graphics card was utilized for implementing the GPU-based stereo matching algorithms. Experiments were executed on eight cabbage plants, eight cucumber plants and three tomato compound leaves. These plant species were selected as examples of plants with: long leaves, leaves spreading vertically, very small leaves, curved leaves, long branches, overlapped leaves, leaves having natural texture (
Figure 10j,k,p) and compound leaves with leaflets attached to a rachis. The ground truth plant heights were measured manually based on the distance from the plant’s base to the top point of the highest leaf. In the same manner, the ground truth internode distances were manually determined based on the distance between two leaf nodes along a plant stem, as illustrated in
Figure 9. The ground truth leaf sizes were measured by the steps of cutting off the leaves, scanning their images and then creating a ruler-based mapping from a pixel unit to a real-world unit (mm) for the leaves. Leaf detection accuracy and errors in estimating the plant height, leaf size and internode distance were quantified.
Figure 10 shows the reconstructed 3D models of the 19 plants.
Table 1 presents a brief description of the plants consisting of plant height and number of leaves. In this experiment, at least five views (at one turn-table angle) were needed to successfully reconstruct a whole plant. The cucumber (
Figure 10i) and the three tomato compound leaves (
Figure 10q–s) required 15 views (five views at a turn-table angle × three turn-table angles with a difference of 45 degrees) because of the high degree of self-overlap and curve-shaped leaves in these species.
Figure 10.
3D model results of eight cabbage plants (a–h), eight cucumber plants (i–p) and three compound leaves from tomato plants (q–s).
Figure 10.
3D model results of eight cabbage plants (a–h), eight cucumber plants (i–p) and three compound leaves from tomato plants (q–s).
Table 1.
List of the 19 plants used for the experiments.
Table 1.
List of the 19 plants used for the experiments.
Plant | Figure 10 | Height | No. of Leaves | Brief Description |
---|
Cabbage 1 | (a) | 114 | 4 | Good leaf shape |
Cabbage 2 | (b) | 150 | 4 | 1 vertically-long leaf |
Cabbage 3 | (c) | 140 | 4 | 1 small and 2 curved leaves |
Cabbage 4 | (d) | 114 | 4 | Long branches |
Cabbage 5 | (e) | 130 | 4 | 2 overlapped leaves |
Cabbage 6 | (f) | 139 | 3 | Long and thin branches |
Cabbage 7 | (g) | 105 | 3 | 1 leaf attaches to plant stem |
Cabbage 8 | (h) | 229 | 2 | 1 curved leaf |
Cucumber 1 | (i) | 242 | 3 | Tall, big leaves |
Cucumber 2 | (j) | 117 | 4 | 2 brown-textured-surface leaves |
Cucumber 3 | (k) | 131 | 3 | 2 brown-textured-surface leaves |
Cucumber 4 | (l) | 115 | 2 | 1 small leaf |
Cucumber 5 | (m) | 113 | 1 | Good leaf shape |
Cucumber 6 | (n) | 123 | 2 | 1 small leaf |
Cucumber 7 | (o) | 132 | 2 | 1 leaf attaches to plant stem |
Cucumber 8 | (p) | 116 | 2 | 1 yellow-textured-surface leaf |
Tomato 1 | (q) | 192 * | 6 | Long and curved leaves |
Tomato 2 | (r) | 253 * | 8 | Long and curved leaves |
Tomato 3 | (s) | 269 * | 8 | Long and curved leaves |
In this system, the parameters of the mechanical structure were fixed, while parameters for the software algorithms were adapted to optimize the performance in phenotyping different species.
Table 2 shows the parameters used for the GPU-based stereo matching and point cloud registration to 3D feature extraction steps. This system required parameter tuning so that high quality results can be obtained. With the selected parameters used in the segmentation step, each plant was completely separated from the background, which helps to improve the stereo matching results significantly. The actual distance of the target plant to the cameras was chosen from 1.2 m–1.7 m, so that all data points outside that range (
i.e., background noise) were discarded when reprojecting disparity values onto 3D space. In the feature extraction step, individual parameters, such as clustering tolerance, minimum cluster size, leaf size threshold, leaf direction threshold and ratio for leaf location, might be adjusted for each point cloud in order to correctly detect all leaves.
Table 2.
Algorithm parameters used for the experiments.
Table 2.
Algorithm parameters used for the experiments.
GPU-Based Stereo Matching | | Point Cloud Registration | | 3D Feature Extraction * |
---|
Plant segmentation | Spatial win-size | 11 | | Registration | Max distance | 25 | | Clustering | Tolerance | 0.03 |
Color win-size | 7 | | Max iteration | 10 | | Min cluster size | 4000 |
Min segment size | 10 | | Outlier rejection | 25 | | Max cluster size | 10 |
Threshold | 10 | | Poisson surface reconstruction | Octree depth | 12 | | Leaf detection | Size threshold | 0.005 |
Stereo block matching | No. of disparities | 256 | | Solver divide | 7 | | Direction threshold | 0.7 |
Win-size | 17 | | Samples/node | 1 | | Ratio: leaf location | 0.25 |
Texture threshold | 60 | | Surface offset | 1 | | * Parameters vary depending on leaf shape |
Bilateral filter | Filter size | 41 | | Face removal w.r.t. edge length | 0.05 | |
No. of iterations | 10 | | Noise removal w.r.t. No. of faces | 25 |
Table 3 presents average measurement accuracies in plant phenotype estimation using the features extracted from 3D reconstruction when using the structured illumination system.
Figure 11 and
Figure 12 show leaf detection accuracies (in terms of precision and recall) and plant height errors accordingly. In
Figure 11, the precision and recall were computed based on the number of correct detections (true positive), the number of incorrect detections (false positive) and the number of missed detections (false negative) to show the robustness of the detection. The leaf/leaflet detection accuracies were 93.75%, 100% and 100% for cabbage, cucumber and tomato, respectively. The main source of error in estimating the phenotyping measurements in cabbage was because the cabbage plants had smaller leaves than the cucumber and tomato. Additionally, cabbage leaves had greater curvature and vertical spread. Positive and negative errors were presented in
Figure 12 to explain that the actual plant height was mostly larger than the estimated height. The average plant height error for all plants is 11.18 mm. The percentage of error in
Table 3 was computed as a normalized value, so that we could directly compare the results between different types of plants. The average error in estimating leaf and internode features, as a percentage of plant height across all three species was, 4.87%, 3.76% and 7.28% for leaf length, width and internode distance, respectively. The main reason for the higher internode distance error was that this distance was estimated based on the plant’s principal axis (as aforementioned in
Figure 9), which was approximated by the most dominant eigenvector of the plant. The approximation of the principal axis by the dominant eigenvector was somewhat inaccurate when leaves or branches of the plant spread unexpectedly in different directions.
Table 3.
Average accuracy in plant phenotype estimation from 3D reconstruction.
Table 3.
Average accuracy in plant phenotype estimation from 3D reconstruction.
Plant Features | Cabbage | Cucumber | Tomato | Average |
---|
Leaf height | Error (mm) | 6.86 | 5.08 | 10.16 | 6.6 |
% error * | 5.58% | 4.36% | 4.36% | 4.87% |
Leaf width | Error (mm) | 5.08 | 4.83 | 5.33 | 5.08 |
% error * | 4.16% | 3.9% | 2.31% | 3.76% |
Internode distance | Error (mm) | 9.65 | 7.87 | 21.34 | 10.92 |
% error * | 7.67% | 6.3% | 8.49% | 7.28% |
Figure 11.
Evaluation of the number of detected leaves/leaflets in terms of precision and recall, from the 3D reconstructed cabbage, cucumber and tomato plants.
Figure 11.
Evaluation of the number of detected leaves/leaflets in terms of precision and recall, from the 3D reconstructed cabbage, cucumber and tomato plants.
Without structured light, many of the leaves of the plants in this study could not be matched correctly using the block matching algorithm, as mentioned in
Figure 5a,b, illustrating the benefit of enhancing the visual texture of plants using structured illumination. In some cases, the leaves had sufficient natural texture to allow successful stereo matching, as in Cucumber 2, 3 and 7. However, sometimes, the natural texture was actually a defect, like a scar or insect damage on the leaf, that created the texture and was not a feature of the plant, but of the environment. The system generally had to work with leaves where the plants were healthy and had less leaf texture. A comparison of the algorithm performance between plant images without and with using structured light to enhance the leaf texture is shown in
Figure 13. The disparity result obtained when using structured light was slightly better than that obtained without using structured light in the leaf areas with less natural texture.
Figure 12.
Errors of plant height (calculated by differentiating the ground truth from the estimated one) of the cabbage (a) and cucumber plants (b). Notice that plant height was not determined for tomato compound leaves, because these leaves were imaged individually and are parts of a plant.
Figure 12.
Errors of plant height (calculated by differentiating the ground truth from the estimated one) of the cabbage (a) and cucumber plants (b). Notice that plant height was not determined for tomato compound leaves, because these leaves were imaged individually and are parts of a plant.
Figure 13.
Comparison between without and with using structured light (SL) on textured leaves. The disparity result of using SL is slightly better than that without using SL in the regions of less natural textures (marked by red rectangles and ellipses). Note that the colorized disparity images are presented here, instead of grayscale ones, for better illustration of the differences.
Figure 13.
Comparison between without and with using structured light (SL) on textured leaves. The disparity result of using SL is slightly better than that without using SL in the regions of less natural textures (marked by red rectangles and ellipses). Note that the colorized disparity images are presented here, instead of grayscale ones, for better illustration of the differences.
A comparison of the system performance for plants having different leaf sizes, leaf shapes and number of leaves was conducted.
Figure 14 shows the results for plants having big leaves
versus plants having small leaves, plants having curved leaves
versus flat leaves, plants having many leaves
versus plants having fewer leaves and plants having long-shaped leaves
versus plants having round-shaped leaves. When comparing leaf size, we found that larger leaves are easier to detect than the small leaves. The main reason for the superior detection of larger leaves is that, at the image resolution of 1920 × 1280 pixels, leaves having a size of 40 mm or less at a distance of more than 1.2 m were tiny objects in the image, and they might give more errors in the stereo matching (with a predefined matching window size for all kinds of leaves in a plant) and point cloud registration (clouds having more points were easier for registration and reconstruction) steps. The internode distance error for the plants having small leaves was significantly higher than for plants having big leaves. Estimation of internode distances was affected by calculation of a plant’s principal axis. Big leaves spreading unexpectedly in different directions might make the plant’s principal axis be appear at a different position than the plant’s stem; therefore, it yielded higher errors in the estimation of internode distances. Leaf size errors were approximately the same in the cases of “curved-leaf
vs. flat-leaf”, “many-leaf
vs. fewer-leaf” and “long-shaped
vs. round-shaped leaf”; indicating that leaf shape and the number of leaves did not actually affect the estimation. Note that in the system, a plant was imaged from many different views in order to have the full shape of the leaves in 3D; thus, it was presumed that acceptable 3D models of leaves were used for the feature extraction step. However, internode distance errors were notably different for various leaf sizes, leaf shapes and numbers of leaves. Curved and long leaves yielded high errors in the internode distance estimation. Plants having fewer leaves gave highly accurate internode evaluation. Vertical error bars in
Figure 14 confirm the significant differences between the internode distance errors in terms of leaf size, leaf shape and the number of leaves.
In the present practice, plant phenotype features are manually measured by destroying the plant, with high human time consumption. As reported in [
39], it normally took more than 2 h for two people to destructively measure the total leaf area of a single row of some large pepper plants. The study [
19] stated that manual phenotypic analysis required approximately 30 min per plant. It took an average of 20 min per plant for manual measurement of our experimental plants. Our system, which is non-destructive, fully utilized C++ and CUDA C languages to create a 3D model of a plant from five stereo image pairs. It required approximately 4 min of processing time in total, which includes: less than 4 s (1.5% of the total time) for transferring the images from the ten cameras via the USB to the computer (camera shutter speed is 250 ms), approximately 5 s (2%) for plant segmentation, 0.3 s for stereo matching, less than 4 s (1.5%) for disparity bilateral filtering, 0.08 s for reprojection of disparity values to the 3D cloud, 180 s (75%) for cloud registration (variable depending on the number of data points and complexity of the transformation between the source and target point clouds), approximately 40 s (17%) for Poisson surface reconstruction and less than a second for leaf detection and 3D feature extraction. By utilizing a GPU, plant segmentation was approximately 20-times faster than the CPU-based implementation [
30]. The processing time of the block matching and disparity bilateral filtering was approximately improved by 30- and 10-times [
36], respectively. We plan to implement the steps of point cloud registration and processing on a GPU for future deployment of the system, so that a total time of approximately 1 min for a complete 3D plant model is expected.
Figure 14.
Comparison of the percentage of error in plant height estimation for plants having different leaf sizes, leaf shapes and numbers of leaves. Errors for leaf length, leaf width and internode distance were considered in order to understand which types of plant leaves yield higher errors. From left to right, top to bottom: plants having big leaves versus plants having small leaves, plants having curved versus flat leaves, plants having many versus fewer leaves and plants having long-shaped versus round-shaped leaves.
Figure 14.
Comparison of the percentage of error in plant height estimation for plants having different leaf sizes, leaf shapes and numbers of leaves. Errors for leaf length, leaf width and internode distance were considered in order to understand which types of plant leaves yield higher errors. From left to right, top to bottom: plants having big leaves versus plants having small leaves, plants having curved versus flat leaves, plants having many versus fewer leaves and plants having long-shaped versus round-shaped leaves.
It was not possible to directly compare the existing systems to ours where different plant features, cameras and software configurations were used.
Table 4 summarizes and compares various camera-based 3D reconstruction systems for plants in terms of their system configuration, features, methods, accuracy and processing speed. Comparing and analyzing such systems enabled us to highlight their advantages and disadvantages. Additionally, it allowed us to relate the performance of our system to the existing ones via the percentage of absolute errors and total processing time. In terms of leaf detection, our system yielded a substantially better accuracy (97%) than that of [
18] (68%). Our plant height error, 8.1%, was smaller than the 9.34% of [
19]. The leaf width and height errors obtained from our system, 3.76% and 4.78%, greatly outperformed the 5.75% and 8.78% errors of [
19]. Our full 3D reconstruction system required approximately 4 min for the whole process from image capturing to plant feature extraction in comparison to 20 min of [
19] and 48 min of [
22], respectively, where commercial 3D modeling software were utilized. It took a minute for full 3D leaf segmentation and modeling in [
13]. In a single view, the systems in [
14,
39] required more than an hour and 5 min respectively, to process a large plant.
Table 4.
Comparison of various camera-based 3D reconstruction systems for plants.
Table 4.
Comparison of various camera-based 3D reconstruction systems for plants.
Study | Camera System | Camera View | Measures | Environment | Techniques | Accuracy | Processing Time |
---|
Alenya, 2011 [13] | ToF and color cameras; robot arm | Multiview for leaf modeling | Leaf size | Indoor | Depth-aided color segmentation, quadratic surface fitting, leaf localization | Square fitting error: 2 cm2 | 1 min for 3D leaf segmentation |
Chene, 2012 [18] | Kinect camera | Top view | Leaf azimuth | Indoor | Maximally stable extremal regions-based leaf segmentation | Detection accuracy 68%; azimuth error 5% | n/a |
Heijden, 2012 [39] | ToF and color cameras | Single view | Leaf size and angle | Greenhouse (for large pepper plants) | Edge-based leaf detection, locally weighted scatterplot smoothing-based surface reconstruction | Leaf height correlation 0.93; leaf area correlation 0.83 | 3 min for image recording, hours for the whole process |
Paproki, 2012 [19] | High-resolution SLR camera, with 3D modeling software [20] | Multiview for full 3D reconstruction | Plant height, leaf width and length | Indoor | Constrained region growing, tubular shape fitting-based stem segmentation, planar-symmetry and normal clustering-based leaf segmentation, pair-wise matching-based temporal analysis | Plant height error 9.34%; leaf width error 5.75%; leaf length error 8.78% | 15 min for 3D reconstruction [20], 4.9 min for 3D mesh processing and plant feature analysis |
Azzari, 2013 [15] | Kinect camera | Top view | Plant height, base diameter | Outdoor | Canopy structure extraction | Correlation: 0.97 | n/a |
Ni, 2014 [12] | 2 low-resolution stereo cameras with 3D modeling software [7] | Multiview for full 3D reconstruction | Plant height and volume, leaf area | Indoor | Utilizing VisualSFM software [7], utilizing [40] to manually extract plant features | n/a | n/a |
Song, 2014 [14] | Two stereo cameras; ToF camera | Single view | Leaf area (foreground leaves only) | Greenhouse (for large plants) | Dense stereo with localized search, edge-based leaf detection, locally weighted scatterplot smoothing-based surface reconstruction | Error: 9.3% | 5 min for the whole process |
Polder, 2014 [23] | 3D light-field camera | Single view | Leaf and fruit detection | Greenhouse (for large
tomato plants) | Utilizing 3D light-field camera to output a pixel to pixel registered color image and depth map | n/a | n/a |
Rose, 2015 [22] | High-resolution SLR camera, with 3D modeling software [41] | Multiview for full 3D reconstruction | Plant height, leaf area, convex hull | Indoor | Utilizing Pix4Dmapper software [41] to have 3D models, plant feature extraction | Correlation: 0.96 | 3 min for data acquisition, 20 min for point cloud generation, 5 min for manual scaling, 10 min for error removal |
Andujar, 2015 [21] | 4 Kinect cameras with 3D modeling software [42] | Multiview for semi-full 3D reconstruction | Plant height, leaf area, biomass | Outdoor | Utilizing Skanect software [42] to have 3D models, utilizing [40] to manually extract plant features | Correlation: plant height 0.99, leaf area 0.92, biomass 0.88 | n/a |
Our system | 10 high-resolution SLR cameras organized into 5 stereo pairs; 2 structured lights | Multiview for full 3D reconstruction | Plant height, leaf width and length, internode distance | Indoor | Texture creation using structured lights, mean shift-based plant segmentation, stereo block matching, disparity bilateral filtering, ICP-based point cloud registration, Poisson surface reconstruction, plant feature extraction | Leaf detection accuracy 97%; plant height error 8.1%, leaf width error 3.76%, leaf length error 4.87%, internode distance error 7.28% | 4 min for the whole process |