1. Introduction
In times of climate change, the topic of phenotyping has become more and more important—in order to ensure global food security, high-yielding and climate-resistant crops are required, making a profound understanding of the interaction between environmental factors and a plant’s phenotype mandatory [
1]. The task of phenotyping involves the monitoring of plants’ appearances (e.g., stem height, size and inclination as well as leaf width, inclination, and area) under controlled but changing environmental influences [
2]. Usually, the investigations are performed under greenhouse conditions, but field-based phenotyping is also becoming more and more important (see, for example, References [
3,
4]). The measurement methods traditionally used in crop science are often destructive or invasive [
5], making a temporal monitoring of the plants impossible or at least complicated. Furthermore the measurements are typically performed manually and, hence, are very tedious [
6]. An example of a non-destructive invasive classical method is the manual digitization of characteristic points of a plant, as presented for example, in Reference [
7]. In addition to its labour-intensity, the strong simplification of the resulting plant model causes falsified values of the plant’s descriptive parameters. As the procedure is invasive, detected movements of the plant cannot be attributed solely to natural environmental influences. Because of these drawbacks and due to the demand to investigate several thousands of plants per day in order to guarantee a successful phenotyping [
8], the developments towards a non-destructive (semi-)automatic phenotyping is progressing, imposing requirements on the measurement technique and on the subsequent analysis of the large amount of acquired data. Initial investigations regarding a non-destructive data acquisition almost exclusively involved two-dimensional (2D) imaging techniques (see e.g., Reference [
9] or Reference [
10]). However, due to occlusions within 2D data that can hardly be resolved solely on the basis of colour information [
11], three-dimensional (3D) data is increasingly used for plant phenotyping [
12]. Due to their relatively low costs, cameras are still the most common sensors used in phenotyping [
13,
14]. In order to obtain 3D information of the plants under investigation, multiview images are used to construct 3D point clouds, for example with structure from motion (see e.g., References [
2,
6,
11,
13,
14]). Another possibility is to complete 2D images with depth information of a time-of-flight camera [
15]. Due to the fast and contactless measuring process, laser scanners are also perfectly suitable for phenotyping tasks. Laser scanners directly yield a 3D point cloud and, hence, avoid the subsequent construction of 3D point clouds. Furthermore, they are more robust with regard to changing lighting conditions than cameras [
5] and, therefore, do not require the subsequent use of post-processing algorithms to deal with different lightning conditions. However, compared to digital cameras, the purchase of a laser scanner is usually expensive. Laser scanner-based approaches for plant phenotyping can be found in References [
5,
12,
16,
17].
Although the developments at sensor level allow for a fast and automated data acquisition, the large amount of acquired data is only manageable when the data analysis is also performed automatically. One of the most crucial tasks within the processing chain is the segmentation step during which the unordered 3D point cloud is partitioned into meaningful subsets. With regard to phenotyping, the segmentation is a grouping of points that describe the same plant organ. The segmentation of digital images and laser scanner-based 3D point clouds is generally an extensively studied field of research, but the segmentation of natural objects like plants still presents a challenge—apart from their complex structure, the different forms of appearance make a segmentation difficult [
11]. Although occlusions can be solved to a large extent by means of 3D data, touching or overlapping plant organs are still very difficult to separate [
6]. Nevertheless, a variety of approaches for plant segmentation—either based on images or on laser scanner-based 3D point clouds—can be found in the literature. In contrast to anthropogenic objects, plant organs cannot be described sufficiently well by simple geometric primitives. For this reason, rule-based methods exist which use, for example, Lindenmayer (L)-systems (for detailed information see, for example, Reference [
18]) in order to build complex models of the plants [
11]. These pre-generated models are compared with the point cloud, and structures similar to these models are identified. However, the creation of suitable models requires expert knowledge of a plant’s structure and is only of limited use. Hence, hybrid approaches, which account for the complexity of different plants as well as the variability within the same plant organs, are usually proposed. In Reference [
15], a pre-segmentation is achieved by a hierarchical color segmentation and is afterwards improved by fitting quadratic surfaces through the sparse 3D data provided by a time-of flight camera. In Reference [
2], the point clouds are coarsely segmented by means of a region-growing algorithm. Afterwards, a primitive fitting is used to segment the stem and the petioles, whereas a clustering according to the vertices’ normal vector distribution as well as symmetry considerations are used to segment the leaves. In Reference [
5], also a multi-step procedure is proposed. Based on a graph constructed from the 3D point cloud, the major intrinsic directions are detected by means of a general data clustering (spectral embedding, for more details see Reference [
5]). Afterwards, a clustering groups the points that describe the same plant organ. A similar approach can be found in Reference [
13]—first- and second-order tensors, which capture local point properties, are used to evaluate neighbouring points with regard to their similarity. A distinction between leaf and stem is made by means of local eigenvalues. In Reference [
19], a surface feature histogram-based approach is used in order to classify grapevine stems and leaves as well as wheat ears from other plant organs.
In the case of a temporal monitoring of plants, the segmentation additionally has to be solved over time. This task has to account for a plant’s movement caused by its environmental interaction as well as by its growth, leading to non-rigid deformations of the plant organs [
15]. In the context of the plant’s growth it must also be taken into account that new plant organs develop over time, whereas others decay [
16]. In Reference [
2] a rough alignment of the spatial segmented point clouds and a subsequent matching based on the plants’ internodes is proposed, whereas Reference [
16] uses a forward-backward approach which uses future information to improve the segmentation results from previous point clouds.
In this contribution a fully automated spatio-temporal segmentation procedure for the phenotyping of cucumber plants based on 3D point clouds is proposed. The strategy focuses on the segmentation of individual leaves as they are the main organ of plants, being responsible for the energy and oxygen supply of the plant by means of photosynthesis and, hence, having a major influence on the plant’s growth [
20,
21]. The species under investigation is the cucumber plant, as it is characterized by large leaves with good-natured shape and, hence, is a suitable species for the initial investigations conducted within this contribution. In future, the investigations conducted may be further developed, allowing for the phenotyping of other plant species. A hybrid alternative to the spatial segmentation strategies presented above is introduced, the results of which are temporally linked in a second step, resulting in spatio-temporally segmented time series of point clouds. In contrast to the majority of segmentation strategies proposed in the literature, our strategy does not focus either on the spatial segmentation or on the temporal one, but directly provides a spatio-temporal segmentation of the acquired point clouds. Both steps are realized in a fully automatic way and do not require any user-interaction. One of its major strength is the use of information gained from the temporal segmentation to improve possible erroneous spatial segmentations. Furthermore, the segmentation strategy automatically implements a noise filtering, so that no filtering pre-processing steps are required. The segmented leaves are afterwards automatically modelled by means of best-fitting B-spline surfaces, and the leaf areas are subsequently computed. The use of best-fitting B-spline surfaces has two major advantages over leaf modelling strategies available in the literature—firstly, the use of approximating surfaces leads to a considerably better noise reduction than non-filtering modelling strategies like for example, triangulation strategies or alpha-shapes (see for example Reference [
19]). Secondly, the enhanced B-spline surfaces are a a very flexible tool and, hence, are more appropriate models for complex leaves than for example, quadratic surfaces. Consequently, the chosen modelling strategy leads to more reliable leaf areas than the majority of strategies proposed in the literature. The phenotyping results yielded by the developed strategy are compared to those yielded by standard methods used in crop science. Based on this comparison the applicability of the used multi-sensor system (MSS) to plant phenotyping is evaluated. Finally, the influence of the measurement noise is investigated by applying the developed approach to a point cloud provided by a common terrestrial laser scanner.