Multispectral Mapping on 3D Models and Multi-Temporal Monitoring for Individual Characterization of Olive Trees

Abstract

3D plant structure observation and characterization to get a comprehensive knowledge about the plant status still poses a challenge in Precision Agriculture (PA). The complex branching and self-hidden geometry in the plant canopy are some of the existing problems for the 3D reconstruction of vegetation. In this paper, we propose a novel application for the fusion of multispectral images and high-resolution point clouds of an olive orchard. Our methodology is based on a multi-temporal approach to study the evolution of olive trees. This process is fully automated and no human intervention is required to characterize the point cloud with the reflectance captured by multiple multispectral images. The main objective of this work is twofold: (1) the multispectral image mapping on a high-resolution point cloud and (2) the multi-temporal analysis of morphological and spectral traits in two flight campaigns. Initially, the study area is modeled by taking multiple overlapping RGB images with a high-resolution camera from an unmanned aerial vehicle (UAV). In addition, a UAV-based multispectral sensor is used to capture the reflectance for some narrow-bands (green, near-infrared, red, and red-edge). Then, the RGB point cloud with a high detailed geometry of olive trees is enriched by mapping the reflectance maps, which are generated for every multispectral image. Therefore, each 3D point is related to its corresponding pixel of the multispectral image, in which it is visible. As a result, the 3D models of olive trees are characterized by the observed reflectance in the plant canopy. These reflectance values are also combined to calculate several vegetation indices (NDVI, RVI, GRVI, and NDRE). According to the spectral and spatial relationships in the olive plantation, segmentation of individual olive trees is performed. On the one hand, plant morphology is studied by a voxel-based decomposition of its 3D structure to estimate the height and volume. On the other hand, the plant health is studied by the detection of meaningful spectral traits of olive trees. Moreover, the proposed methodology also allows the processing of multi-temporal data to study the variability of the studied features. Consequently, some relevant changes are detected and the development of each olive tree is analyzed by a visual-based and statistical approach. The interactive visualization and analysis of the enriched 3D plant structure with different spectral layers is an innovative method to inspect the plant health and ensure adequate plantation sustainability.

1. Introduction

Olive plants are among the most ancient cultivated fruit trees. For many centuries, the production of olive trees has had an important impact on the Spanish economy. Specifically, Jaén, a southern region of Spain, is considered one of the most relevant producers of virgin olive oil around the world [1]. This province contains 550 thousand hectares of olive groves and produces around 50% (600 thousand tons per year) of the total national olive oil production, and more than 20% of the world’s total production of olive oil. Therefore, advances to increase the final production and farming sustainability of olive trees have a high impact on the local economy.

Remote sensing techniques are effective solutions for data acquisition with different spatio-temporal resolutions [2]. Early approaches proposed methods to analyze the evolution of olive trees using satellite images and Geographic Information Systems (GIS) [3,4,5]. According to the emergence of novel sensors for plant phenotyping, some studies were presented to assess plant sustainability using heterogeneous data (spectral indices and temperature) [6,7]. More recently, Zarco et al. [8] used a hyperspectral sensor (visible and near-infrared (VNIR) model; Headwall Photonics) and a thermal camera (FLIR SC655; FLIR System) to study the Xylella fastidiosa propagation in olive trees through the analysis of some physiological traits over orthomosaic maps. In general, previous works include interesting methods for monitoring and feature extraction of olive trees. However, these are based on bi-dimensional information using a multilayered plant-trait scheme. Therefore, many values from the image-based observations have to be interpolated in the same pixel of the orthomosaic map. Thus, the overall measurements of the plant traits are computed by a coarse estimation. The novelty of our approach lies in focusing on automatically mapping meaningful plant traits on a 3D model of an olive orchard. Our study object is the point cloud of every olive tree, which is enriched by the leaf reflectance response to plant stress. Undoubtedly, the contribution of drone technology in this field is highly positive to get a higher spatial resolution of 3D models and detailed observations of plant reflectance by a more efficient approach [9,10].

Among the available aerial remote sensing platforms, Unmanned Aerial Vehicles (UAVs) are considered to be cost-effective and they can capture high-resolution imagery from multiple viewpoints [11,12]. In contrast to satellite images, UAV-based cameras provide a higher spatial resolution and a more detailed observation of individual plants [13]. These solutions are an increasingly used trend for several monitoring tasks using various types of sensors [14]. As a result, the use of drone provides a high versatility during the acquisition process, by planning custom flights at different heights or angles to capture images. Regarding some UAV-based applications in Precision Agriculture (PA), Vanegas et al. [15] used hyperspatial and hyperspectral data for improving the plant pest surveillance in vineyards. Other approaches focused on using drones either for the classification of tree species [16] or the acquisition of thermographic and multispectral features to inspect the plant status [17,18]. In this paper, the input images are collected by two UAV-based systems (a high-resolution RGB camera and a multispectral sensor), which provide accurate data of plant shape and reflectance response. Moreover, a multi-temporal approach is considered to monitor the variation of observed plant features.

According to the study of plant morphology, the 3D reconstruction of the branching structure can be modeled by photogrammetric techniques as well as Light Detection and Ranging (LiDAR) sensors. Nevertheless, the modeling of 3D plant structures is still challenging because olive trees contain many overlapping branches in the crown and a high leaf density, which make the generation of plant geometry quite complicated. Some studies proposed different methods for the assessment of geometric features such as the height, area or volume of fruit trees [19,20,21] and for the detection of potential phytosanitary problems at the canopy level [22,23]. Undoubtedly, the complete plant geometry provides a real perception of many morphological traits, which cannot be directly identified in the image. Zarco-Tejada et al. [24] measured the height of olive trees through the photogrammetric process based on UAV-based imagery using a consumer-grade camera onboard a low-cost unmanned platform. The generated 3D scenes had a sufficient resolution to quantify a single-tree height with a similar accuracy to more complex and costly LiDAR systems. However, the resulting point cloud mainly represents the ground and higher branches of olive trees with a significant lack of lateral and internal structures. Our method provides a more accurate reconstruction of plant structure with a higher density of 3D points for each olive tree. Consequently, the trunk, main branches, and many leaves are modeled on a dense point cloud. The efficient managing of these heavy geometric models to merge with spectral layers is also discussed in this work.

Research in ecology studies the evolution of plants through a detailed plant inventory by monitoring at different canopy scales and image-based remote sensing [25]. Unlike a direct visual disease detection on-site, UAV-based sensors are commonly used to measure plant reflectance in several narrow-bands [26]. Leaf reflectance can be observed at different wavelengths, in which the reflected light depends on some biochemical components of the leaf internal structure (chlorophylls and carotenoids) [27]. Multispectral sensors capture several significant bands to detect many properties for diagnosing the plant physiological status such as the drought and heat stress, nutrient content, and plant biomass. Several approaches provide advances for the vegetation assessment using different spectral traits of plants [17,28]. Thus, vegetation monitoring is possible through feature extraction to promote sustainable farming [29]. Other approaches used multispectral data to propose a 2D-based analysis for disease detection [30] and segmentation of vegetation areas [31]. In addition, recent contributions are provided by the fusion of LiDAR and hyperspectral remotely sensed data [32]. Regarding the novelty of our methodology to previous works, we propose the generation of a reflectance map for each multispectral image to project all pixel values on the point cloud. The plant reflectance is observed from multiple viewpoints to detect the light interactions with top, lateral and lower branches of the tree.

Recently, several studies have presented novel methods by fusing heterogeneous image datasets to extract key feature patterns of the monitored plantation. Nevalainen et al. [33] used hyperspectral imaging for individual tree detection with UAV-based photogrammetric point clouds. Degerickx et al. [34] provided an urban tree health assessment using airborne hyperspectral and LiDAR imagery. In addition, other studies focus on multi-temporal plant diagnosing from very high-resolution satellite images [35] and individual crop measurements based on the clustering of terrestrial LiDAR data [36]. Specifically, UAV-based approaches use drone sensors for an accurate reconstruction of photogrammetric point clouds [37] and even forest inventory [38]. The resulting 3D models of the plant structure can be used for feature extraction of tree height and volume, which provides information about the morphology of olive trees. By applying the proposed methodology, many input feature layers can be mapped on the 3D model of plants without any human intervention. A fully automated method is proposed to enrich the point cloud with spectral data relating to plant health. Our results can be inspected in a 3D virtual environment for visual-based assessments by an expert.

The individual tree detection plays an increasingly significant role in an automated plant-monitoring process. Mohan et al. [39] used UAV-based imagery for individual tree detection using a local-maxima based algorithm on Canopy Height Models (CHMs). Individual Tree Crown Detection and Delineation (ITCD) algorithms have advanced through novel approaches by the integration of heterogeneous data sources [40,41]. Marques et al. [42] proposed a fully automated process to monitor chestnuts plantations. This method is based on RGB and multispectral imagery for tree identification and counting as well as feature plant extraction. However, it is based on an image segmentation instead of a 3D classification. Previous works mainly describe an image-based segmentation by using orthomosaics. In our research, the multispectral and geometric data are used to identify every olive tree as a single entity on the point cloud. In this way, a multi-temporal inventory is developed to study the evolution of morphological and spectral features of olive trees.

Regarding the focus of our study, it is based on the fusion of multispectral imagery and photogrammetric 3D point clouds and multi-temporal analysis of individual olive trees. UAV image sets have been acquired using a high-resolution camera and a multispectral sensor. The mapping of image-based spectral information on the 3D model of the olive plantation is the core of this paper. The capability of multi-temporal analysis of plant development by multispectral monitoring is also approached. The presented methodology provides a novel framework for a fully automated fusion of UAV-based heterogeneous data over high detailed 3D tree models. This paper is organized as follows: Section 2 describes the materials and methods used in this research; Section 3 shows the results of the proposed methodology; Section 4 analyzes the results and the novelty of our approach; Section 5 presents the main conclusions.

2. Materials and Methods

The study area characterization, the description of the used UAV and sensors and the methods applied to acquire, process, and analyze the data models are presented in this section. The methodology is based on four main stages: multispectral image mapping on the point cloud, individual tree segmentation, feature extraction, and multi-temporal analysis.

2.1. Study Area and Data Acquisition

Our method takes input data from the monitoring of an olive plantation, which is located in Jaén. The study area covers 2 hectares of olive trees in which the proposed methodology has been tested and optimized. This is also characterized by some buildings, human objects, a road and other types of vegetation. Figure 1 presents a general overview of the study area. In this paper, the geometry and leaf reflectance of olive trees are the target features, which are acquired and analyzed for individual tree.

The use of UAV in remote sensing provides the collection of heterogeneous data through optimal trajectories and a high detail level of the plant shape. Instead of satellite images, UAV-based acquisition process has a high versatility and provides multiple observations of target crops from different viewpoints. The morphological structure of the olive tree is very complex, with many self-hidden branches, which makes the plant monitoring more difficult. In this research, a professional drone (model: DJI Matrice 210) is used to collect heterogeneous-aerial imagery. On board the drone a high-resolution digital camera (model: Sony Alpha 7 RIII) and a multispectral sensor (model: Parrot Sequoia) are mounted. These devices are very different from each other. On the one side, a full-frame RGB camera takes photos with 48 megapixels (MP), thereby observing the study area with a great spatial resolution. On the other side, the multispectral sensor captures reflectance in four spectral bands: the near-infrared (NIR) from 770 nm to 810 nm, the red from 640 nm to 680 nm, the green from 530 nm to 570 nm, and the red-edge (REG) from 730 nm tp 740 nm. This device has a wide-angle lens with a focal length of 4mm to cover more ground area in a single capture. However, a high visual deformation is presented in the image. In this way, the development of plant reflectance is monitored from every wavelength range, which captures meaningful changes in plants related to their health status. A detailed explanation of feature extraction for each multispectral band is described in Section 3.4.

This work includes the temporal domain to detect crucial changes of olive trees at different time frames. Two campaigns have been carried out in the same season to avoid some pruning effects for the estimation of morphological properties.

Table 1. Flight programming and key features of the resulting models.

Date	Drone	Sensor	Overlapping (%)	Images
15 August 2018				264 RGB
	DJI Matrice 210	Multispectral: Parrot Sequoia (1280 × 960)	frontal: 90%	179 (×4) Multi
25 August 2019		RGB: Sony Alpha 7RIII (48 Mpx)	side: 80%	280 RGB
				210 (×4) Multi

summarizes the main parameters for each one. The flights were conducted above all the olive trees in the plantations capturing multiple overlapping images (90%) with both cameras simultaneously. As mentioned above, the lateral and lower branches and shoots of these trees are very important to evaluate some key traits directly related to plant vigorousness, stunted growth, nutritional deficiency, viral infection, etc. For this reason, our approach uses 3D plant models, which were generated by a high-resolution camera and the capturing angle was determined between 65 and 70 degrees. Then, open-source software (Pix4Dcapture) was used to plan the flights, in which the user determines the area of interest, flight direction, longitudinal and lateral overlapping, flight height and the Ground Sampling Distance (GSD). These were conducted close to the solar noon time to minimize the plant shadows and specular lighting. The height of flight was the same for all acquisition processes, 30 meters by considering an adequate GSD in RGB (1 cm) and multispectral (2.8 cm) images.

2.2. Data Processing

The methodology described in this article consists of a fully automated multi-temporal monitoring of olive plantations through the multispectral image mapping on a comprehensive 3D model of individual olive trees. Moreover, the K-means [43] algorithm is used to classify individual olive trees on the point cloud according to their spatial, geometric and spectral features. The proposed framework also supports a multi-temporal analysis and can assume input data at different time frames without any human intervention for the data fusion process. Hence, an individual tree inventory is carried out to evaluate the spectral response for a period of time as well as its relationship with morphological properties such as the tree height and volume.

In summary, the main contributions are (1) multispectral image mapping on comprehensive 3D models of olive trees, (2) multi-temporal monitoring and analysis of the spectral reflectance for individual olive trees. The flow diagram of our framework is shown in Figure 2. This scheme illustrates the main steps, which have been performed. As the first step, input RGB and multispectral images are used for the 3D reconstruction of the olive plantation. Once the position and orientation of the high-resolution model are corrected by some tie points, both point clouds are aligned to be set in the same coordinate system. Before the multispectral image mapping, the reflectance maps are calculated for every multispectral image. Then, the 3D model is enriched by reflectance values, which are weighted considering the viewpoint of the multispectral camera. The next step in the process is the method for the classification of individual olive trees. Finally, according to the data acquired from the two flight campaigns, the variability of morphological and spectral traits is extracted and multi-temporal analysis of these results is presented.

This section presents all the methods developed to create this framework. The evolution of olive trees has been analyzed by fusing multispectral and morphological traits over the 3D models. The 3D reconstruction of the olive plantation, the multispectral image processing, the mathematical model applied for image mapping on the 3D models and the feature-based extraction of individual olive trees are the main techniques.

2.2.1. Point Cloud Reconstruction

The precise modeling of olive trees requires a high spatial resolution to generate most of their thin branches and tiny leaves. Geometry-based methods may reconstruct botanically correct skeletal structures of existing plants. In this article, we mainly focus on image-based modeling as an efficient technique for plant reconstruction in natural environments with acceptable results. In this regard, UAV-based cameras serve as cost-effective solutions to acquire a higher spatial resolution rather than satellite imagery or airborne LiDAR. The use of drone systems provides the capability to plan custom flights with different heights, thereby enabling a more detailed and full observation of each olive tree.

In this paper, the first step of the proposed methodology, Figure 2, is the 3D reconstruction of the olive plantation. According to image sequences captured by the RGB camera and multispectral sensor, two point clouds are generated where multiple olive trees, the ground, buildings, and some human objects appear. The RGB high-resolution point cloud contains a high detail of the plant geometry and is used to measure the height and volume of every olive tree. The multispectral point cloud has a lower spatial resolution and is only used for the alignment process. Then, it is discarded.

The photogrammetric processing is applied through Pix4Dmapper Pro software (Pix4D SA, Lausanne, Switzerland), which is based on the structure-from-motion (SfM) algorithm [44]. This method can detect the same regions of overlapping images, determine their geometric relationships and infer the rigid scene structure (point set) with the pose (position and orientation) of all cameras.

Table 2. Point cloud densification details.

Campaign	Sensor	3D Densified Points	Ground Sampling Distance(cm)
1	RGB	101.846.488	0.84
	Multispectral	3.513.641	3.53
2	RGB	153.441.547	0.78
	Multispectral	3.776.247	3.37

shows the main features of the resulting point clouds. The densification of the 3D models is studied by considering the size of the point clouds and the ground-sampling distance (GSD). Due to the lower resolution of multispectral images, the mean GSD is 3.53 cm, which means a larger ground area per pixel than RGB images. Consequently, the densification of the point cloud is more sparse. Unlike previous datasets, the RGB-based camera provides a higher image resolution, resulting in the mean GSD is 0.78 cm and the point cloud being much higher. Therefore, the pixel size is under 1 cm, which implies a high resolution of olive tree modeling.

2.2.2. Reflectance Map Computing

The plant status is greatly influenced by many environmental effects in the surrounding area. In terms of the assessment of vegetation health, the multi-temporal monitoring of the spectral response of crops plays a key role. Our method uses multispectral images to observe the reflectance of olive trees for two flight campaigns. This section focuses on multispectral image processing from each viewpoint and the extraction of some Vegetation Indices (VIs). The used multispectral system is composed of a multispectral camera, which captures four narrow bands, and a sunshine sensor to measure the incident sunlight. The input images are processed to calculate a reflectance map for each one by applying Algorithm 1.

Algorithm 1: Reflectance Map Computing

The reflectance of any object is its effectiveness to reflect the radiant energy, which means the fraction of incident electromagnetic power reflected. In our study, the spectral reflectance is calculated by measuring the incoming sunlight irradiance and reflected irradiance by the surface of the object captured. Equation (1) is used to estimate the reflectance value (R) for each pixel of every multispectral image.

$$R=k_i\left(\frac{{\mathsf{\Phi }}_e^r}{{\mathsf{\Phi }}_e^i}\right)cos\left(\theta \right)$$

(1)

where $K_i$ is the calibration coefficient of every band i, ${\mathsf{\Phi }}_e^r$ is the radiant flux reflected by the object captured, ${\mathsf{\Phi }}_e^i$ is the radiant flux incidence by the sun and $\theta$ is the angle between the direction vector of sun rays and direction vector of the sunshine sensor.

The incoming sunlight irradiance (${\mathsf{\Phi }}_e^i$) is measured by the sunshine sensor, which is mounted on top of the UAV. This device is continuously capturing the lighting conditions during the flight time. Moreover, the angle between the sunshine sensor and the sunlight direction must be considered to compensate for the light reflection. The mathematical formulation of this magnitude is defined in Equation (2).

$${\mathsf{\Phi }}_e^i=\frac{\nu }{g\tau }$$

(2)

where $\nu$ is a sensor count value, g is the relative gain factor and $\tau$ is the exposure time in seconds.

The next step is to calculate the reflected irradiance (${\mathsf{\Phi }}_e^r$) by using metadata (Exif) stored in the image. Every pixel p in the image I provides a reflected irradiance value, which is calculated by applying the Equation (3).

$${\mathsf{\Phi }}_e^r=f^2\frac{\rho -B}{A\gamma \epsilon +C}$$

(3)

where g is the f-number = 2.2, p is defined by pixel intensity, $\epsilon$ is the exposure time, $\gamma$ is the ISO parameter = 100 and A, B and C are the calibration coefficients measured per camera in production.

Finally, the plant reflectance must be radiometrically calibrated using a panel, which provides known reflectance values of the target surface. Three images are captured over this panel for each multispectral band with different exposure levels. This process is made at the beginning and at the end of each flight. In this way, multi-temporal data, which are taken at different time frames and weather conditions, can be compared to each other. To calculate this calibration coefficient K, Equation (4) is applied.

$$K_i=R_i\left(\frac{{\mathsf{\Phi }}_e^i}{{\mathsf{\Phi }}_e^r}\right)$$

(4)

where $R_i$ is the known reflectance for each band i, ${\mathsf{\Phi }}_e^i$ is the radiant flux incidence from the sunlight and ${\mathsf{\Phi }}_e^r$ is the radiant flux reflected by the calibration panel.

Regarding previous approaches to calculate the plant reflectance, these usually generate either a single orthomosaic or reflectance map by every flight campaign of the study area [45]. Nevertheless, this approach implies the interpolation of many values, which are overlapped on the same pixel coordinates. Consequently, just the mean value may be stored for each pixel. In this regard, the novelty of our approach is the calculation of a reflectance map for each multispectral image, which provides significant values about the spectral response of olive trees from a singular viewpoint. Then, the reflectance maps are going to be mapped on the RGB point cloud by considering the occlusion detection and the viewpoint of the multispectral camera (Section 2.2.3).

In addition, some spectral bands have been combined to estimate some VIs [46,47]. Firstly, the sharp contrast between soil and leaf reflectance in the NIR and red bands is typically used to calculate the Normalized Difference Vegetation Index (NDVI) [48]. It is a well-known spectral index applied in remote sensing. In general, NDVI evaluates the green biomass, leaf cover or chlorophyll per unit ground area. Secondly, green and NIR bands are also used together to estimate the Green-Red Vegetation Index (GRVI), which is strongly influenced by changes in leaf pigments. Thirdly, Ratio Vegetation Index (RVI) [49] is used for the estimation of biomass and leaf area index (LAI). Finally, the Normalized Difference Red-Edge Index (NDRE) is also calculated by combining red-edge and near-infrared bands7. It is more suitable than NDVI for monitoring the growing season when plants accumulate a critical level of leaf cover and chlorophyll content [50]. In the following

Table 3. The calculation of vegetation indices (VIs).

Index	Formula
NDVI	$NDVI=\frac{NIR-RED}{NIR+RED}$
Green Ratio Vegetation Index	$GRVI=\frac{NIR}{GREEN}$
Ratio Vegetaion Index	$RVI=\frac{NIR}{RED}$
Normalized Difference Red-Edge	$NDRE=\frac{NIR-REG}{NIR+REG}$

, the mathematical formula for the calculation of each vegetation index [51].

2.2.3. Multispectral Image Mapping on 3D Model

The ratio of plant absorption and reflection is highly influenced by morphological characteristics of plants such as branching structure, leaf density, canopy volume, etc. The relationship between plant geometry and the reflectance response of plants is one of the main contributions of this paper. We provide an automatic method to enrich the high-resolution point cloud with meaningful traits of the plant status. Therefore, the comprehensive 3D model contains geometric and spectral features by mapping the multispectral images on the high-resolution point cloud. In this section, the third and fourth methods in Figure 2 are described: the data alignment and the inverse 3D projection.

The alignment of multispectral images and a high-resolution RGB point cloud is carried out by the development of an automatic method to set the same coordinate system for both datasets. A possible solution might be the use of ground control points (GCPs) in the multispectral point cloud to correct its position and scale. However, it implies the human intervention by a time-consuming task. Moreover, the multispectral images have a lower resolution than RGB images and being in a grayscale palette makes the tie points recognition difficult. To ensure a fully automated multispectral image mapping we have applied the iterative-closest-point (ICP) algorithm [52] for the geometry-based alignment of multispectral and RGB cloud for every campaign. This method assesses the corresponding point pairs based on a weighting calculation from distance computing and compatibility of normal vectors. A normal vector defines how a surface responds to lighting. The amount of light reflected by a surface is proportional to the angle between its normal vector and the lighting direction. In this work, normals are computed through the reconstruction method (SfM). As a result, a rigid transformation is obtained minimizing the sum of the squared error in Equation (5).

$$E(R,t)=\frac{1}{N_p}\sum _{i=1}^{N_p}\left|\right|x_i-R{p}_i-t{\left|\right|}^2$$

(5)

where: $x_i$ and $p_i$ are corresponding points, t is the translation vector and R is the rotation matrix.

Once both 3D models are aligned, the multispectral point cloud is discarded and the resulting transformation matrix is applied for all multispectral images. In this way, the position of these images is corrected and can be correctly projected to the high-resolution RGB point cloud.

The following step, once output data are in the same coordinate system, is the multispectral image mapping on the 3D model. In general, UAV-based multispectral sensors use fisheye lenses to capture extensive areas at a low altitude. The monitoring of large plantations requires devices with a wide field of view (FOV) to get a high overlapping rate of all images. However, the quality and image resolution of the resulting data-set are highly reduced comparing to digital cameras. Moreover, multispectral images do not comply with the central perspective projection and present a high geometric deformation. As a result, the distortion model of the camera has been considered for the image mapping on the high-resolution 3D model.

For this purpose, an inverse 3D projection is developed to assign each 3D point of the point cloud the corresponding pixel of the multispectral image, in which it is visible. Every 3D point (with coordinates: X, Y, Z) is mapped to image coordinates ($x_d$, $y_d$) by considering the fisheye camera model. This distortion model is determined by the parameters C, D, E, F, which describe an affine deformation of the circular image in pixel coordinates. The polynomial fisheye, with the coefficients $p_2$, $p_3$, $p_4$, is defined in Equation (6).

$$\rho =\theta +p_2{\theta }^2+p_3{\theta }^3+p_4{\theta }^4$$

(6)

where:

$$\theta =\frac{2}{\pi }arctan\left(\frac{\sqrt{X^2+Y^2}}{Z}\right);\theta ϵ[0,1]$$

where: X, Y and Z are the coordinates of 3D point.

By applying the previous distortion model of the multispectral camera, the pixel coordinates ($x_d$, $y_d$) of the 3D point projection is calculated by the Equation (7):

$$\left[\begin{array}{c}{x}_d\\ y_d\end{array}\right]=\left[\begin{array}{cc}C& D\\ E& F\end{array}\right]\left[\begin{array}{c}{x}_{hbt}\\ y_{hbt}\end{array}\right]+\left[\begin{array}{c}{c}_x\\ c_y\end{array}\right]$$

(7)

where:

$$\left[\begin{array}{c}{x}_{hbt}\\ y_{hbt}\end{array}\right]=\left[\begin{array}{c}\frac{\rho X}{\sqrt{X^2+Y^2}}\\ \frac{\rho Y}{\sqrt{X^2+Y^2}}\end{array}\right]$$

and $(c_x,cy)$ is the principal point in pixel coordinates.

As mentioned before in Section 2.1, the oblique angle of drone images provides a full observation for every olive tree. Lateral and lower branches can be modeled as well as the plant reflectance is measured from multiple viewpoints. However, occlusion problems and inaccurate measurements of the reflectance has to be solved to get a correct mapping.

A weighting procedure is developed to assess the reliability of reflectance from each viewpoint. The multispectral camera captures a more reliable reflectance of an object’s surface if its direction vector is similar to the normal vector of such an object. If the angle of both vectors is very high, the reflectance detected is more irregular and less reliable. Our method focuses on the comparison between the vector direction of each multispectral view and the normal vector of every 3D point. If its value is close to 0${}^{\circ }$, the 3D point is observed from almost a perpendicular view in the multispectral image. In this way, reflectance maps are calculated considering 3 ranges: a perpendicular view (0${}^{\circ }$ to 25${}^{\circ }$), an oblique view 25${}^{\circ }$ to 60${}^{\circ }$ and an indirect view (greater than 60${}^{\circ }$). The left image in Figure 3 presents a theoretical example of the previous explanation. The green arrow is the normal vector of a 3D point and the purple arrow is the direction vector of the camera. The angle between both vectors is showed as alpha.

Then, an occlusion test is carried out to get an accurate image mapping on the point cloud. Although a 3D point is located in the visibility range of multispectral imaging, it may be occluded by a closer geometry to the viewpoint of the camera. In this regard, we propose a method to check the point visibility from every multispectral camera. The right image in Figure 3 presents the scheme for this approach. Firstly, 3D points (blue points) inside the view frustum of the camera are candidates to be projected in the image plane. Secondly, these candidate points are ordered by measuring the Euclidean distance between the camera and the 3D point position. Thirdly, from the nearest-to-farthest points, a minimal triangulated surface is formed by considering the target point (the yellow point) and their neighbors. The nearest neighbor search is based on the radius and the angle between normal vectors. These parameters are set to ensure one thousand points at least. The resulting minimal surfaces are used to detect occluded points (black points) which are discarded to be projected in that image. This algorithm is repeated until all points are checked. As a result, the geometry of the olive plantation is correlated to its multispectral response and thus, a meaningful feature pattern can be detected for the segmentation of individual olive trees in the point cloud.

2.2.4. Individual Tree Segmentation

Our approach supports the multi-temporal monitoring at individual tree level. To this end, a point cloud classification method is carried out to identify every olive tree in the surveyed area. 3D geometry and multispectral data are the input data for the segmentation of individual trees. Unlike previous works based on image segmentation for individual tree registration [53], which does not work properly in areas with high overlapping of trees or hard shadows, we propose an efficient 3D-based segmentation of individual olive trees.

Initially, a coarse segmentation of the point cloud is developed to classify the vegetation areas and the soil surface. At the canopy level, the sharp contrast between the soil and leaf reflectance is used for the automated identification of vegetation areas in the ratio from the NIR to the red band. For this reason, the NDVI is the most adequate index for olive tree recognition and also it is not highly affected by the shadow cast of trees.

Nevertheless, some outliers of the NDVI cause errors in the point-based classification. To overcome this problem, an optimization method is carried out by applying k-nearest neighbors (KNN) [54]. The category of every 3D point is compared to their neighbors and it changes if most of these points have been classified in other classes. The density of the point cloud is very high, so a higher radius for the neighbor selection might affect the final performance negatively. The radius for the neighbor selection is set as 50 cm by considering the GSD of the point cloud and the impact on performance. According to the k-dimensional tree (k-d tree) [55], which organizes the point cloud in a space with k dimensions, an efficient range search is performed. This data structure is three-dimensional and each level splits all subdivisions along a specific dimension using a perpendicular hyper-plane to the corresponding axis.

The following step is to identify each olive tree as an individual entity. For this purpose, 3D points belonging to the vegetation class are partitioned into different clusters by the application of the K-means algorithm. The number of possible clusters is not explicit, thus it depends on the heterogeneity of the scene. Through this method, a search for nearest-neighbor points is developed regarding these constraints: the distance between the 3D points, the minimum cluster size, and the direction of normal vectors. In this work, we have defined the maximum distance lower than 15 centimeters, the maximum degree between normal vectors from 0 to 180 degrees and the cluster size must be greater than one thousand points.

Once each olive tree is identified as a unique entity on the point cloud, multi-temporal monitoring and analysis may be performed. According to the previous method for fusing multispectral data and 3D reconstruction of the olive trees, several semantic layers may be overlapped to study different environmental variables. Moreover, an individual inventory for each olive tree can be performed to study its evolution by considering a multi-temporal data series.

2.2.5. Morphological-Feature Extraction

According to the sixth step in Figure 2, the morphological development of olive trees is studied by monitoring the changes in height and volume. These parameters are necessary to determine adequate plant growth. In this regard, pruning activity has a high influence on these properties. This issue has been considered and the collected data in the same season of two different olive harvesting.

The plant height has been used to measure crop growth [24]. This feature provides additional information that complements spectral vegetation indices for predicting growth and yields. In this study, the plant space is decomposed by a 3D octree with a spatial resolution of 20 cm. This value is enough to estimate an accurate overall tree volume and assure the efficient performance of the method. This data structure is based on a tree topology in which every internal node has exactly eight children [56]. Firstly, a bounding box is defined to calculate the maximum dimensions of the tree. In this way, the height can be directly obtained from this geometric shape. Then, the whole space is decomposed into multiple voxels and a voxel-based inclusion test is applied. In this phase, we have to check if any voxel contains one 3D point at least. Finally, the total volume is calculated by the sum of each voxel volume, which partially contains some parts of the olive tree.

The proposed framework allows the processing of multi-temporal data to analyze the evolution of plant morphology and leaf spectral reflectance over time. In this regard, the variability of studied features (

Table 4. A summary of all characteristics under study.

Feature	Description
Multispectral bands
Green	The highest plant reflection is visible in this band.
Near-infrared	It is the least sensitive band to chlorophyll.
Red	It is mainly influenced by the humidity, biomass and soil minerals.
Red-edge	This band is relevant for stress status assessment.
Vegetation indices
NDVI	It is used for vegetation recognition and the assessment of the crop health.
RVI	It can be used for biomass and leaf area index (LAI) assessments.
GRVI	This index is used for the leaf density or vigor of vegetation.
NDRE	It is sensitive to chlorophyll content in leaves and soil background effects.
Morphological features
Plant height	The maximum distance from the soil to the highest branch
Plant volume	The space occupied by the 3D structure of olive trees

) is carried out by statistical analysis to detect key changes for two flight campaigns.

In this study, multispectral and morphological features are considered for the analysis process. The evolution of olive trees is analyzed by a qualitative and quantitative approach. On the one hand, the measurement of spectral traits is meaningful to assess the plant health, thereby the crucial changes of the reflectance between both campaigns are studied. In this regard, the leaf pigments play a key role in the reflected light in the observed narrow bands. Moreover, NDVI, RVI, GRVI, and NDRE are also compared by analyzing the significant variability of each one on graphics and colorized 3D models. On the other hand, the development of the plant shape is also included in the analysis stage. The height and volume are calculated for each olive tree. These values are fundamental to get a comprehensive knowledge of the plant growth. Our approach provides a fully automated procedure to study the multispectral information and high detailed geometry of an olive plantation. The results of our analysis are presented in Section 3.

2.3. Validation Procedure

In this study, the extraction of morphological and multispectral features of olive trees has to be validated for each flight campaign. Both flights were performed in the same season, corresponding to the olive fruit final ripening stage. At this time, olive trees are not affected by pruning factors, which are performed after harvesting olive fruit. Firstly, to ensure the calibration of the measured reflectance in two acquisition processes, a calibration panel is used. A calibration coefficient (K) is calculated for each narrow band as mentioned in Section 2.2.2.

Moreover, the geometry of the RGB point cloud has to be measurable to obtain accurate morphological features of the olive trees (the height and volume). Although the UAV includes a Global Navigation Satellite System (GNSS) receiver, it has a random error so the resulting scale and position of the point cloud are not correctly determined. To overcome the previous problem, six GCPs are distributed throughout the surveyed area. These points are acquired using a Real Time Kinematic (RTK) GNSS (Topcon GR5) with a centimeter accuracy linked to the Andalusian Positioning Network. The overall accuracy is studied by the RMSE for n observed checkpoints, as the below Equation (8) and (9). For a 3D point with coordinates (X, Y, Z), the residuals are calculated by subtracting the coordinates that were measured by GNSS and the interpolated corresponding 3D reference point on the point cloud. The resulting model has been corrected using previous GCPs, which have been clearly identified in the point cloud.

$$RMS{E}_{X,Y}=\sqrt{\frac{1}{n}{\mathsf{\Sigma }}_{i=1}^n{\left(X_{i,ref}-X_{i,GNSS}\right)}^2+{\left(Y_{i,ref}-Y_{i,GNSS}\right)}^2}$$

(8)

$$RMS{E}_Z=\sqrt{\frac{1}{n}{\mathsf{\Sigma }}_{i=1}^n{\left(Z_{i,ref}-Z_{i,GNSS}\right)}^2}$$

(9)

Regarding the use of fixed marks to measure the GPS points on the ground, it is not adequate because many tillage techniques are carried out in the olive plantation. Moreover, one of our goals is the automated flow, which is why significant objects in the surveyed area are used to determine the location of GCPs (Figure 4). In this way, the GPS measurements are required for the first time.

Once GCPs are set, a vertical and horizontal adjustment is applied to the point cloud. Therefore, the 3D model is translated, rotated and scaled to improve the accuracy of the point cloud. According to the validation of the height and volume of olive trees, the great size of the tree crown, as well as the irregular shape of the canopy makes it difficult for field data extraction [57]. In this study, these parameters have been checked by the measurements of physical objects in the olive plantation, which are visible from the aerial images. Specifically, the height and volume of ten reference objects are measured (Figure 5). These human-made objects can be measured much easier than olive trees and their correct dimensions validate the geometric precision of olive tree models. In this way, due to the shape of these physical objects being fixed, the manual field data acquisition only was made for the first time.

3. Results

According to the application of previous algorithms, the olive plantations can potentially benefit from monitoring and analytic operations. The proposed framework provides the capability to study the evolution of olive trees by considering some morphological and spectral features in a 3D space where lateral and lower regions are observed from multiple viewpoints. The following results prove the novelty of this method.

3.1. Characterization of Study Area

The researched olive plantation is characterized by a high-resolution point cloud with 0.78 cm GSD in the first flight campaign and 0.84 cm in the second one. This 3D model contains a high detail level of the plant geometry. Figure 6 presents the result of the photogrammetric process.

The overall shape of the plant canopy has been accurately generated. The trunk, main branches and multiple leaves of olive trees can be correctly identified in the 3D model. More detailed views of some olive trees are shown in the bottom images of Figure 6, which have high point-based densification around 250 thousand of 3D points as average. Some holes appear in the ground because these regions were occluded by the plant structure.

Likewise, multispectral images have been captured to characterize the surveyed area with the spectral response of olive trees in two flight campaigns. Figure 7 shows all observed bands from the multispectral camera. Each one measures the reflectance from a different wavelength range, and these measurements provide meaningful information to determine key features of the plant status. According to these images, a point cloud is generated for each flight campaign. These are just used as input data for the automated alignment process between the RGB point cloud and multispectral imagery.

3.2. Accuracy Assessment

Regarding the accuracy of the previous photogrammetric process for each flight campaign,

Table 5. Error of computed point to ground control points (GCP) and theoretical error in each direction (X, Y, Z) on the six (GCPs) for each flight.

GCP Point (m)			Flight Campaign	Error Distance to GCP (m)			Theoretical Error (m)
X	Y	Z		X	Y	Z	X	Y	Z
398,765.87	4,212,946.76	235.08	1	0.49	−1.00	−6.66	0.047	0.020	0.216
			2	−1.00	−2.36	11.53	0.004	0.004	0.015
398,727.11	4,212,954.01	231.29	1	0.45	−0.96	−6.53	0.002	0.002	0.028
			2	−0.98	−2.43	11.42	0.006	0.005	0.021
398,741.43	4,212,919.09	232.66	1	1.39	−1.77	−6.35	0.021	0.037	0.328
			2	−0.28	−1.99	11.59	0.001	0.001	0.002
398,772.76	4,212,932.51	235.38	1	0.52	−1.107	−6.41	0.049	0.048	0.115
			2	−1.01	−2.28	11.62	0.002	0.002	0.009
398,768.31	4,212,916.33	235.71	1	0.814	−0.657	−6.52	0.036	0.063	0.103
			2	−0.65	−1.91	11.68	0.008	0.011	0.029
398,755.93	4,212,954.08	238.060	1	0.591	−0.965	−6.71	0.017	0.008	0.098
			2	−0.93	−2.35	11.49	0.003	0.004	0.020

presents the absolute and relative errors of the reconstructed point cloud. The error in the GCP is the distance between the initial position of computed points in the 3D model and the position in their corresponding GCPs. It means the accuracy of drone GPS. A Higher deviation was found in the Z-axis (the height) whereas the error rate was much lower in both, the X and Y axes. According to the theoretical error for each direction (X: easting, Y: northing, and Z: height), it indicates the relative accuracy using the error ellipsoid axes of GCPs. The relative accuracy is quite great under 5 cm in X and Y and 33 cm on the Z-axis. A summary of the RMSE for both acquisition stages is presented in

Table 6. Root mean square error (RMSE) of global and relative accuracy in each direction (X, Y, Z) for each flight campaign.

Flight Campaign	RMSE/Global (m)		RMSE/Relative (m)
	(X,Y)	(Z)	(X,Y)	(Z)
1	1.313	6.53	0.042	0.148
2	2.877	11.555	0.005	0.016

. In general, the absolute error on the second point cloud is higher than on the first and the relative error is slightly more accurate for the second reconstruction.

Our methodology provides a fully automatic process for the analysis of the evolution of olive trees in two time frames by considering some morphological and physiological features. One of our goals is to avoid human intervention to acquire UAV-based data and measurements taken in the field. Therefore, the validation of the estimated height and volume is carried out using fixed physical objects in the olive plantation. In this regard, ten reference objects are considered to be measured in the field for the first time. The estimated measurement is very similar to the field data. The mean error in height is 5 cm and the mean error to estimate the volume is 0.4 m${}^3$. These results demonstrate the adequate georeferencing of the point cloud and the precise geometry of the 3D model.

3.3. Heterogeneous Data Fusion

In this section, we present the results by the fusion between the high-resolution point cloud and multispectral images. The position of multispectral images is corrected by the alignment of the multispectral and RGB point cloud (Figure 8). This last one is taken as reference, which has been previously georeferenced by multiple GCPs. Regarding the accuracy of this method, a global RMSE is used to measure the differences between the aligned point clouds. In the first campaign, the error is 2.80 cm and 2.20 cm in the second campaign. These values are considered because both are lower than the GSD of the multispectral point cloud (3.53 cm). Then, by applying the proposed method for multispectral image mapping on the point cloud, the RGB point cloud is also enriched with meaningful semantic layers, which describe the spectral development of plants. The merge of both data types is useful for multiple monitoring tasks and assists with the automated identification of vegetation in the 3D model.

According to the method for the automated recognition of olive trees (Figure 2), these are classified by considering some key features. The left image of Figure 9 shows the segmentation of existing vegetation and the right image depicts the recognition of individual olive trees. Firstly, the NDVI is used to identify the vegetation areas. Then, meaningful spectral traits are studied to differentiate the olive trees to the rest of the existing vegetation. The right image of Figure 9 presents the individual segmentation of every olive tree. According to the leaf densification of this plant specie, the olive trees can be identified with a reflectance in NIR as 0.30 and REG as 0.20. Other existing plants use to have a more leafy canopy than olive trees, so the mean reflectance slightly increases in previous bands. Moreover, the spatial component helps to K-means to identify every olive tree. This method works properly because all the olive trees in the plantation (72) have been detected as unique 3D models.

3.4. Morphological and Spectral Features

Once olive trees have been identified in the olive orchard, the height and volume are measured for each 3D model. A part of the point cloud is shown in Figure 10 to explain the results of this process. The bounding box of every olive tree is calculated to determine the height and the initial volume for the plant space decomposition. The volume is estimated by a voxel-based partitioning of the plant space. The voxels, which have been partially occupied by some 3D points of the plant geometry, are colored red.

In terms of the estimated height and volume for every olive tree, Figure 11 presents the resulting measurements of 72 olive trees, which have been recognized in the surveyed area. The X-axis represents different ranges of height (m) or volume (m${}^3$) and the Y-axis shows the number of olive trees for each group. In general, the mean volume of olive trees is close to 16.7 m${}^3$ in the first campaign. For the second one, the volume measurements slightly increase, it is 17.28 m${}^3$ on average. According to the height variability, the mean value for both time frames is 2.9 m. These results are discussed in more detail in Section 4.

According to multispectral features, which are studied for each olive tree, Figure 12 presents the distribution of reflectance values in every wavelength (left image) and the vegetation indices (right image) for the two campaigns. Regarding the reflected light captured by every narrow band, the higher values are detected in NIR. The vegetation spectrum typically absorbs in the red, slightly reflects at the green wavelength and strongly reflects in the NIR and REG bands. In both flight campaigns, the NIR values are the highest close to 0.35 on average. In the REG band, the mean reflectance is close to a 20% of incident light. The lowest values are measured in the visible range, the green and red bands, where leaves absorb most of the sunlight. Regarding the results of VIs, a positive trend is detected if we compare the resulting values between the first and second campaigns. The overall value of the NDVI in olive trees changes from 0.55 to 0.81, which means a significant improvement in the plant vigor. Relating to RVI values, these are much higher in vegetation areas in which a strong contrast is observed between the red and NIR bands. The GRVI measures the relationship between the NIR and green bands. In general, the reflected light in the green band is higher than the red band, thus GRVI values are usually upper than RVI. Finally, the NDRE presents the lowest values in the first campaign and these significantly increase for the second one. It is due to the higher leaf reflectance, which is observed by the REG band in the second flight campaign. A more detailed discussion of the plant development is described in the following Section 3.5.

3.5. Multi-temporal Analysis

The evolution of plant traits has been studied in two flight campaigns. Multispectral and morphological features are extracted for each olive tree by observing the top, lateral, and low branches. These characteristics are analyzed by the variability of data through different graphics and by a visual-based inspection in the 3D model of the olive plantation.

Figure 13 presents a summary of the morphological and spectral changes in the olive plantation. According to the variability of data in general, the trend in the crop sustainability is positive. Firstly, the height presents minimal changes close to 0.04 m on average. It is due to the fact that during the pruning stage, farmers remove the highest branches to facilitate the olive fruit harvest. Secondly, the mean volume of olive trees has increased 0.53 m${}^3$, and the overall volume of the olive plantation has increased to 113.17%. Regarding the main changes of the spectral traits in olive trees, a greater difference is detected in the infrared domain rather than in visible bands. Most of the incident light is absorbed by plant leaves so a lower reflectance is detected in green and red bands. In the second campaign, the visible light is absorbed by 2% more than the first one. In contrast to the red and green bands, the plant reflectance increases in the NIR and REG bands for the second campaign. Finally, the variability of the vegetation indices demonstrates the healthy evolution of olive trees. The estimated VIs are higher in the second campaign. Specifically, the NVDI increases from 0.55 to 0.81, the mean NDRE values in the plant canopy changes from 0.15 to 0.62, which means better health in the studied crops.

Figure 14 presents the evolution of studied vegetation indices on the point cloud for two different campaigns. The color scale is defined by the highest values in the green color and the lowest values in the black color. By the visual inspection of the enriched 3D model for each campaign, an initial assessment of the crop health can be made. The interactive visualization of every plant regions from different viewpoints is very useful to study the plant status. In the first campaign, high contrast of olive trees and ground can be detected for each VI model. According to the NDVI point cloud, a high contrast is visible between the ground and olive trees. Nevertheless, in the second campaign, the emergence of low vegetation around the olive trees supposes higher values on ground level. The RVI point cloud provides a saturated color for the ground reflectance. The variability of this index in the plant canopy is not very significant. The visualization of the GRVI 3D model shows a greener color for olive trees and a higher value on the ground level due to the increase of reflectance detected in the green band. Finally, the NDRE is the most sensitive index to the leaf area and orientation. This index can provide a better measurement of the reflectance variability in areas where the NDVI measures uniform values. In the second campaign, the reflectance of the REG band significantly increases on plant and ground level for the second campaign. Consequently, the resulting NDRE point cloud shows higher values for the ground and olive trees. In general, the overall development of olive trees can be considered to be quite healthy. The variability of studied indices demonstrates that in the second campaign, olive trees have more vigorous branches with a higher leaf and canopy reflectance in the near-infrared domain. In all VI models, olive trees present a higher green colorization, which implies better crop sustainability.

4. Discussion

The results provided by the application of our methodology demonstrate its utility to extract meaningful traits on the plant structure of olive trees. In this farming sector, the use of novel remote sensing techniques during every tillage stage is not still sufficient for monitoring the plant status and early detection of some diseases, which cause a negative effect of the olive production. Therefore, the use of emerging platforms like UAVs and the development of efficient methods are becoming increasingly important gather and combine heterogeneous data, which provide meaningful features of an olive tree health [9,10]. This paper proposes a novel approach focused on fusing geometric and spectral traits of an olive orchard. The extraction of morphological and multispectral features of comprehensive 3D models, as well as the multi-temporal monitoring of these features to assess the plant development of olive trees, are the core of our method.

Remote sensing methods for the estimation of plant height and volume increase in importance for PA, using either LiDAR or photogrammetry. In this study, image-based sensors are used to observe the structure and reflectance for each olive. Consequently, olive trees were characterized by the spectral reflectance and some morphological properties such as the height and volume. Yan et al. [58] proposed the use of a concave hull operation for the extraction of structural plant parameters. In contrast, our solution is based on a voxel-based decomposition of the plant space using a 3D octree, which is widely used for the partition of three-dimensional space by recursive subdivision [56,59]. An advantage of our method is that it is not affected by the irregular shape and disperse branching of the tree canopy whereas the calculation of the concave hull is not the most adequate solution in this case. Moreover, the triangulation of the concave hull requires the correction of non-manifold geometry in the photogrammetric model and the generation of the 3D mesh, which negatively affects the final performance.

One of the goals of our approach is to ensure the automatic process for each step of the proposed methodology. Unlike other approaches [45] based on field data acquisition and GCPs measurement in every flight campaign, the validation of the morphological features was made by measuring the dimensions of fixed physical objects in the olive plantation. It supposes a higher efficient technique to acquire validated data for further flight campaigns without repeated time-consuming acquisitions of field data. Regarding the result of this study, the creation of a spatio-temporal inventory of individual olive trees by the 3D plant structures, spectral reflectance and VIs, as well as the vegetation evolution are discussed in the following sections.

4.1. Inventory of Individual Olive Trees

One of the main contributions of our approach is the automatic fusion between the photogrammetric point cloud of olive trees and multispectral images. The resulting geometry of olive trees has been modeled with a high spatial density (20 thousand of points by a cubic meter on average). The managing of the high detailed model of the olive plantation has been accelerated by a spatial data structure, the k-d tree. Therefore, an interactive visualization of the point cloud can be performed in real-time, using the canvas of the proposed application. It is an important novelty of our method in contrast to other approaches based on bidimensional and fixed observations [6,7,8]

In this study, the high-resolution RGB point cloud is enriched with a reflectance response at various multispectral bands. The RMSE of data alignment is lower than 3 cm, which is considered valid because the mean GSD of multispectral images is greater than 3.38 cm. As a result, every 3D point contains many reflectance values at visible bands (green and red), REG and NIR bands and some vegetation indices (NDVI, RVI, GRVI, and NDRE). By the previous characterization of the point cloud, different semantic layers can be analyzed using the proposed framework. It means a novel advance in PA in order to review the spectral response of target crops from any viewpoint in a 3D scenario. Previous works, which use the VIs for the plant assessment [60], can benefit from our approach by using our methods to study the spatial distribution of reflectance on the point cloud.

Moreover, the characterization of the point cloud with multispectral features is meaningful for the recognition of individual olive trees. By applying our method, all olive trees are correctly identified, 72 plants in total. Therefore, the overall reflectance and the height and volumes can be obtained for each tree. In contrast to previous work based on satellite or air-bone to acquire images for monitoring of olive trees [3], we can characterize an olive plantation with a higher spatial resolution of plant structures and their multispectral responses from a 3D perspective. This work focuses on the automation of every method to reduce the time-consuming tasks for heterogeneous data fusion and field data acquisition. Consequently, an efficient multi-temporal approach is provided to assess the evolution of plant health by visualizing the enriched 3D models and the analysis of data variability.

4.2. Vegetation Evolution

The use of several sources of remotely-sensed data and field data, which may differ in spatial resolution, spatial–temporal coverage and sensor origins, is becoming increasingly popular due to the recent rapid development in remote sensing techniques [33,34]. In this work, we address the evolution of olive trees by considering morphological and multispectral traits of olive trees. During both flight campaigns, the vegetative cycle of olive trees was the same when the final stage of olive ripening was underway. In terms of a qualitative and quantitative analysis of the plant development, which is observed in Figure 13 and Figure 14, some meaningful conclusions can be extracted.

In the first campaign, olive trees present 16 m${}^3$ and 2.9 m like mean volume and height values. The reflectance in the visible range is quite low, 8% in green and 6% in red. Otherwise, NIR and REG wavelengths present a higher reflected light close to 30% and 17%, respectively. This ratio of values is greatly determined by leaf pigments of the plant. The most visible light, red light is whereas infrared light is least sensitive to chlorophyll and is more reflected. According to the studied VIs by combining previous multispectral bands, the mean NDVI of olive trees has a low value (0.55). It means that plants might present nutritional stress, a low vigor, or a mid-low canopy cover. By visualizing the enriched 3D model, a high contrast is presented between the soil and trees. Focusing on olive trees, healthier regions of the plants are easily detected by the green color, in contrast to stressed parts that become more brown. In the middle of the surveyed area, a high number of olive trees were detected with lower values of VIs. These key observations were taken into account for the next monitoring.

In the second campaign, the evolution of olive trees shows a positive trend in comparison to the first one. The mean volume of every tree is 17.28 m${}^3$, which means 3% of the volume growth. No significant differences in the tree height measurements have been detected. In the studied multispectral bands, the reflectance in visible bands has decreased to 5% in green and 6% in red. Regarding NIR and REG values, these have increased to 35% and 20% respectively. These changes are interpreted as a significant improvement in plant health. It is justified by the fact that VIs provide better results about the plant sustainability. The NDVI increases to 0.81 and the mean NDRE is 0.59. These differences are caused by the higher reflectance in the infrared domain. Moreover, GRVI and RVI present a slight increase to 0.35 and 0.47 respectively. The visual-based analysis, by using our 3D scenario, demonstrates the adequate evolution of olive trees in the studied plantation. A high difference is detected in the NDRE model, where the soil and plants show higher values. It is due to the emergence of low vegetation around olive trees in this time frame. Finally, regarding the area in which the lowest reflectance values were obtained in the first campaign, only three olives maintain a similar behaviour. In these cases, the NIR values are lower than the rest close to 20%. Nevertheless, the surrounding olive trees have increased the NDVI by 10%. This approach enables a more complete characterization of olive trees parameters as well as the development of a fully automatic process to analyze the plant evolution by a visual and statistical analysis of the individual crop’s profile.

5. Conclusions

The potential of novel sensors in precision farming creates a great opportunity to advance in the development of efficient methods and techniques for UAV data processing. The huge quantity of data, which this technology provides in a non-intrusive way, and the high number of available sensors to monitor the vegetation status require advanced applications for fusing heterogeneous information.

The proposed innovative tool has a high utility in PA for using heterogeneous and multi-temporal data series. Our approach proposes novel advances in morphological and spectral feature extraction, 3D segmentation of vegetation and the fusion of the plant geometry and multispectral traits to characterize the comprehensive plant structure. In addition, a visual-based analysis is carried out by using the interactive canvas of our framework. An intuitive interaction with the point cloud, as well as the fluent visualization of every 3D model, becomes a potential scenario to monitor and find out interesting variables in the target crop. Likewise, the variability of data is studied by a statistical approach to detect meaningful changes of morphological and multispectral features in two flight campaigns. Our method is based on mapping many multispectral images on the plant geometry, thereby obtaining spectral information about every region in the 3D plant space. One of our goals was the generation of an efficient and fully automated process for fusing multispectral data with the plant geometry. Thus, experts can directly inspect the plant health on a detailed 3D model with multispectral traits of the tree structure.

Although this solution focuses on olive trees, it can be also applied to other fruit trees. In this study, the surveyed area contains 72 olive trees. High-resolution RGB and multispectral images were captured by UAV-based cameras to reconstruct a 3D model of the olive plantation and to measure the reflected light in some specific spectral bands. The resulting geometry was highly detailed with accurate modeling of branches, the trunk, and the tree crown. The 3D model of each olive tree was characterized by the height and volume as well as the mean spectral reflectance and VIs by combining some narrow bands. To compare these features in different time frames, the position, orientation and scale of the RGB point clouds were corrected by the use of GCPs. As a result, an enriched point cloud was obtained with a comprehensive 3D model and accurate reflectance measurements of the plant structure. According to the variability of the studied features, some conclusions were drawn. In general, the target olive plantation shows significant symptoms, which indicate a positive trend. According to our analysis, the mean volume of each olive tree slightly increases in the second campaign. The NDVI value and infrared light increase significantly in the plant space. Most olive trees present more greenery in the second campaign and adequate plant growth is observed. This trend is visible by reviewing the enriched models of olive tree structure in the 3D environment.

Several open problems can be subject to further research. Regarding the gathered features about the morphology and spectral response for each tree, we will focus on the study of disease detection. Moreover, we want to use the proposed method for soil monitoring. Finally, the application of this framework for the plant species classification is an interesting topic to approach in the future.