Development and Testing of a UAV-Based Multi-Sensor System for Plant Phenotyping and Precision Agriculture

Abstract

Unmanned aerial vehicles have been used widely in plant phenotyping and precision agriculture. Several critical challenges remain, however, such as the lack of cross-platform data acquisition software system, sensor calibration protocols, and data processing methods. This paper developed an unmanned aerial system that integrates three cameras (RGB, multispectral, and thermal) and a LiDAR sensor. Data acquisition software supporting data recording and visualization was implemented to run on the Robot Operating System. The design of the multi-sensor unmanned aerial system was open sourced. A data processing pipeline was proposed to preprocess the raw data and to extract phenotypic traits at the plot level, including morphological traits (canopy height, canopy cover, and canopy volume), canopy vegetation index, and canopy temperature. Protocols for both field and laboratory calibrations were developed for the RGB, multispectral, and thermal cameras. The system was validated using ground data collected in a cotton field. Temperatures derived from thermal images had a mean absolute error of 1.02 °C, and canopy NDVI had a mean relative error of 6.6% compared to ground measurements. The observed error for maximum canopy height was 0.1 m. The results show that the system can be useful for plant breeding and precision crop management.

1. Introduction

High-throughput phenotyping is a method of using sensing and automation technologies to measure phenotypic traits in order to accelerate breeding programs for major crops [1] in support of increased plant productivity and resilience to biotic and abiotic stresses impacting plant health and growth. Traditional methods for collecting field data to support crop breeding require significant human labor. As a result, designing an automatic phenotyping platform to work in the field has attracted the attention of researchers in both plant science and engineering communities. Early solutions for an automatic data collection platform utilized ground vehicles (either tractor or robotic platforms) equipped with sensors to acquire the desired data [2,3,4]. During data collection, ground vehicles are driven either manually or automatically through each plot, and data are collected and stored in a data logger or computer. Ground vehicles have the advantage of supporting large payloads and can easily carry multiple sensors simultaneously. In addition, these vehicles can control the data collection environment, for instance, by using enclosures to guarantee data quality and to ensure similar acquisition conditions (e.g., illumination and wind). Ground platforms also have several disadvantages, however, including low data scanning speed, the need to adjust planting layout for certain crops to accommodate vehicle characteristics, soil compaction resulting from frequent data collection, and challenges of adapting the platform to different crops once the design is fixed.

Unmanned aerial vehicles (UAVs) offer an alternative to ground data collection in support of high-throughput phenotyping and can address disadvantages of ground vehicles to some degree. Compared to ground platforms, UAVs can provide higher data scanning speed and can cover relatively larger fields. Due to the physical separation from plants, UAVs can be adapted easily to different types of crops and at different growth stages. Furthermore, UAVs can be controlled automatically by their onboard autopilot system; thus, they require less human intervention during data collection. Limitations of UAVs include reduced payload when compared to ground vehicles, resulting in equipment weight restrictions. In addition, the quality of data collected by UAVs is more likely to be affected by the environment due to lack of environmental controls. The spatial resolution of aerial data is usually lower than that of ground systems.

There have been many studies on UAV-based high-throughput phenotyping published in recent years [5,6,7], and a variety of sensors have been integrated into UAV platforms, including multiple types of cameras (RGB, thermal, multispectral, and hyperspectral) and Light Detection and Ranging (LiDAR) sensors [8]. Cameras and LiDAR can provide spatial and spectral information about plants, which can be used to measure directly or indirectly various phenotypic traits. RGB images can be used to estimate leaf area index (LAI), crop emergence, and to count flowers [9,10,11]. The Structure from Motion (SfM) technique can be used with RGB images to generate a Digital Surface Model (DSM), which can be used to estimate plant height, predict biomass/yield, detect crop lodging, and derive canopy structure metrics [12,13,14,15,16]. Multispectral imaging can provide vegetation indices, such as the widely used Normalized Difference Vegetation Index (NDVI), which have proven to correlate with leaf area index, plant disease, plant nutrient deficiency, and yield [17,18,19,20,21]. Thermal images have been used frequently to measure canopy temperature, which is an indicator of stomatal conductance and plants’ response to water stress [22,23]. Therefore, thermal images can be used to detect water stress [24,25,26]. Hyperspectral imaging provides more detailed spectral information of targets when compared to multispectral cameras and can be used to further monitor plants’ physiological status and related processes [27]. For example, one study showed that the Photochemical Reflectance Index (PRI) from hyperspectral imaging can be used to access the water stress in maize [28], whereas several other studies applied hyperspectral images to estimate biomass and yield, as well as to detect plant diseases [29,30,31,32].

Despite the great potential and progress in the field of UAVs when applied to high-throughput phenotyping, several application challenges remain to be further studied. First, most UAV platforms have been designed based on a particular aerial vehicle and one or more sensors. For example, although multi-sensor UAV systems were developed to acquire complementary information to measure complex phenotypic traits such as water stress and yield [33,34], these systems are typically limited to up to two specific sensors without flexible data acquisition software system that can adapt to new sensors. Rigid designs are difficult to expand, adapt to other needs, and reuse for other research or applications. This limitation can be circumvented by designing a multi-sensor data acquisition system that can be deployed on different UAV platforms. Second, since sensors are affected by environmental conditions that could make the data collected from different sessions inconsistent and non-comparable, calibration methods need to be developed for different types of sensors. For example, thermal images are affected by the atmosphere and need to be calibrated properly using environmental data to obtain accurate thermal readings [35]. Lastly, converting sensor data into meaningful phenotypic traits for variety selection and plant growth assessment requires a complex set of data processing algorithms and tools. Therefore, designing a framework that can incorporate existing and future algorithms remains a major task for researchers. Additionally, although UAVs have been used widely for plant phenotyping, it has remained a challenge for researchers without an engineering background to develop a feature-rich data collection UAV system.

In order to address the above challenges, the main goal of this paper is to design a flexible unmanned aerial data acquisition system as well as an airborne sensor calibration and data processing framework that can be used by researchers in phenomics and precision agriculture. Specific objectives include the following: (1) Designing a data acquisition system that can be attached easily to different UAV platforms, (2) developing camera calibration protocols that are transferable to other data collection efforts, and (3) designing a data processing pipeline for extracting phenotypic traits from the raw data. The design of the system was open-sourced and is available to researchers and other individuals.

2. System Design

We adopted a modularized approach to design the unmanned aerial system (UAS), aiming to separate the full system into an aerial platform and a data acquisition system (DAS). In this manner, the DAS can be carried by different aerial platforms or even by ground platforms. We designed two versions of DAS over the past five years. The first version consists of a DSLR camera (Lumix G6, Panasonic, Kadoma, Japan), a multispectral camera (RedEdge, MicaSense, Seattle, WA, USA), and a thermal camera (Tau 2, FLIR Systems, Wilsonville, OR, USA) (

Table 1. Specification of the sensors.

	Thermal Camera	Multispectral Camera	DSLR RGB Camera	Industrial RGB Camera	LiDAR
Manufacturer	FLIR systems	MicaSense	Panasonic	FLIR systems	Velodyne
Model	Tau 2	RedEdge	Lumix G6	GrassHopper3	VLP-16
Dimensions (mm)	$44.5\times 44.5\times 30$	$113\times 65\times 46$	$122\times 85\times 71$	$44\times 29\times 58$	$103\times 103\times 72$
Weight (g)	112	150	390	90	830
Resolution	$640\times 512$	$1280\times 800$	$4608\times 3456$	$2448\times 2048$	Vertical: $1.33$$°$ Horiz.: 0.1$°$–0.4$°$
Focal length (mm)	25	5.4	14–42	5	N/A
Max FPS (Hz)	30	1	1	75	20
Spectral range ($\mathrm{n}$$\mathrm{m}$)	7500–13,500	475, 560, 668, 717, 840	N/A	N/A	N/A
Accuracy (cm)	N/A	N/A	N/A	N/A	Up to ±3

). This version of the DAS has a simpler design and does not include the Robot Operating System (ROS). The second version of the DAS replaced the DSLR camera with an industrial camera (GrassHopper3, FLIR Systems, Wilsonville, OR, USA) and added a 3D LiDAR sensor (VLP-16, Velodyne Lidar, CA, USA). In addition, this DAS has an improved mechanical design and can be controlled using ROS. This paper briefly introduces the design of the first version of the DAS and focuses on the development of the second version.

2.1. First Version of the Data Acquisition System

The DSLR, multispectral, and thermal cameras integrated into this DAS were controlled by a single-board computer (Raspberry Pi 3) (Figure 1A). The cameras and Raspberry Pi were mounted on a camera case, which was attached to a high payload UAV (S1000+, DJI, Shenzhen, China) (Figure 1B). A GNSS/IMU (VN200, VectorNav, Dallas, TX, USA) was used to record the position and pose of the images. Two DC-DC converters were used to power the thermal and the multispectral cameras, as well as the Raspberry Pi (power supplied by the UAV). Software was developed to run on the Raspberry Pi triggered cameras and saved thermal images and geolocation to the on-board memory.

2.2. Second Version of the Data Acquisition System

The second version of the DAS consisted of three imaging sensors (thermal, industrial RGB, and multispectral cameras), a LiDAR sensor, and a single-board computer (Manifold, DJI, China). For simplicity, the remainder of this paper will use RGB camera to refer to the industrial RGB camera. Table 1 lists specifications for the sensors. All sensors were connected mechanically by a sensor bracket, which was designed in CAD software and fabricated by using a 3D printer. All CAD designs created for this implementation were open-sourced and can be downloaded from our GitHub (https://github.com/UGA-BSAIL/UAS_sensor_mount, accessed on 18 June 2021).

Sensor brackets (blue part in Figure 2) were printed in ABS plastic using a 3D printer (uPrint SE Plus, Stratasys, Eden Prairie, MN, USA). Connection brackets (green parts in Figure 2) are used to connect the DAS to the UAV and were printed in carbon fiber by using another 3D printer (Markforged Onyx Pro, Markforged, Watertown, MA, USA) to enhance the strength of connection. Connection brackets have a slide slot which allows easy attachment and detachment of the DAS to and from the UAV. The DAS also can be carried by other rotatory UAVs with adequate payload to carry the DAS. Connection brackets may need to be redesigned to fit the specific drone.

Each camera is mounted to a bracket; thus, the camera can be individually detached from the DAS and used as an independent system. This flexibility benefits multiple use cases, including when only images are needed or only low-payload UAVs are available. Cameras can be assembled to work horizontally or vertically (Figure 3). Frontward facing cameras can benefit applications that require a side-looking view of plants, such as imaging between rows of a vineyard. The vertical mount can be used to acquire nadir views of plant canopies.

2.2.1. Electronic Design

The main power (18 $\mathrm{V}$) of the DAS comes from the UAV and directly powers the single-board computer. A DC-DC converter is used to convert the main power to 12 $\mathrm{V}$ and another DC-DC to convert 12 $\mathrm{V}$ to 5 $\mathrm{V}$ (Figure 4). The LiDAR is powered at 12 $\mathrm{V}$, the thermal and multispectral cameras are powered at 5 $\mathrm{V}$, and the RGB camera is powered at 5 $\mathrm{V}$ through the USB port from the single-board computer. The DAS can be powered with a voltage source from 14 $\mathrm{V}$ to 26 $\mathrm{V}$.

The single-board manifold computer, running Ubuntu 14.04 armhf, is used to control sensors and to acquire data from the sensors (Figure 4). Controlling the RGB and thermal cameras is achieved through USB and images are transmitted through USB. LiDAR data are transmitted to the single-board computer through Ethernet. Since the multispectral camera is an independent system that has its own image processing and storing capabilities, we only implemented a trigger function through a GPIO pin to synchronize collection with other cameras. Complete control of the multispectral camera is also possible by using the serial and Ethernet interfaces on the camera. The single-board computer can interface with the UAV through the Universal Asynchronous Receiver/Transmitter (UART) to obtain real-time flight data (e.g., position and posing of the vehicle). However, additional positioning hardware (GNSS and IMU) is needed for data post processing if the design incorporates a UAV platform that cannot output its flight data. The High-Definition Multimedia Interface (HDMI) port of the single-board computer is connected to the HDMI transmitter on the UAV so that the user can see the Ubuntu desktop in real-time.

The FLIR Tau 2 camera originally only provided a 50-pin interface that includes a parallel digital port to output digital data. We designed a circuit based on the Cypress EZ-USB chip (CY7C68013A, Cypress Semiconductor, San Jose, CA, USA) to convert the parallel digital output to a USB protocol so that it could interface with the single-board computer through a USB port. This circuit makes it possible to acquire raw digital radiometric data from the camera without using a frame grabber. In order to support the calibration of the thermal camera, ground data including atmospheric conditions (air temperature, humidity, and air pressure) and temperature of the thermal calibration targets were recorded by using a customized device. The ground data are transmitted to the single-board computer in real time through xBee (xBee Pro S1, Digi International, Hopkins, MA, USA).

2.2.2. Software Design

The data acquisition software was implemented by using the Robot Operating System (ROS) release indigo running on the single-board computer (Figure 5). Implemented nodes include the following: flir_tau2 (retrieves raw digital images from the thermal camera), rededge (triggers the multispectral camera), rgb (captures RGB images), and velodyne (obtains LiDAR data). All cameras are synchronized through the trigger topic. We use the ROS package dji_sdk provided by DJI to obtain UAV location and pose. The weather_sensor node is used to receive data from the sensor on the station and to publish atmospheric data as a ROS topic. Data recordings are conducted by the rosbag node. The rosbag node saves the sensor data, drone flight data, and weather data to a bag file.

Three data visualization nodes, thermal_display, rgb_display, and pcl_visualization, were implemented to visualize the RGB image, thermal image, and the LiDAR data. Data visualization nodes create a window on the Ubuntu desktop, allowing users to display and monitor sensor data in real time through radio transmission. In addition, users can check ROS topics on a ground computer, if the computer and the single-board computer flying on the UAV are connected to the same wireless network.

3. Camera Calibration

3.1. Geometric Calibration

Geometric calibration included the estimation of geometric parameters of cameras, which were used to correct for lens distortion. We used the Camera Calibrator app from MATLAB (MATLAB 2018, MathWorks, Natick, MA, USA) to find calibration parameters for each camera, including the parameters for the camera’s intrinsic matrix, radial distortion model (Equation (1)), and tangential distortion model (Equation (2)) [36]:

$$\left[\begin{array}{c}{x}_{distorted}\\ y_{distorted}\end{array}\right]=(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)\left[\begin{array}{c}x\\ y\end{array}\right]$$

(1)

$$\left[\begin{array}{c}{x}_{distorted}\\ y_{distorted}\end{array}\right]=\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}2p_{1}xy+p_2(r^2+2x^2)\\ p_1(r^2+2y^2)+2P_{2}xy\end{array}\right]$$

(2)

where $r=\sqrt{{(x-x_p)}^2+{(y-y_p)}^2}$ is the distance between the pixel $(x,y)$ to the principal point $(x_p,y_p)$, $k_1,k_2,k_3$ are radial distortion coefficients of the lens, and $p_1,p_2$ are tangential distortion coefficients of the lens.

In order to calibrate the RGB and multispectral cameras we used a chessboard pattern printed on white paper. Two tungsten halogen lamps were used for the multispectral camera to provide additional light sources for the NIR band. For the thermal camera, a chessboard pattern was printed using a poster printer. The difference in emissivity between the ink and the paper was used to create the pattern when collecting thermal images, and four heat lamps were used to heat the printed chessboard in order to enhance the definition of the pattern. The grid size of the chessboard is $21\mathrm{mm}\times 21\mathrm{mm}$ for the RGB and multispectral cameras and $110\mathrm{mm}\times 110\mathrm{mm}$ for the thermal camera. A larger chessboard pattern was used for the thermal camera because that camera needs to collect images at a larger distance for images to be in focus. Each camera was used to collect ten images from different angles.

3.2. Vignetting Calibration

Vignetting is an effect of the radial falloff of intensity from the center of the image, including natural vignetting, pixel vignetting, optical vignetting, and mechanical vignetting [37]. There are two methods to model the vignetting effect. The first method models vignetting ($V\left(x,y\right)$) as a high order polynomial (Equation (3)) and assumes zero vignetting effect ($V(x_v,y_v)=1$) at the center of vignetting $(x_v,y_v)$:

$$V(x,y)=V\left(r\right)=1+{\alpha }_{1}r+{\alpha }_2{r}^2+{\alpha }_3{r}^3+{\alpha }_4{r}^4+{\alpha }_5{r}^5+{\alpha }_6{r}^6$$

(3)

$$r=\sqrt{{(x-x_v)}^2+{(y-y_v)}^2}$$

(4)

where r is the distance of the pixel from the center of vignetting, and ${\alpha }_1$ to ${\alpha }_6$ are the model parameters.

The second method models the vignetting effect as a look up table that stores the correction coefficients for each pixel. The second method can provide more accurate results, but it requires multiple tables for different camera settings. Since the first method is simple to implement and has been considered effective, it was chosen to model the vignetting of the multispectral camera.

In order to calculate model parameters, we measured the vignetting effect by imaging a scene with uniform luminance. For the multispectral camera, we used an integrating sphere to create such a scene. The setup of the integrating sphere is similar to [38], but we added a Teflon sheet to cover the exit port so the light could be more evenly distributed after passing the Teflon sheet. A black body was used for the same purpose for the thermal camera. We collected ten images for each camera and used the mean image to find the model parameters. The parameters ($x_v$, $y_v$ and ${\alpha }_1$ to ${\alpha }_6$) were estimated using least squares fitting. The value of $V(x,y)$ for fitting the model is the raw pixel intensity subtracting the black level and normalized against the pixel intensity at the center of vignetting.

4. Data Processing

4.1. Overall Pipeline

The data processing pipeline includes data preprocessing and phenotypic traits extraction (Figure 6). Data preprocessing calibrates images, generates georeferenced orthomosaics, and creates 3D models. Several phenotypic traits can be extracted using the preprocessed data, including canopy morphological traits, canopy vegetation index, and canopy temperature. Data preprocessing can partially be performed by the computer onboard the UAV in real time, which helps with field decisions and coordination with personnel and ground robots. We implemented software for online image geometric correction for the thermal and RGB images. We used Metashape (Metashape Professional 1.6.4, Agisoft, St. Petersburg, Russia) and Python to process images offline.

4.2. Data Preprocessing

4.2.1. Vignetting Correction

Vignetting correction was performed using the vignetting model obtained from the vignetting calibration. The corrected image intensity ($I_{corrected}$) is calculated using Equation (5) from the original image intensity (I), the sensor black-level (BL), and the vignetting correction factor (V) [37]. For the multispectral image, the vignetting correction is integrated as a part of the radiometric calibration.

$$I_{corrected}(x,y)=\frac{I(x,y)-BL(x,y)}{V(x,y)}$$

(5)

4.2.2. Geometric Correction

The geometric correction was performed using the coefficients of the lens distortion model obtained from the geometric calibration. For the RGB and the thermal cameras, we used the image_geometry package coded in the ROS nodes. The calibrated images were published as rgb/rect_image and flir_tau2/rect_image topics.

4.2.3. Multispectral Image Calibration

Raw multispectral images were processed to compute reflectance of ground objects. Processing included three steps: radiometric calibration, sunlight intensity correction, and reflectance calculation. Radiometric calibration was used to convert raw digital values of the multispectral image to absolute spectral radiances. The radiometric calibration model provided by MicaSense (https://support.micasense.com/hc/en-us/articles/115000351194-RedEdge-Camera-Radiometric-Calibration-Model, accessed on 1 October 2020) was used to compensate for sensor black level, the sensitivity of the sensor, sensor gain and exposure settings, and optical vignetting effects (Equation (6)):

$$L(x,y)=\frac{1}{V(x,y)}·\frac{a_1}{g}·\frac{p(x,y)-p_{BL}}{t_e+a_{2}y-a_3{t}_{e}y}$$

(6)

where L is the absolute spectral radiance value in $\mathrm{W}{\mathrm{m}}^{-2}{\mathrm{sr}}^{-1}{\mathrm{nm}}^{-1}$, V is the correction factor from the vignetting model, p is the normalized raw pixel value, $p_{BL}$ is the normalized black level value, $a_1$ to $a_3$ are the radiometric calibration coefficients that can be obtained from the image metadata, $t_e$ is the image exposure time, g is the sensor gain setting, y is the row number, and $(x,y)$ is the pixel position.

After radiometric calibration, radiance values for each multispectral image were corrected against incident sunlight; thus, radiance values from different images represent the same illumination conditions. The intensity of the sunlight was measured using the downwelling light sensor provided by MicaSense.

In order to account for changes in illumination during flights, a calibrated reflectance panel (MicaSense, Seattle, WA, USA) was imaged before and after each flight. Average values of radiance for pixels collected over the panel area were then linearly interpolated over time to compute a reflectance calibration factor for each spectral band. In order to compute reflectance bands, radiometrically calibrated bands representing radiance values were multiplied by reflectance calibration factors. Following the implementation of image calibration procedures in Metashape, we directly used Metashape’s reflectance calibration function to compute reflectance.

4.2.4. Thermal Image Calibration

The accuracy of temperature values derived from thermal images largely depends on how much environmental effects during acquisition can be compensated using image processing. In particular, environmental effects can be significant when objects are far from the camera and when atmospheric contributions increase.

According to the Stefan–Boltzmann law, the total radiation emitted by the object is expressed as Equation (7):

$$B\left(T\right)=\epsilon ·\sigma ·T^4$$

(7)

where $\epsilon$ is the emissivity, and $\sigma$ is the Stefan–Boltzmann constant ($5.67\times {10}^{-8}$ $\mathrm{W}$${\mathrm{m}}^{-2}$${\mathrm{K}}^{-4}$). T is the absolute temperature (K).

The total radiation received by the thermal camera ($W_{sensor}$) consists of emission of the object ($E_{obj}$), the emission of the surroundings and reflected by the object ($E_{refl}$), and the emission of the atmosphere ($E_{atm}$) (Equation (8)) [39].

$$W_{sensor}=E_{obj}+E_{refl}+E_{atm}$$

(8)

$W_{sensor}$ can be expressed as follows:

$$W_{sensor}=\sigma ·T_{sensor}^4$$

(9)

where $T_{sensor}$ is the apparent temperature provided by the thermal camera by setting the emissivity of the object to 1 and distance to 0.

Since the surface radiance is received partially by the camera and some is absorbed by the atmosphere, $E_{obj}$ can be expressed as Equation (10) as a function of the transmittance of the atmosphere ($\tau$) and the object’s temperature ($T_{obj}$).

$$E_{obj}=\tau ·\epsilon ·\sigma ·T_{obj}^4$$

(10)

$E_{refl}$ is the reflected radiation of the surroundings (which is the downwelling atmosphere radiation), and it is partly absorbed by the atmosphere. Thus, $E_{refl}$ can be expressed as the following equation using the reflected apparent temperature ($T_{refl}$),

$$E_{refl}=\tau ·(1-\epsilon )·\sigma ·T_{refl}^4$$

(11)

$E_{atm}$ is the emission of the atmosphere that reaches the thermal camera, which is the upwelling atmosphere radiation. It can be expressed as the following equation:

$$E_{atm}={\epsilon }_{atm}·\sigma ·T_{atm}^4=(1-\tau )·\sigma ·T_{atm}^4$$

(12)

where $T_{atm}$ is the temperature of the atmosphere (air temperature), and $(1-\tau )$ is the emissivity of the atmosphere.

Equation (13) can be used to retrieve the temperature of the object and results from combining Equations (8)–(12).

$$T_{obj}=\sqrt[4]{\frac{T_{sensor}^4-\tau ·(1-\epsilon )·T_{refl}^4-(1-\tau )·T_{atm}^4}{\tau ·\epsilon }}$$

(13)

In order to solve Equation (13), the transmittance of the atmosphere, the emissivity of the object, the air temperature, and the reflected apparent temperature need to be supplied. The air temperature can be measured by a temperature sensor. The reflected apparent temperature can be indirectly measured by measuring the apparent temperature of an aluminum plate or directly measured by measuring the sky’s apparent temperature using an infrared thermometer.

The transmittance of the atmosphere ($\tau$) is estimated using the water vapor content (WVC) and flight height above the ground (distance) with the following equation [40]:

$$\tau =K_{atm}·exp\left[-\sqrt{d}\left({\alpha }_1+{\beta }_1\sqrt{\omega }\right)\right]+(1-K_{atm})·exp\left[-\sqrt{d}\left({\alpha }_2+{\beta }_2\sqrt{\omega }\right)\right]$$

(14)

where $\omega$ is the water vapor content (mm), d is the distance (m), and $K_{atm}=1.9$ is the scale factor of atmospheric damping. ${\alpha }_1=0.0066$ and ${\alpha }_2=0.00126$ are the attenuation of the atmosphere without water vapor, and ${\beta }_1=-0.0023$ and ${\beta }_2=-0.0067$ are the attenuation of water vapor. The WVC can be estimated using the air temperature and humidity using the following equation [40]:

$$\omega ={\omega }_{\%}·exp\left(h_1·T_a^3+h_2·T_a^2+h_3·T_a+h_4\right)$$

(15)

where ${\omega }_{\%}$ is the relative humidity (ranging from 0 to 1; dimensionless), $T_a$ is the air temperature (°C), $h_1=6.8455\times {10}^{-7}$, $h_2=-2.7816\times {10}^{-4}$, $h_3=6.939\times {10}^{-2}$, and $h_4=1.558$ [40].

It has been reported that the emissivity of vegetation ranges from 0.96 to 0.99, while the emissivity of soils is around 0.98 [41,42]. The inaccuracy in emissivity can result in temperature errors of up to several degrees Celsius [43], and in this study we estimated emissivity of vegetation canopy and soil using the Normalized Difference Vegetation Index (NDVI), as proposed by [44].

The contribution of $E_{refl}$ and $E_{atm}$ to $W_{sensor}$ is different under different atmospheric conditions. Equations (14) and (15) show that higher air temperatures, humidity, and flight altitude result in lower transmittance of the atmosphere, which makes $E_{atm}$ contribute more to $W_{obj}$. In other words, the error of $T_{atm}$ contributes more to the error of $T_{obj}$ at lower atmospheric transmittance. $T_{refl}$ contributes more when the object has lower emissivity, which means that the error of $T_{refl}$ has more effect on the error of $T_{obj}$ for objects with lower emissivity. Since vegetation and soil have emissivity close to one, the effect of $E_{refl}$ is small relative to the measurement of their real temperature.

In order to assist the thermal calibration and validate temperature estimates, we made two thermal calibration targets and a customized device to record atmospheric conditions, including air temperature, air pressure, and humidity (Figure 7). Similar to other studies, the calibration target was made from a $0.6\mathrm{m}\times 0.6\mathrm{m}$ aluminum plate for which its top surface was painted using flat paint, and the back was covered by polystyrene insulation foam [35]. The edge of the plate was wrapped with aluminum tape to assist auto-detection of the target. One target was painted in white to create a low-temperature reference (cold target), and the other target was painted in black to create a high-temperature reference (hot target). The top surface temperature of the targets was measured by a resistance temperature detector (RTD) (700-102AAB-B00, Honeywell, NC, USA) attached to the back of the aluminum plate. Emissivity calculation used the reference temperature method [43], and emissivities were 0.968 (white paint) and 0.984 (black paint). A thermal camera pointing upward was used to measure the reflected temperature. All ground measurements can be transmitted to the DAS through xBee in real time and be used to calibrate the thermal images.

4.2.5. LiDAR Data Processing

LiDAR scans can be assembled into point clouds using the scans’ position and pose in the East North Up (ENU) frame. RTK-GNSS or PPK-GNSS and IMU should be used to measure the accurate position and posing of the platform/sensor; thus, LiDAR scans can be assembled accurately. Since the current UAS does not have an RTK-GNSS and IMU, the LiDAR data were not used in this study. However, the LiDAR data processing was integrated as part of the overall data processing pipeline and is ready for the additional positional and attitude data, for example, by using an RTK-GNSS and IMU to obtain the precise location and orientation of each LiDAR scan [45].

4.2.6. Orthomosaics and 3D Model

Agisoft Metashape and Structure from Motion (SfM) techniques were used to process RGB images collected by individual UAS flights and to generate orthomosaics, 3D point clouds, and digital surface models (DSM) for each camera.

4.2.7. Data Registration

Data registration includes the registration of the orthomosaics from different cameras and the registration of the orthomosaics and the LiDAR data. In this project, the registration of orthomosaics is based on positional information stored with those products, including coordinates, spatial resolution, and projection. The accuracy of the registration will depend on the quality of this information and should increase when high-accuracy GNSS and ground control points (GCPs) are incorporated into data acqusition and processing.

Since different orthomosaics have different ground sampling distances (GSD), pixel-wise registration between two orthomosaics requires resampling one orthomosaic to a larger or smaller GSD in order to match the orthomosaic being used as reference. If differences in spatial resolution are considerable, downsampling to a coarser GSD can involve substantial loss of information. Conversely, upsampling can introduce unrealistic representations of detail to a scene. Considering the characteristics of the cameras incorporated into our system, for pixel-wise registration, higher-resolution orthomosaics (those derived from RGB images) were downsampled to spatially match the lower-resolution thermal orthomosaics.

4.3. Extraction of Phenotypic Traits

Multiple phenotypic traits can be extracted from data captured by the integrated cameras and LiDAR sensor. Basic traits can be extracted directly following data preprocessing, including vegetation indices (VI) derived from multispectral images and canopy temperature (CT) derived from thermal images, as well as morphological traits such as canopy height (CH), canopy cover (CC), and canopy volume (CV). More complex traits, such as biomass and water stress, are out of the scope of this paper but can be obtained from the basic traits using regression models [12,21]. A program was created to extract phenotypic traits automatically by using QGIS 3.10 and Python 3. The extracted traits were saved to a csv file and visualized in QGIS.

4.3.1. Plot Segmentation

Data collected for an entire field were partitioned into individual plots using a plot layout map. Data for each plot contained 2D images and a 3D model of the plot. Since each plot can be observed by several UAV images, a plot can have multiple plot images, including images extracted from the orthomosaics. Images from different view angles can provide additional information about the plot and can, for instance, support the detection of fruits and flowers [11].

The 3D model for each plot was normalized against the ground level to create a plot height model; thus, the height of data points represented the height above ground. A 2D plane representing the ground level can be found by fitting the plot model using the Maximum Likelihood Estimation Sample Consensus (MLESAC) [46]. If the plot model does not have enough ground points due to high canopy closure, the ground plane can be represented by a bare ground model or the Digital Elevation Model (DEM) collected before plants germinated [12]. In this case, additional data collection and accurate georegistration between the bare ground model and the plot model are required. Following estimation, the ground level was subtracted from the plot model to represent height above ground.

4.3.2. Canopy Segmentation

Each plot was segmented in order to isolate and extract canopy specific metrics representing phenotypic traits. The segmentation result is a canopy mask generated from the 2D plot image or a canopy model generated from the plot height model. Color, spectral, and height differences between the canopy and ground can be used for the segmentation [20]. In this study, the thresholding method was verified to adequately segment canopies by using an RGB image and the 3D model for the plot. Other methods, such as deep neural networks and including Mask R-CNN, may offer segmentation improvements but they are out of this paper’s scope.

4.3.3. Morphological Traits

Morphological traits are important phenotypic traits that are often related to plant growth status. For example, canopy height and volume can be used to quantify cotton biomass, which is correlated with yield. This work computed two morphological traits (canopy height and canopy volume) for each plot.

Canopy Height

Canopy height was calculated using the canopy model. Strategies for canopy height computation vary and some studies use an average or median canopy height to represent the height of the crop. This approach is suitable for crops whose canopy are evenly distributed, such as barley and wheat [12,16,47]. Other studies have used a percentile of the canopy height to represent the height of the canopy, which can reduce the effect of uneven distribution of the canopy [15,48]. This study used an exploratory approach and combined both strategies to calculate canopy height using the average, maximum, median, and 50th to 99th percentiles with a 10th interval.

Canopy Volume

The canopy model was used also to compute canopy volume, represented by the volume of the mesh that encloses the 3D canopy model. The mesh can be found using a convex hull algorithm. However, since canopy models generated by aerial images or LiDAR usually lack information below the canopy surface, computing volume directly from the mesh can underestimate the canopy. Instead, we proposed to calculate the 2.5D volume as the canopy volume. The canopy model was first converted to a depth image for which its intensity represented height, and then the 2.5D volume of the depth image was calculated using Equation (16):

$$CV=\mathsf{\Delta }x\mathsf{\Delta }y\sum _i\sum _j{d}_{i,j}$$

(16)

where $\mathsf{\Delta }x$ and $\mathsf{\Delta }y$ are the ground sample distances, and $d_{i,j}$ is the depth of the pixel.

Canopy Cover

Canopy cover is defined as the ratio of the vertical projected area of the canopy to the area of the plot. After segmenting the canopy, the area of the canopy can be calculated as the ratio of the pixel area of the canopy mask to the pixel area of the plot.

4.3.4. Vegetation Indices

Multiple indices have been proposed in the literature to describe vegetation status and processes based on multispectral data [49]. In this paper, we characterized field plots by calculating one widely used vegetation indices: the Normalized Difference Vegetation Index (NDVI). For each plot, a representative vegetation index value was calculated by averaging all vegetation indices for the area covered by the canopy.

4.3.5. Canopy Temperature

Canopy temperature was calculated using values extracted from the thermal orthomosaic that correspond to each plot area. Similarly to the computation of vegetation indices, canopy temperature was calculated as the average temperature within the canopy for each individual plot.

5. Data Collection

5.1. Testing Field

Tests of the data collection platform were conducted using a cotton field located on a university research farm in Watkinsville, Georgia, United States. The field comprised of 96 plots arranged in 8 columns and 12 rows (Figure 8). The length of each plot was approximately 3 m, and there was a $1.5$ m long alley between the plots. The distance between the crop rows was $1.8$ m.

5.2. GCP Deployment and Field Mapping

The testing field was prepared prior to data collection in order to facilitate collections throughout the growing season, as well as data processing and analyses in the office. First, in order to improve the accuracy of the georeferencing of the orthomosaic, ground control points (GCPs) were deployed in the field. GCPs were ground targets with known coordinates and that had a distinct pattern that allowed for easy identification in images. Some photogrammetry software, Metashape included, provide GCP patterns that can be identified automatically by the program. In this study, we made two types of GCPs: (a) GCP in black and white for RGB and multispectral cameras and (b) GCP for thermal camera. For the RGB and multispectral cameras, we used 12-bit circular coded targets generated by Metashape as GCP patterns (Figure 9). The patterns were made of black vinyl sheet and pasted onto a white acrylic panel. The size of the acrylic panel was $0.6\mathrm{m}\times 0.6\mathrm{m}$, and the center point radius of the circular coded target was 30 mm. GCPs for the thermal camera were made of $0.8\mathrm{m}\times 0.8\mathrm{m}$ wood panels. The panels were evenly divided into four quadrants with two opposite quadrants painted in black and the other two covered by aluminum tape (Figure 9). Due to the distinct emissivities of the black paint and aluminum, GCPs for the thermal camera could be easily identified in thermal images. For our test field, we used nine GCPs for the RGB and multispectral collections, and the GCPs were fixed in the field throughout the growing season. The GCPs were raised above the ground to avoid being covered by the weeds and plants. Coordinates for those GCPs were measured periodically over the season using an RTK-GNSS to account for unintended movement of GCPs by farm operations during the season. For the thermal collections, we used six GCPs, which were deployed in the field during UAV operations as the GCPs were not weather proof. Coordinates for the thermal GCPs were obtained from the georeferenced RGB orthomosaic.

In addition to GCP deployment, we conducted a flight prior to plant emergence to map the testing field. Pre-collection mapping of a field site was a key step of our workflow and supports data processing by the following: (a) producing a plot layout map used for plot segmentation and (b) allowing for the generation of a bare ground model. Field mapping was performed one week after planting.

5.3. Flight Path Planning

Most UAVs are equipped with autopilot and can fly planned flight paths. To plan our flights, we paid particular attention to flight height above ground and percentage of image overlap. Flight height affects the ground resolution of aerial images and the size of image footprints. Image overlap (forward and side overlap) affects the quality of image stitching and 3D reconstruction, being a function of flight height, flight speed, image recording frequency, and flight path configuration. Within the same flight path, lower flight height, faster flight speed, and/or lower image recording frequency results in lower forward overlap. Conversely, higher forward overlap can be obtained by increasing flight height, by reducing flight speed, and/or by increasing image recording frequency. Side overlap can be increased by increasing flight height and/or by reducing the distance between flight lines. In addition, image overlap can be increased by using a cross-stitch flight pattern [50].

Lower flight height and higher image overlap can effectively increase image resolution and the quality of image stitching and 3D reconstruction. On the other hand, these settings increase flight time and result in a larger number of images being collected, which increases data processing time. Considering the above, we proposed a general multi-step decision-making process to balance flight height, flight speed, and image overlap. First, we selected the appropriate flight height based on the desired aerial image resolution. Second, we define the side overlap percentage. One study suggested that side overlap should be at least 67% to obtain complete 3D reconstruction for forest [51]. Based on our experience, a good range of side overlap is 70% to 80% considering flight time, area coverage, and processing time. Following the definition of flight height and side overlap, a flight path can be generated. The final step is to decide the forward overlap and flight speed. Higher forward overlap can result in more complete and better quality 3D reconstructions [51]. We suggest a forward overlap from 80% to 90%. Flight speed is defined considering that the camera should not move more than half the Ground Sampling Distance (GSD) during exposure to avoid motion blur. As a result, flight speed should be smaller than $\frac{ph}{2T_{e}f}$, where p is the pixel pitch, h is the flight height, $T_e$ is the exposure time, and f is the focal length. For example, the flight speed for the RGB camera used by our system should not exceed 8 $\mathrm{m}$/$\mathrm{s}$ at 25 $\mathrm{m}$ flight height and 1 $\mathrm{m}$$\mathrm{s}$ exposure time. A higher flight speed can be used with a smaller exposure time without increasing the flight height (or GSD) and creating blurry image.

5.4. Flight Campaign

A DJI Matrice 600 equipped with the second version of the DAS was used for data collection. Pixel4Dmapper was used to generate the flight path and control the UAV. The flight path was generated considering a 25 $\mathrm{m}$ flight height above the ground, 70% side overlap, and 90% forward overlap (Figure 10). The speed of the drone was set at 1 m/s. The flight path was calculated based on the specifications of the thermal camera, considering that camera has the smallest field of view among all the cameras used. This choice resulted in higher image overlap for the RGB and multispectral cameras. Due to the DJI Manifold’s limited data saving speed, the camera triggering frequency was set to $0.5$ $\mathrm{Hz}$ to avoid data loss, although both the RGB and the thermal cameras support much higher trigger frequency.

The flight campaign was performed on 1 October 2020 at 2:30 p.m. The weather was sunny, with an average wind speed of $1.5$ m/s. The reflectance target for the multispectral camera was imaged before and after the flight. The thermal calibration targets were mounted on a mobile robot that moved along the west border of the field so that the thermal calibration targets could be imaged multiple times by the thermal camera during the flight (Figure 10). The thermal calibration targets were imaged nine times during the collection.

5.5. Ground Data Collection

Ground data were manually collected for one crop row in the field simultaneously to the flight campaign and were used as reference to validate the extracted traits from images. The height of each plant was manually measured by using a ruler, and the average canopy height and maximum canopy height were then calculated for each plot as the average and maximum height of the plants within the plot. Ground measurements of NDVI used a handheld NDVI sensor (GreenSeeker, Trimble, Sunnyvale, CA, USA) and we calculated canopy NDVI for each plot as the average of four NDVI measurements conducted at different locations inside the plot. Temperature measured by the thermal calibration targets was used as ground reference for the targets’ surface temperature and was compared with the temperature measured by the thermal camera. Similarly, canopy height and canopy NDVI resulting from manual measurements were compared to corresponding values derived from aerial images. The mean absolute error (MAE) (Equation (17)) and mean relative error (MRE) (Equation (18)) were used to evaluate deviations from measurements (ground reference).

$$\mathrm{MAE}=\frac{1}{N}\sum _{i=1}^N\left|\mathrm{UAV}{\mathrm{derived}}_i-\mathrm{ground}{\mathrm{reference}}_i\right|$$

(17)

$$\mathrm{MRE}=\frac{1}{N}\sum _{i=1}^N\frac{\left|\mathrm{UAV}{\mathrm{derived}}_i-\mathrm{ground}{\mathrm{reference}}_i\right|}{\mathrm{ground}{\mathrm{reference}}_i}$$

(18)

6. Results

6.1. Camera Calibration

6.1.1. Camera Geometric Distortion

Table 2. Camera calibration parameters for the RGB, multispectral, and thermal cameras.

	RGB Camera	Multispectral Camera					Thermal Camera
		Blue	Green	Red	RedEdge	NIR
Focal length (mm)	5.004	5.470	5.513	5.469	5.477	5.499	25.099
$x_p$ (mm)	0.072	0.036	0.041	0.023	0.028	0.049	−0.049
$y_p$ (mm)	0.020	0.061	0.000	−0.045	−0.015	0.098	−0.240
Skew angle (rad)	8.26 $\times {10}^{-4}$	−1.00 $\times {10}^{-3}$	−1.87 $\times {10}^{-3}$	−7.75 $\times {10}^{-4}$	−1.14 $\times {10}^{-3}$	−5.82 $\times {10}^{-4}$	−3.12 $\times {10}^{-3}$
$K_1$	−5.42 $\times {10}^{-2}$	−9.78 $\times {10}^{-3}$	−9.85 $\times {10}^{-3}$	−2.61 $\times {10}^{-2}$	−8.10 $\times {10}^{-3}$	−1.09 $\times {10}^{-2}$	−3.27 $\times {10}^{-1}$
$K_2$	1.08 $\times {10}^{-1}$	−1.57 $\times {10}^{-1}$	−1.27 $\times {10}^{-1}$	−5.42 $\times {10}^{-2}$	−2.61 $\times {10}^{-1}$	−7.20 $\times {10}^{-2}$	1.77 $\times {10}^0$
$K_3$	−4.44 $\times {10}^{-2}$	−4.38 $\times {10}^{-1}$	−3.90 $\times {10}^{-1}$	−4.66 $\times {10}^{-1}$	1.71 $\times {10}^{-1}$	−8.09 $\times {10}^{-1}$	−3.61 $\times {10}^1$
$P_1$	−1.34 $\times {10}^{-3}$	2.08 $\times {10}^{-3}$	5.72 $\times {10}^{-4}$	−1.55 $\times {10}^{-3}$	−2.10 $\times {10}^{-3}$	6.55 $\times {10}^{-5}$	8.50 $\times {10}^{-3}$
$P_2$	1.25 $\times {10}^{-3}$	5.86 $\times {10}^{-5}$	2.76 $\times {10}^{-3}$	1.95 $\times {10}^{-3}$	7.78 $\times {10}^{-4}$	1.64 $\times {10}^{-3}$	6.73 $\times {10}^{-4}$

shows the cameras’ intrinsic parameters and the parameters of the distortion models. The multispectral camera’s nominal focal length is $5.4$ mm, but the focal lengths over the five bands ranged from $5.469$ mm in the red band to $5.513$ mm in the green band. The RGB camera’s and the thermal camera’s focal lengths are very close to their nominal focal length (5 mm and 25 mm, respectively). For the multispectral camera, the NIR band has the largest offset of the principal point from the origin (central pixel). The thermal camera’s principal point also has a large offset from the origin in the y-axis. The total distortion (combination of radial and tangential distortions) of the three cameras is shown by Figure 11 and Figure 12. The three cameras have low distortion, and the distortion is only noticeable near the border of the image.

In practice, camera distortion optimization is usually part of the bulk bundle adjustment of the photogrammetry process for image stitching and 3D reconstruction. Thus, it is unnecessary to perform the correction separately. However, the model parameters in Table 2 can be used as initial values for the camera distortion optimization in the photogrammetry process.

6.1.2. Camera Vignetting

Figure 13 shows vignetting for images collected by the multispectral camera, where image centers are brighter than borders. Measured and modeled vignette show that each image band is affected differently by vignetting, as a function of their individual lens system and aperture. The correction of vignetting is, thus, conducted on a per-band basis, and the results of vignette correction are also shown by Figure 13.

The vignetting pattern for the thermal camera was comparable to the pattern observed for the multispectral camera (Figure 14). However, unlike the multispectral camera, for which vignetting mainly results from optical properties of the lens, the vignetting of an uncooled thermal camera can be caused by differences in response to irradiance by each detector element, which depends on ambient temperature and the temperature of the detector. To address vignetting of thermal images, multiple vignetting correction models need to be created for different ambient temperatures. Effects of vignetting on temperature estimates for pixels near the edge of images can be significant, as temperature for those pixels can be offset by up to 3 °C [52]. In this study, we did not perform vignetting correction for the thermal camera. Instead, we discarded pixels close to edges and measured temperature by using only the central portion of images (within a distance of 100 pixels from the optical center). A more thorough study addressing corrections of vignetting effect for the thermal camera is planned for future work.

6.2. Data Preprocessing

6.2.1. Accuracy of the Thermal Camera Calibration

During the data collection flight, the air temperature and humidity varied from $28.3$ °C to $31.7$ °C and from 35.7% to 40.5%, respectively (Figure 15). Conditions resulted in high transmittance of the atmosphere ($mean=0.9249$, $standarddeviation=0.0011$) and resulted in little impact of upwelling atmosphere radiation on thermal images. After image calibration, temperatures of the two calibration targets derived from thermal images were highly correlated with temperatures measured by the RTD sensor, and pre-calibration MAE ( $6.62$ °C) was significantly reduced to $1.02$ °C (Figure 16).

6.2.2. Results of Orthomosaic Generation

We used GCPs and image locations provided by the UAV as input to Metashape to successfully generate georeferenced orthomosaics for RGB, multispectral, and thermal images (Figure 17). In order to ensure adequate spatial resolution and to facilitate overlay and comparisons, the GSDs of processing outputs were set to $0.016$ m for all cameras. The RGB camera had the largest FOV, followed by the multispectral camera and the thermal camera. Resulting from the larger FOV of the RGB camera, the RGB orthomosaic had the largest ground coverage and included areas outside the cotton field. When preparing flights, the FOV of the RGB camera can be reduced by increasing the camera’s focal length. This results in reduced GSD without losing the coverage of the field.

6.3. Phenotypic Traits Extraction

6.3.1. Results of the Canopy Segmentation

The performance of canopy segmentation is key to the extraction of phenotypic traits that depend on the number, size, and location of segments in order to calculate plant and ground cover metrics. Considering the significant differences in color between canopies and soil at the field site, a simple thresholding method was able to segment canopies and to create canopy masks efficiently (Figure 18). For crops with a large canopy, such as cotton, the plot height model can be used to refine results and to remove weeds that are lower than a threshold ($0.2$ m for this study).

6.3.2. Accuracy of the Canopy Height

Canopy height values derived from RGB images were highly correlated with manual measurements. The MAE of maximum canopy height derived from images was $0.1$ m and the majority of UAV-based measurements were lower than manual measurements. This is consistent with other studies in which the error associated with canopy height estimates based on photogrammetric methods is within the range of $0.07$ m to $0.1$ m (Figure 19) [15,16,20]. Errors linked to canopy height estimates largely depend on the accuracy of the DSM, which is affected by the spatial resolution of images. Incorrect detection of the ground plane can also result in large errors when generating the plot height model and can usually occur under close canopy and little ground exposure. In this case, the bare ground model should be used to define the ground plane. Other environmental factors, such as plant movement because of wind, also contribute to increased error.

The 99th percentile canopy height had the smallest MAE compared with the average of manually measured canopy heights (

Table 3. Comparisons between canopy height derived from images and average canopy height manually measured in the field. ${\mathrm{H}}_{\mathrm{average}}$: average canopy height. ${\mathrm{H}}_{\mathrm{median}}$: median canopy height. ${\mathrm{H}}_{\max }$: maximum canopy height. ${\mathrm{H}}_{\mathrm{np}}$: n-th percentile canopy height MAE: mean absolute error. ${\mathrm{R}}^2$: coefficient of determination using linear model.

	${\mathbf{H}}_{\mathbf{average}}$	${\mathbf{H}}_{\mathbf{median}}$	${\mathbf{H}}_{\mathbf{max}}$	${\mathbf{H}}_{50\mathbf{p}}$	${\mathbf{H}}_{60\mathbf{p}}$	${\mathbf{H}}_{70\mathbf{p}}$	${\mathbf{H}}_{80\mathbf{p}}$	${\mathbf{H}}_{90\mathbf{p}}$	${\mathbf{H}}_{99\mathbf{p}}$
MAE (m)	0.501	0.531	0.100	0.531	0.462	0.395	0.308	0.196	0.060
${\mathrm{R}}^2$	0.742	0.799	0.691	0.799	0.651	0.543	0.481	0.619	0.766

), but the median canopy height had the largest coefficient of determination. It is hard to conclude which statistical metric can better represent the true plot-level canopy height with such a small validation dataset. The choice of metric can be dependent on the type of crop being analyzed and should be determined experimentally by collecting validation data in the field.

6.3.3. Accuracy of the Canopy Vegetation Index

The image-extracted canopy NDVI had an MAE of 0.0518 with an MRA of 6.6% compared with the ground-measured NDVI. Results show that our workflow can derive accurate computations of this vegetation index. Other studies have confirmed that the MicaSense RedEdge camera was able to provide accurate NDVI measurements when compared to the GreenSeeker sensor [53]. It is noteworthy that multispectral images can reveal NDVI variations within the canopy, which can be an advantage over point-based sensors with limited FOVs.

6.3.4. Data Visualization

The extracted phenotypic traits were visualized in QGIS (Figure 20), where polygons delineate individual plots, and the shade of a color represents a range of values for a given metric or trait. For instance, those plots with lower vegetation correspond to lower NDVI and NDRE and higher canopy temperature. The visualization is helpful for researchers to examine the spatial distribution of the data and to present the results to the general audience. It also supports interplot comparisons and analyses with other co-located data, such as those associated with irrigation or use of fertilizers.

7. Discussion

UAVs have been used widely for plant phenotyping, and the literature shows a variety of platforms being used for the calculation of biophysical descriptors based on RGB, thermal, and multispectral cameras. In general, a multi-sensor system is able to collect more feature-rich data and derive more complex phenotypic traits than a single sensor. For example, cotton yield can be estimated more accurately using multiple features extracted from RGB, multispectral, and thermal images than a single image feature extracted from a single camera [54]. Different types of sensors can provide complementary information to improve the accuracy of phenotypic trait extraction. For example, high-resolution RGB images can support more detailed segmentation of canopies and can more accurately guide the calculation of canopy temperature from low-resolution thermal images. The downside of multi-sensor systems is the relatively higher payload requirement when UAVs need to carry multiple sensors simultaneously. Compared to existing UAV systems, our system was designed to allow for system customization, considering the need for flexible payload configurations based on study requirements and potential budget limitations. Each sensor can be attached to or detached from the UAV system as needed. Our system is not tied to specific crops and can be used for different fields including row crops and vineyard, with no specific requirement for the field layout (e.g., row spacing). For example, the UAS can be used to collect multispectral image to detect tomato spot wilt disease in peanuts and color images to measure the blueberry bush morphological traits [19,55]. Other potential applications include forestry and environment monitoring. In order to ensure good image quality, it is preferred to collect images under uniform lighting conditions, such as a clear sunny or overcast day. It is also preferred to collect data with no strong wind since the movement of the crops can affect the accuracy of the 3D reconstruction from images and LiDAR scans.

We also provided a complete data processing pipeline to calibrate multispectral and thermal images, to generate registered orthomosaics and 3D models, and to extract basic phenotypic traits. The basic phenotypic traits can be used to support the computation of more complex phenotypic traits. For example, morphological traits can be used to monitor crop growth and estimate the yield. Vegetation indices and canopy temperature can be used to evaluate the water status of plants. The data processing pipeline used to extract phenotypic traits was implemented in open-source software Python and QGIS, which allows for easy integration of other methods and data visualization. The data processing pipeline can be improved by replacing the commercial software Metashape with open-source software, such as OpenSfM (www.opensfm.org, accessed on 1 March 2021), to render it more streamlined.

8. Conclusions

This paper presented a design of a multi-sensor unmanned aerial systems with ROS that can be used for plant phenotyping and precision crop management. Both laboratory and field calibration methods were developed and evaluated for the RGB, multispectral, and thermal cameras with satisfactory results. A data processing pipeline was presented to derive plot-level phenotypic traits from calibrated images, and it achieved a comparable performance with other similar studies. The design of the UAS was open-sourced and can be used by other researchers. We will continue to improve our implementation by adding an RTK-GNSS and IMU to the system so that LiDAR scans can be correctly registered. A data management system that integrates data storing, data processing, and data visualization is also planned for future work.