From Do-It-Yourself Design to Discovery: A Comprehensive Approach to Hyperspectral Imaging from Drones

Abstract

This paper presents an innovative, holistic, and comprehensive approach to drone-based imaging spectroscopy based on a small, cost-effective, and lightweight Unmanned Aerial Vehicle (UAV) payload intended for remote sensing applications. The payload comprises a push-broom imaging spectrometer built in-house with readily available Commercial Off-The-Shelf (COTS) components. This approach encompasses the entire process related to drone-based imaging spectroscopy, ranging from payload design, field operation, and data processing to the extraction of scientific data products from the collected data. This work focuses on generating directly georeferenced imaging spectroscopy datacubes using a Do-It-Yourself (DIY) imaging spectrometer, which is based on COTS components and freely available software and methods. The goal is to generate a remote sensing reflectance datacube that is suitable for retrieving chlorophyll-A (Chl-A) distributions as well as other properties of the ocean spectra. Direct georeferencing accuracy is determined by comparing landmarks in the directly georeferenced datacube to their true location. The quality of the remote sensing reflectance datacube is investigated by comparing the Chl-A distribution on various days with in situ measurements and satellite data products.

1. Introduction

Remote sensing, particularly the field of imaging spectroscopy, has evolved into an indispensable instrument for many users, with its applications steadily increasing. Agriculture, mining, fisheries, archeology, urban planning, environmental monitoring, defense, and security are just a few of the industries that have already embraced this fast-evolving technology [1].

Remote sensing from space often lacks the desired resolution and flexibility needed by the end user. Manned airborne platforms, while offering higher resolution, are expensive and subject to many regulations and restrictions. Drone-based remote sensing, on the other hand, offers high resolution comparable to that achieved by manned airborne platforms, in combination with higher versatility, efficiency, and cost-effectiveness that cannot be matched by satellites or traditional aircrafts [2,3].

Smaller and more powerful electronic components enable the miniaturization of sensors and Unmanned Aerial Vehicles (UAVs) [4], and hence push the boundaries for what is possible with these devices. A trend towards the use of UAVs for remote sensing applications, as well as new use cases for drone-based remote sensing applications, can therefore be observed [2]. This trend is not limited to airborne remote sensing platforms but reflects a general shift towards the use of robotics in field research [5].

1.1. Remote Sensing

The primary objective of the UAV mission presented in this work was to provide a conceptual proof-of-concept and test the novel miniaturized imaging spectroscopy payload and the data processing pipeline presented in this paper. This payload and proposed data processing approach require minimal ground support and allow for the retrieval of marine science data products related to habitat maps, ocean color, phytoplankton, and Chlorophyll-A (Chl-A) concentrations. The payload used for this has been designed and built in-house, including the push-broom imaging spectrometer [6]. The main scientific contribution of this paper is the experimental validation of this payload for oceanic research applications. More details about the payload can be found in Section 2 and in [7].

1.2. Contribution—Holistic Imaging Spectrometer Payload and Image Processing Pipeline

The high spectral and spatial resolution offered by push-broom imaging spectrometers [8], and the possibility to build such devices from Commercial Of-The-Shelf (COTS) components, make them particularly interesting for a wide range of applications [1]. Over the years, a multitude of miniaturized “Do-It-Yourself” (DIY) imaging spectrometer designs have been conceptualized and published [9,10,11]. Various imaging spectrometer designs were intended for air- and spaceborne platforms [6,12] and prompted the development of more complex imaging spectroscopy payloads for UAVs [13,14,15,16] and small aircrafts [17].

With a few exceptions, e.g., [18], these designs implement the push-broom imaging spectrometer architecture due to its simplicity and its high spectral resolution. While these DIY payloads are usually simpler, lighter, and more versatile than their commercial counterparts, such as [19,20,21,22], they lag behind their commercial alternatives when it comes to ruggedness, data quality, as well as tailored post-processing pipelines and operational user-friendliness. Versatile implies that it can be easily customized or modified for various applications, and it can be integrated into small UAVs.

In contrast to [23,24], where commercial imaging spectrometers were utilized in UAV payloads, the lightweight payload introduced in this work is novel in its design for economic long-range missions over rugged, coastal terrain. It uniquely features the capability for direct reflectance calculations by measuring downwelling irradiance and direct georeferencing, similar to [25], enabled by an INS/GNSS sensor suite that is precisely synchronized with the imaging spectrometer.

Sensor synchronization, particularly between the INS/GNSS sensor suite and the imaging spectrometer, is achieved using a dedicated hardware device. This device allows for synchronization with an accuracy of up to 10 $\mathrm{n}$$\mathrm{s}$, making a dedicated synchronization algorithm, e.g., as suggested in [26], redundant.

Commercial software used in remote sensing and imaging spectroscopy, such as ENVI 5.6, typically require calibrated and georeferenced datacubes as input. The processing software to obtain such a datacube is normally proprietary and comes with the procured imaging spectrometer hardware. Scientific publications related to hyperspectral data processing normally also refer to the processing and analysis of the calibrated and georeferenced hyperspectral datacube and not the low-level processing steps to obtain such a data product. Scientific publications related to obtaining a hyperspectral datacube with DIY imaging spectrometer designs and payloads are sparse. There are very few publications available that are specifically related to UAVs [13,14,15,16]. Most of the publications that do exist are related to other platforms, e.g., satellites [27,28] or focus only on a few selected processing steps. A comprehensive and complete approach to creating a calibrated and georeferenced remote sensing reflectance ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacube from raw data is missing in the scientific literature. Figure 1 represents a general approach to UAV-based imaging spectroscopy, showing the data gathering with UAV-based sensors in blue on the left, and the final product on the bottom right (green). The data Processing Levels, such as PL-L1 A/B to PL-L2 A/B in Figure 1, are rendered according to the processing levels used by ESA and NASA (data processing levels (accessed on 7 June 2024): https://www.earthdata.nasa.gov/engage/open-data-services-and-software/data-information-policy/data-levels).

The payload presented was deliberately designed without a gimbal. In this way, a weight- and energy-saving design can be achieved. The reduced mechanical complexity, however, places high demands on precise state estimation, which furthermore must be precisely synchronized to the imaging spectrometer frame captures.

The payload and methodology presented in this paper aim to bridge the gap for DIY imaging spectroscopy payloads on UAVs. This paper presents a holistic approach ranging from UAV-payload design, calibration, and georeferencing to operational considerations and finally the data product: a calibrated hyperspectral datacube. Last, various applications of the reflectance data product are presented, with a focus on the spectral angular mapping (SAM) and a non-linear spectral unmixing method to estimate Chl-A distribution over a sample area.

1.3. Paper Overview

Section 2 showcases the methodology, the overall system setup, and the specific payload designed and deployed on the field campaigns. Operational considerations and payload operation are discussed in Section 3. Section 4 and Section 5 elaborate on the data processing and calibration of the imaging spectrometer data. State estimation and georeferencing are covered in Section 6 and Section 7, respectively. Section 8 presents data analysis and information extraction from the gathered data. The paper concludes with the results and the conclusion in Section 9.

2. Payload Description

The miniaturized remote sensing payload designed for this mission is illustrated in Figure 2. The payload was first described in [7] and was first tested during the campaign described in [29]. The optical remote sensing components of the payload are a Red–Green–Blue (RGB) camera, a push-broom imaging spectrometer based on [6], and an upward-facing spectrometer. These sensors are described in Section 2.1: Optical Sensors.

An Inertial Measurement Unit (IMU) and two Global Navigation Satellite System (GNSS) receivers are used for precise state estimation and navigation. Section 2.2: Sensors for State-Estimation and Navigation elaborates further on this. These core elements are connected through various auxiliary components described in Section 2.3: Data Processing, Synchronization, and Storage.

Figure 3 shows a comprehensive layout of the payload components and their integration into the described remote sensing payload. The payload design presented in this paper is a significant improvement compared to the payload used in [13,14]. Key enhancements include the mechanically more rigid integration of the imaging spectrometer and the IMU into the payload frame (see Figure 2), which significantly reduces a major source of georeferencing uncertainty identified in [14]. Additionally, the direct integration of an RGB camera into the payload offers much greater situational awareness. The incorporation of an additional RTK-GNSS receiver further augments the precision of the state-estimation and thus enhances the accuracy of the directly georeferenced datacube. All this is complemented with a lighter and less power-hungry single Board Computer (SBC), and increased storage capacity. In the following sections, the most important components of the payload are described in more detail.

2.1. Optical Sensors

One of the most common imaging spectrometer designs is the push-broom imaging spectrometer. Their high spectral resolution makes these sensors particularly interesting for applications such as ocean color observation or Chl-A estimation [8,30]. The push-broom imaging spectrometer is the main remote sensing sensor of the presented payload. This push-broom imaging spectrometer is an extremely lightweight design weighing only 160 $\mathrm{g}$. More information on the push-broom imaging spectrometer can be found in [6]. An RGB camera complements the imaging spectrometer [31]. The RGB camera permits the generation of photomosaics or Digital Surface Maps (DSM) of the overflown area using photogrammetry. A Camera Serial Interface (CSI) connects the RGB camera with the SBC (Figure 3). A Camera Serial Interface (CSI) connects the RGB camera with the SBC (Figure 3). Due to the reasonably high resolution of 8 MP, the RGB camera offers good image quality, compensating for the large Field-Of-View ($\mathrm{FOV}$). An upward-facing spectrometer [32] is used to measure downwelling irradiance. The main purpose of this sensor is to directly calculate remote sensing reflectance as the ratio of the upwelling radiance ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)$ and the downwelling irradiance ${\mathit{E}}_{\mathrm{d}}\left(\lambda \right)$. We note that due to issues of overexposure described in [7], this sensor was not used for the data analysis presented later in this paper. The optical properties of the sensors described in this section are listed in

Table 1. Spectral ranges and resolutions.

Sensor	Used Spectral Range [$\mathbf{n}\mathbf{m}$]	Spectral Resolution (FWHM) [$\mathbf{n}\mathbf{m}$]
HSI-v4	400.0–800.0	3.6
Hamamatsu C12880MA	310.0–879.0	12.0
Light Observ. [33]	378.9–943.7	3.29

and

Table 2. Optical properties.

	Unit	Img. Spec. HSI-v4	RGB Camera
Focal length f	$\left[\mathrm{m}\mathrm{m}\right]$	16	3.47
Sensor pixel height sh	$\left[pi{x}_{\mathrm{h}}\right]$	1936	3840
Sensor pixel width sw	$\left[pi{x}_{\mathrm{w}}\right]$	1216	2160
Across track fov $\zeta$	$\left[°\right]$	10.61	160
Along track fov $\kappa$	$\left[°\right]$	0.18	160
Slit dimensions $S_{\mathrm{l}}\times S_{\mathrm{w}}$	$\left[\mathrm{m}\mathrm{m}\times \mathsf{\mu }\mathrm{m}\right]$	3 × 50	—

.

2.2. Sensors for State-Estimation and Navigation

The Safran STIM-300 [34], a tactical grade IMU, configured to output incremental velocity $\mathsf{\Delta }\mathbf{\nu }$ and incremental angles $\mathsf{\Delta }\mathit{\phi }$, is used as the primary navigation sensor. The sampling rate is configured to be $f_{\mathrm{IMU}}=1000\mathrm{Hz}$.

Specific force ${\mathit{f}}_{\mathrm{IMU}}^{\mathrm{b}}\left[\mathrm{m}/{\mathrm{s}}^2\right]$ and angular rate ${\omega }_{\mathrm{IMU}}^{\mathrm{b}}\left[\mathrm{rad} {\mathrm{s}}^{-1}\right]$ are related to the measurements as follows:

$$\begin{array}{cc}\hfill {\mathit{f}}_{\mathrm{IMU}}^{\mathrm{b}}& ={\displaystyle \frac{\mathsf{\Delta }{\mathit{\nu }}_{\mathrm{IMU}}^b}{T_{\mathrm{IMU}}}}\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill {\mathit{\omega }}_{\mathrm{IMU}}^{\mathrm{b}}& ={\displaystyle \frac{\mathsf{\Delta }{\mathit{\phi }}_{\mathrm{IMU}}^{\mathrm{b}}}{T_{\mathrm{IMU}}}}\hfill \end{array}$$

(2)

To convert the IMU measurements to the UAV’s body-fixed coordinate frame $\left\{\mathrm{b}\right\}$, a rotation matrix ${\mathit{R}}_{\mathrm{IMU}}^{\mathrm{b}}$ is used (3).

$${\mathit{R}}_{\mathrm{IMU}}^{\mathrm{b}}=\left(\begin{array}{ccc}0& 1& 0\\ -1& 0& 0\\ 0& 0& 1\end{array}\right)+{\mathit{R}}_{\mathrm{tuning}}^{\top }$$

(3)

The tuning matrix ${\mathit{R}}_{\mathrm{tuning}}$ is required to align the orientation of the IMU in the UAV and the corresponding trimmed flight attitude. The tuning matrix ${\mathit{R}}_{\mathrm{tuning}}$ is defined so that the gravity vector $g_{\mathrm{b}}^{\mathrm{e}}$ points to the center of the earth in level flight. ${\mathit{R}}_{\mathrm{tuning}}$ and the corresponding tuning values are given in Appendix A.

Two GNSS receivers [35] are used to provide absolute position measurements. The positions of the GNSS antennas on the Mini Cruiser UAV, along with other sensors are illustrated in Figure 4. The GNSS antennas are labeled as $g_1$ and $g_2$, respectively. The GNSS receiver for the GNSS antenna $g_2$ in Figure 4 is configured as a “moving baseline”, while the GNSS receiver for the GNSS antenna $g_1$ is configured as a “rover” ([36] (Ch. 10)). Both GNSS receivers were configured to output RAW-GNSS observables with a frequency of $f_{\mathrm{GNSS}}=1\mathrm{Hz}$. With additional base station data, this allows for the use of post-processing kinematics (PPK-GNSS) in the navigation state-estimation algorithm.

2.3. Data Processing, Synchronization, and Storage

The SBC, Khadas-Vim3 [37] is the core component of the onboard data processing system. It was chosen for its light weight, good computational performance, low power consumption, and its small form factor. LSTS-DUNE [38] running on the operating system Ubuntu “Server 20.04.6 LTS” is used as a runtime environment to handle all tasks related to remote sensing, navigation, remote control, and data management.

Precise timestamping of the imaging spectroscopy data as well as timestamping of the navigation sensor data has a significant impact on the direct georeferencing accuracy. Special attention is therefore given to this task. A dedicated device, the SentiBoard [39] is used to timestamp all sensor data, except for the RGB camera, to an accuracy of $\mathsf{\Delta }t=10 \mathrm{n}\mathrm{s}$. This setup allows precise synchronization of the imaging spectrometer and navigation data. The absolute time reference for the SentiBoard is provided through a Pulse Per Second (PPS) signal from one of the GNSS receivers (see Figure 3).

The data from all sensors is stored on a Solid State Drive (SSD) with a capacity of 2 $\mathrm{TB}$. An Ethernet switch allows the payload to interface with the drone’s radio link, and through it, with an operator on the ground. These and all other components are schematically shown in Figure 3. The payload operator has the ability to modify the payload settings during operation in real-time through LSTS-NEPTUS [38] and monitor the data capturing process. This allows for remote control and a flexible mission execution, with on-the-fly adjustment of remote sensing parameters as well as the UAV flight plan. Furthermore, it is a stepping stone to closer integration, interoperability, and autonomy between robotic agents.

2.4. UAV Platform

Figure 5 shows the payload integrated into the UAV platform: “ET-Air/Mini Cruiser” [40]. The UAV platform was heavily customized by NTNUs UAVlab and provides the radio downlink through which real-time communication between an operator and the payload is possible.

3. Payload Operation and Operational Considerations

Image quality for airborne push-broom imaging spectrometers depends on the interplay between speed over ground v, altitude above the imaged surface h, UAV pitch rate $\dot{\theta }$, imaging spectrometer frame rate $f_{\mathrm{IS}}=T_{\mathrm{IS}}^{-1}$ ($T_{\mathrm{IS}}$ is the frame interval), and exposure time $e_{\mathrm{IS}}$. Figure 6 shows the push-broom imaging from the side and from the back. The $\mathrm{FOV}$ along track $\kappa$ and the $\mathrm{FOV}$ across track $\zeta$ are visualized in this image.

The following equations can be found in Appendix A, and describe pixel overlap $O_{\mathrm{pix}}$ and the pixel length along track $l_{\mathrm{pix}}$ for an airborne imaging spectrometer operating with speed v, altitude h, pitch rate $\dot{\theta }$, and the imaging spectrometer settings $f_{\mathrm{IS}}$ and $e_{\mathrm{IS}}$:

$$O_{\mathrm{pix}}=h\left(tan\left({\displaystyle \frac{\kappa }{2}}\right)+tan\left({\displaystyle \frac{\kappa }{2}}-\dot{\theta }\left(f_{\mathrm{IS}}^{-1}-e_{\mathrm{IS}}\right)\right)\right)-v\left(f_{\mathrm{IS}}^{-1}-e_{\mathrm{IS}}\right)\ge 0$$

(4)

$$l_{\mathrm{pix}}=h\left(tan\left({\displaystyle \frac{\kappa }{2}}\right)+tan\left({\displaystyle \frac{\kappa }{2}}+\dot{\theta }·e_{\mathrm{IS}}\right)\right)+v·e_{\mathrm{IS}}$$

(5)

It is assumed that all parameters in these equations ($\dot{\theta }$, v and h) are constant during one imaging spectrometer frame capture and that the UAV is flying parallel to the ground. Figure 7 illustrates the relationship between the different parameters, which are described by Equations (4) and (5).

The overall image quality depends highly on these settings. To achieve optimal image quality, we propose the following approach. Given the imaging spectrometer parameters, the swath width (Figure 6) is determined by the altitude h. The aerial platform’s cruise speed and the local wind conditions determine the speed over the ground v. Given these parameters, the push-broom imaging spectrometers framerate $f_{\mathrm{IS}}$ and the exposure time $e_{\mathrm{IS}}$ ought to be chosen so that the pixel length $l_{\mathrm{pix}}$ is minimal and there is a positive overlap $O_{\mathrm{pix}}$ between consecutive pixels along the track.

While longer exposure times are usually beneficial for a better Signal-to-Noise (S/N) ratio, they also lead to elongated irregularly shaped pixels, which, depending on the aerial platform’s movements, do not necessarily remain rectangular. Long exposure times further increase motion blur in the image. Both effects cause a decrease in image quality (spatial resolution). Moreover, the inequality condition (4) must be fulfilled to avoid “gaps” between consecutive frames. There is a trade-off between high spatial resolution along the track, achieved by adopting a high frame rate $f_{\mathrm{IS}}$, or a high S/N ratio, achieved by adopting a long exposure time $e_{\mathrm{IS}}$. In addition, the exposure time is constrained by the need to avoid over-exposure.

Previous missions [14] have shown that if spatial resolution is important, a high frame rate $f_{\mathrm{IS}}$ is preferable to a long exposure time $e_{\mathrm{IS}}$, despite the lower signal-to-noise ratio. This applies in particular to maneuverable aircrafts such as small UAVs. To achieve the best S/N ratio, (e.g., when spatial resolution is less important) the exposure time should be set to the highest value. Imaging spectrometers on small maneuverable platforms provide the best results for spatially resolved datacubes when the highest achievable frame rate $f_{\mathrm{IS}}$ (considering the limitations of S/N ratio and overexposure) and the exposure time $e_{\mathrm{IS}}=f_{\mathrm{IS}}^{-1}=T_{\mathrm{IS}}$ are selected. Consequently, (4) simplifies to: $O_{\mathrm{pix}}=2·h·tan\left({\displaystyle \frac{\kappa }{2}}\right)$. With these settings the inequality condition (4) will always be fulfilled, and the pixel length $l_{\mathrm{pix}}$ will be minimal, leading to the best spatial resolution for the unprocessed image spectrometer datacube.

These settings might differ from push-broom imaging spectrometer settings on other platforms, such as satellites, aerostats, or high-altitude long endurance (HALE) platforms. Such platforms may fly using much less dynamic maneuvers and therefore allow for longer exposure times $e_{\mathrm{IS}}$. All flights that were conducted as part of the experiments described in this paper followed these considerations.

4. Processing of Payload Data

The following section elaborates on the onboard and offline data processing pipeline. A complete overview of the data processing pipeline for all sensors as it was implemented for the described UAV payload is shown in Figure 8. Much of the data processing steps shown in Figure 8 were performed offline. These processing steps are further discussed in Section 5, Section 6 and Section 7.

4.1. Processing of Navigation Sensors

The SentiBoard functions as a timestamping and data-concentration device. IMU and GNSS data (among others) are received by the SentiBoard, where the data are timestamped with a precision of $\mathsf{\Delta }t=10 \mathrm{n}\mathrm{s}$ and sent as one data stream to the SBC (see Figure 3) which saves the data on the internal SSD (Section 2.3). All subsequent post-processing of navigation data is performed offline. The time reference for the sensor-logs timestamped by the SentiBoard is provided by the GNSS time from the moving-base GNSS receiver $g_2$ shown in Figure 4. This allows the conversion of all sensor data timestamped by the SentiBoard to UNIX time.

The RAW-GNSS observables were initially processed with the software RTKLIB, version b34g [41] before being integrated into the state-estimation algorithm (Section 6). Additionally, the PPK-GNSS logs were resampled and converted to UNIX time (Section 6.1.5). Post-processing of the IMU data involves the conversions (1) and (2), and the conversion of the measurement timestamp to UNIX time.

4.2. Image Spectrometer Data Processing

The software DUNE [38] manages the thread responsible for the push-broom imaging spectrometer captures. Four key parameters need to be configured for the correct operation of this sensor. A region of interest, abbreviated as $\mathrm{w}$, must be defined on the imaging spectrometer’s complementary metal–oxide semiconductor (CMOS) sensor. This region of interest specifies an area on the sensor that is illuminated using the imaging spectrometer’s optics (see Figure 6). Excluding non-illuminated areas saves valuable processing time. The spectral dimension is binned with a binning factor of $\mathrm{bin}=8$. This is done to reduce the frame size and data rate, consequently allowing for higher frame rates. The chosen binning factor does not affect the spectral resolution for this sensor, since the binned pixel is smaller than the full-width half maximum (FWHM) of the imaging spectrometer optics. Exposure time $e_{\mathrm{IS}}$ and frame rate $f_{\mathrm{IS}}$ are the final two parameters that must be configured by the user. This can either be done pre-flight or during field operations through the software LSTS-NEPTUS [42].

During operation, the imaging spectrometers camera sensor retains the set parameters until the DUNE thread updates them. When a frame is captured, the flash output of the imaging spectrometer sensor is raised to $5 \mathrm{VDC}$ from its default state of $0 \mathrm{VDC}$, triggering a timestamp capture on the SentiBoard (Figure 3). In post-processing, this can be used to accurately determine the time at which a frame capture started. The imaging spectrometer frames and the imaging spectrometer timestamps generated by the SentiBoard are processed and stored separately on the payload.

Table 3. HSI and RGB settings for flights 1, 5, and 8.

	HSI			RGB
	Exposure Time e [ms]	fps [s−1]	HSI Startup Delay [s]	fps [s−1]
Flight 1	25.0	18.27	15.93	1.0
Flight 5	25.0	40.0	0.0	1.0
Flight 8	25.0	40.0	6.38	1.0

shows an overview of the exposure times and frame rates chosen for the push-broom imaging spectrometer and the RGB camera for the flights which are analyzed in this paper.

4.3. RGB Data Processing

The exposure time for this sensor is adjusted by the camera’s internal drivers and the frame rate is handled by a dedicated DUNE task and can be chosen by the operator. All images are timestamped with the corresponding UNIX time derived from the SBC and saved in RAW file format in 8 MP image quality on the payload’s internal SSD (Figure 3). The post-processing of the stored files is performed offline.

During field operations, every 30 $\mathrm{s}$, an RGB image is downscaled to a dimension of 468 × 432 pixels and sent to the payload operator through the UAV’s radio link connection as a JPG-file. This thumbnail image provides very useful information during field operations and allows the operator to perform real-time adjustments during the mission.

Due to varying lighting conditions and the camera’s internal drivers reacting to it, the RGB image quality varied widely within one flight. Addressing these inconsistencies during post-processing posed considerable challenges. The post-processing steps which are shown in Figure 9 resulted in the best image quality and the least color variations between consecutive image captures. Figure 10 shows an orthophoto generated with color-corrected images and the software Open Drone Map (ODM) version 3.5.3 [43].

5. Calibration

5.1. Preliminaries Push-Broom Imaging Spectrometer

The imaging spectrometers “spectral–spatial” frame, captured at timestamp $t_k\in \mathbb{R}$ with time index k, is denoted as ${\mathit{F}}_{\mathrm{IS}k}\in {\mathbb{R}}^{b\times n}$. The imaging spectrometer datacube is denoted as ${\mathit{X}}_{\mathrm{IS}}\in {\mathbb{R}}^{n\times c\times b}$. The spatial dimension n is equal to the pixels across the track, the dimension c is equal to the number of timestamps $c=k$, and the spectral dimension b is equal to the number of bands.

5.2. Imaging Spectrometer Calibration

The imaging spectrometer was spectrally and radiometrically calibrated. A smile correction algorithm was used to further improve the data quality. This section describes the calibration procedure in more detail.

5.2.1. Spectral Calibration

The imaging spectrometer bands were spectrally calibrated with a second-order polynomial (6). Each band index b gets assigned a wavelength ${\lambda }_{\mathrm{b}}$. The coefficients for (6) were determined using a calibration procedure similar to [44].

$${\lambda }_{\mathrm{b}}=a_0+a_1·b+a_2·b^2$$

(6)

An argon and a mercury lamp were used in the Ulbricht sphere for this calibration step. The calibration coefficients determined by this calibration procedure are listed in

Table 4. Coefficients for spectral calibration.

Spectral Coefficients	${\mathit{a}}_2$ $[\mathbf{n}\mathbf{m}/{\mathit{b}}^2]$	${\mathit{a}}_1$ $[\mathbf{n}\mathbf{m}/\mathit{b}]$	${\mathit{a}}_0$ $[\mathbf{n}\mathbf{m}]$
	$1.8147·{10}^{-5}$	$1.8935·{10}^{-1}$	$2.9097·{10}^2$

.

5.2.2. Radiometric Calibration

The radiometric calibration data were collected during the same calibration procedure as depicted in Figure 11. Radiometric calibration data consist of the radiometric calibration matrix ${\mathbf{F}}_{\mathrm{c}}$ and a dark current image ${\mathbf{F}}_{\mathrm{dark}}$. The radiometric calibration matrix ${\mathbf{F}}_{\mathrm{c}}$ was recorded with an exposure time of $0.04 \mathrm{s}$ in the setting shown in Figure 11. The same exposure time was used for the dark current matrix ${\mathbf{F}}_{\mathrm{dark}}$. With these calibration data, the radiometric calibration can be applied to all frames as shown below:

$${\mathit{F}}_{\mathrm{IS}k}=\left({\mathit{F}}_{\mathrm{IS}\mathrm{raw}k}-{\overline{\mathit{F}}}_{\mathrm{dark}}\right)·{\displaystyle \frac{{\mathit{F}}_{\mathrm{c}}}{e_k}}$$

(7)

where $e_k$ is the exposure time during frame capture k. The matrices ${\mathbf{F}}_{\mathrm{c}}$ and ${\mathbf{F}}_{\mathrm{dark}}$, each with dimensions of $1216\times 242$, were cropped to the region of interest w (see Figure 6) and binned with the same factor as ${\mathbf{F}}_{\mathrm{IS}k}$ ($\mathrm{bin}=8$).

5.2.3. Spectral Smile Correction

Spectral smile and keystone are common optical distortions that reduce the image quality in push-broom imaging spectrometers [8]. The imaging spectrometer used in this project suffers especially from spectral smile distortions due to its small lenses and the resulting large curvature of these lenses. Reducing spectral smile was therefore prioritized in this work. The approach used is based on the calibration procedure ([45] (Ch. 8.2.2)). The central band of the spectral calibration matrix ${\mathit{F}}_{\mathrm{c}}$ is used as a reference, while the spectra in the other rows are interpolated and resampled onto the same wavelength using basis-spline curve fitting.

5.3. Remote Sensing Reflectance and Atmospheric Corrections

Remote sensing reflectance ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ is defined as

$${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)={\displaystyle \frac{{\mathit{L}}_{\mathrm{u}\mathrm{BOA}}\left(\lambda \right)}{{\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)}}$$

(8)

where surface-leaving radiance at the Bottom Of the Atmosphere (BOA) is denoted as ${\mathit{L}}_{\mathrm{u}\mathrm{BOA}}\left(\lambda \right)$. ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)$ is the upwelling radiance measured by the imaging spectrometer, which is ${\mathit{L}}_{\mathrm{u}\mathrm{BOA}}\left(\lambda \right)$ altered by the atmosphere between the UAV and BOA. ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ is the downwelling irradiance at the observed object due to solar illumination from the sky. These measurements were used to calculate ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$.

If ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)$ is not measured at the surface level (BOA), it has to be corrected for atmospheric influence. For satellite-based imaging spectrometers, the atmosphere typically contributes 90–95% of the received signal ([30] (Ch. 15)). Since UAV-based imaging spectrometers have the major part of the atmosphere above them, the contribution of the atmosphere to the received signal is much smaller. Nevertheless, even a small effect will significantly reduce the radiometric accuracy. We therefore estimate the contribution of the atmosphere with a simple atmospheric correction algorithm.

Let ${\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right)$ be the atmospheric absorption of the entire atmosphere, expressed as a fraction related to the downwelling irradiance at the Top Of the Atmosphere (TOA) ${\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)$:

$$\begin{array}{cc}\hfill {\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right)& ={\displaystyle \frac{{\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)-{\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)}{{\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)}}\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)& ={\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right){\displaystyle \frac{p_0-p\left(h\right)}{p_0}}\hfill \end{array}$$

(10)

The absorption of a fraction of the atmosphere $\mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)$ can be calculated using the ratio of the local pressure $p\left(h\right)$ to the pressure on the ground $p_0$. This relationship is applicable because the atmospheric column above the UAV is directly proportional to the local air pressure $p\left(h\right)$ at the UAVs flight level, as expressed in (10), and because there is a linear relationship between gas concentration and light attenuation due to Beer–Lambert’s law for absorption (see: Appendix B.2).

The radiance ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)$ measured at the UAV’s altitude must be atmospherically corrected. This is done by reformulating (8) with (10), resulting in (11). The relationship (11) is more comprehensively elaborated on in [7].

$${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)={\mathit{L}}_{\mathrm{u}\mathrm{BOA}}\left(\lambda \right)·\left(1-\mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)\right)$$

(11)

$${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)={\displaystyle \frac{{\mathit{L}}_{\mathrm{u}}\left(\lambda \right){\left(1-\mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)\right)}^{-1}}{{\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)}}$$

(12)

The derivation of (12) relies on the assumption that the mean atmospheric gas ratios are similar above and below the UAV flight level. Since the mass of distinctive atmospheric layers with different gas ratios (e.g., the ozone layer) are small in comparison to the whole atmosphere, and since most of the atmosphere is above the UAV, this approach can be justified for low-flying UAVs during bright weather conditions. It can therefore be used to calculate ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ and corrected with regards to the minor atmospheric influences found at this altitude. ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)$ in (12) is derived in the next two subsections, Section 5.3.1 and Section 5.3.2.

5.3.1. Calculation of the Atmospheric Absorption

The atmospheric absorption ${\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right)$ is calculated by (9), where ${\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)$ is derived from the spectral blackbody irradiance of the sun. The spectral blackbody irradiance from the sun’s surface ${\mathit{F}}_0\left(\lambda \right)$ is calculated as shown in (13) [46].

$${\mathit{F}}_0\left(\lambda \right)={\displaystyle \frac{2·\pi ·h·c^2}{{\lambda }^5\left(exp\left(\frac{h·c}{k·\lambda ·T_{\mathrm{Sun}}}\right)\right)-1}}$$

(13)

The black body irradiance at the earth ${\mathit{F}}_{\mathrm{E}}\left(\lambda \right)$ is calculated as shown in (14):

$${\mathit{F}}_{\mathrm{E}}\left(\lambda \right)={\displaystyle \frac{R_{\mathrm{Sun}}^2}{D_{\mathrm{Sun}\mathrm{Earth}}^2}}·{\mathit{F}}_0\left(\lambda \right)$$

(14)

To calculate the irradiance ${\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)$ from ${\mathit{F}}_{\mathrm{E}}\left(\lambda \right)$, a cosine correction is done:

$${\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)={\mathit{F}}_{\mathrm{E}}\left(\lambda \right)·cos\left({\theta }_{\mathrm{Sun}}\right)$$

(15)

${\theta }_{\mathrm{Sun}}$ is the angle between the unit vector on the local tangential plane ${\mathit{n}}_{\mathit{E}}$ and unit vector pointing at the sun ${\mathit{n}}_{\mathbf{S}}$, as illustrated in Figure 12. The angle ${\theta }_{\mathrm{Sun}}$ can be determined as follows:

$${\theta }_{\mathrm{Sun}}={90}^{\circ }-{ϵ}_{\mathrm{Sun}}$$

Table 5. The sun’s angles ${ϵ}_{\mathrm{Sun}}$, ${\psi }_{\mathrm{Sun}}$ and ground pressures $p_0$.

Date/Time DD:MM:YY hh:mm	Sun Elev. ${\mathit{ϵ}}_{\mathbf{Sun}}$ [°]	Sun Azim. ${\mathit{\psi }}_{\mathbf{Sun}}$ [°]	Pressure at Sea Level ${\mathit{p}}_0$ [Pa]
22:05:22 12:00	31.22	193.98	$1013.0\times {10}^5$
22:05:22 13:11	29.87	213.24	$1013.1\times {10}^5$
26:05:22 10:19	31.214	166.270	$1009.3\times {10}^5$
26:05:22 12:00	32.125	193.899	$1009.4\times {10}^5$
27:05:22 12:00	32.126	193.899	$1016.9\times {10}^5$
27:05:22 17:30	19.94	279.52	$1019.9\times {10}^5$

shows the sun’s elevation ${ϵ}_{\mathrm{Sun}}$ and the pressure on the ground $p_0$ on the relevant days of the field campaign in Ny-Ålesund, Svalbard [47]. All constants used in these calculations are shown in

Table 6. Constants used for atmospheric correction.

Description	Symbol	Value	Unit
Sun surf. Temp	$T_{\mathrm{Sun}}$	5778	$\left[\mathrm{K}\right]$
Light speed in vacuum	c	$2.99792458·{10}^8$	$\left[\mathrm{m}/\mathrm{s}\right]$
Planks constant	h	$6.62606957·{10}^{-34}$	$\left[\mathrm{J}\hspace{0.17em}\mathrm{s}\right]$
Boltzmann constant	$k_{\mathrm{B}}$	$1.3806488·{10}^{-23}$	$\left[\mathrm{J}/\mathrm{K}\right]$
Sun diameter	$R_{\mathrm{Sun}}$	$6.957·{10}^8$	$\left[\mathrm{m}\right]$
Distance Sun Earth	$D_{\mathrm{Sun}\mathrm{Earth}}$	$149.5978707·{10}^9$	$\left[\mathrm{m}\right]$

.

5.3.2. Indirect Illumination

In our experiments, the downwelling irradiance on the ground, ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$, was measured at the Ny-Ålesund ArcLight observatory [33]. However, the downwelling irradiance ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ does not only consist of rectilinear propagating light rays from TOA to BOA. Several effects cause indirect illumination and cause higher light measurements than expected when considering only light absorption and rectilinear light propagation, see also Appendix B. The two main effects which increase downwelling irradiance ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ in this scenario are:

• Downwelling light reflected into the atmosphere by ArcLights surrounding. This reflected light is backscattered by the atmosphere into the ArcLight sensor. If ArcLight is surrounded by a surface with a high albedo, such as snow, this effect is more pronounced.
• Rayleigh and forward scattering. This effect can lead to higher ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ measurements, especially during low sun elevations [48]. In contrast to the first described effect, this also happens on the ocean’s surface.

Eliminating these effects is not in the scope of this paper, however, a simple correction can reduce their impact, as described next.

Figure 13 shows in the top row a comparison of the downwelling irradiance at 12:00 UTC+1 (red, dotted) and at 17.30 UTC+1 (blue, dotted). The plot at the top left shows ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ as measured at the ArcLight observatory. The top right plot shows ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)$ (corrected), where ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$ was reduced by a correction factor f to adjust the atmospheric absorption during the time with the highest sun position ${ϵ}_{\mathrm{Sun}}$. ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)$ can be calculated as:

$${\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)={\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)·\left(1-f\right)$$

(16)

which allows for the calculation of the atmospheric absorption $\mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)$ with (9) and (10). The lower left diagram show the absorption of the entire atmosphere calculated according to (9), ${\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right)$. The indirect effects described above play a smaller role at higher sun elevations ${ϵ}_{\mathrm{Sun}}$ and therefore also at times with high downwelling irradiance, ${\mathit{E}}_{\mathrm{d}\mathrm{TOA}}\left(\lambda \right)$ or ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}}\left(\lambda \right)$.

This approach does not eliminate the second-order effects described above but reduces them to a similar level for different lighting conditions. Therefore, a similar amount of atmosphere is removed from ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right)$ for different flights under the same atmospheric conditions. This approach is considered sufficient for low-altiude UAVs, as only part of the atmosphere needs to be removed (see Figure 12). Appendix B further justifies this approach with a simulation of the downward irradiance with RADTRANX [49]. The value $f=0.129$ was estimated using regression, minimizing the Root Mean Square Error (RMSE) between the absorption spectra ${\mathit{\alpha }}_{\mathrm{d}}\left(\lambda \right)$ for the time with the highest sun elevation and the flight time specified in Appendix A, Table 10. The regression between the two curves resulted in a $\mathrm{RMSE}=0.02$ (see bottom right diagram in Figure 13).

5.3.3. Remote Sensing Reflectance

Using (16) in (12) allows for the calculation of an atmospherically corrected ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacube. Figure 14 shows on the left side the downwelling irradiance ${\mathit{E}}_{\mathrm{d}\mathrm{BOA}\mathrm{corr}}\left(\lambda \right)$ measured (and corrected) by ArcLight (blue) and the atmospherically corrected upwelling radiance ${\mathit{L}}_{\mathrm{u}}\left(\lambda \right){\left(1-\mathsf{\Delta }\mathit{\alpha }\left(\lambda \right)\right)}^{-1}$ measured by the imaging spectrometer on the UAV over the ocean (red). The corresponding ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ is shown on the right-hand side of Figure 14. The spectrum shows a typical shape for ocean water [30].

6. State Estimation

A Multiplicative Error-state Kalman Filter (MEKF) is used to accurately estimate the UAV’s position and attitude. The results of this state estimation algorithm are used as input in the georeferencing algorithm described in Section 7. This section explains the state-estimation algorithm in detail.

6.1. State-Estimation Preliminaries

6.1.1. Coordinate Systems

Coordinate frames are denoted with $\left\{·\right\}$. ${\mathit{z}}_{\mathrm{bc}}^{\mathrm{a}}\in {\mathbb{R}}^3$ denotes a vector $\mathit{z}$, from point $\left\{\mathrm{b}\right\}$ to point $\left\{\mathrm{c}\right\}$ decomposed in coordinate frame $\left\{\mathrm{a}\right\}$.

Five different coordinate systems were used in this paper (see Figure 15 and Figure 16), [36,50]. The Earth-Centered Inertial (ECI) frame, denoted with $\left\{\mathrm{i}\right\}$. The Earth Centered Earth Fixed (ECEF) frame, denoted with $\left\{\mathrm{e}\right\}$. The North-East-Down (NED) frame, denoted with $\left\{\mathrm{n}\right\}$. These coordinate frames were used for state estimation (Section 6) and georeferencing (Section 7). Furthermore, the body-fixed frame of the UAV is denoted with $\left\{\mathrm{b}\right\}$ and the imaging spectrometer coordinate frame is denoted with $\left\{\mathrm{s}\right\}$. The imaging spectrometer coordinate frame is situated such that $z^{\mathrm{s}}$ is perpendicular to the imaging spectrometer central axis, and $y^{\mathrm{s}}$ and $x^{\mathrm{b}}$ are collinear but point in opposite directions (see Figure 4 and Figure 16).

Latitude and longitude on Earth are represented by $\mu \in \left[-\pi /2,\pi /2\right]$ and $\lambda \in \left(-\pi ,\pi \right]$.

6.1.2. Attitude Representations and Relationships

Quaternions of the Hamiltonian representation are used to describe the attitude of the UAV. The unit quaternion, representing a rotation from frame $\left\{\gamma \right\}$ to frame $\left\{\beta \right\}$, is defined as follows:

$${\mathit{q}}_{\beta }^{\gamma }=\left(\begin{array}{c}{q}_{\mathrm{s}}\\ {\mathit{q}}_{\mathrm{v}}\end{array}\right)=\left(\begin{array}{c}{q}_{\mathrm{s}}\\ q_{\mathrm{x}}\\ q_{\mathrm{y}}\\ q_{\mathrm{z}}\end{array}\right)\in \mathbb{H}$$

(17)

where $\mathbb{H}:=\left\{{\mathit{q}}_{\beta }^{\gamma }\otimes {{\mathit{q}}_{\beta }^{\gamma }}^{★}=1;q_{\mathrm{s}}\in {\mathbb{R}}^1;{\mathit{q}}_{\mathrm{v}}\in {\mathbb{R}}^3\right\}$ [50,51]. The quaternion can be used to calculate the rotation matrix ${\mathit{R}}_{\beta }^{\gamma }\in SO\left(3\right)$ [51]:

$${\mathit{R}}_{\beta }^{\gamma }=f\left({\mathit{q}}_{\beta }^{\gamma }\right)=\left(q_{\mathrm{s}}-{\mathit{q}}_{\mathrm{v}}^{\top }{\mathit{q}}_{\mathrm{v}}\right){\mathit{I}}_3+2{\mathit{q}}_{\mathrm{v}}{\mathit{q}}_{\mathrm{v}}^{\top }+2q_{\mathrm{s}}\mathit{S}\left({\mathit{q}}_{\mathrm{v}}\right)$$

(18)

The Hamilton quaternion product (⊗) is defined as in: [51].

$${\mathit{q}}_3={\mathit{q}}_1\otimes {\mathit{q}}_2=\left(\begin{array}{c}{q}_{\mathrm{s}1}{q}_{\mathrm{s}2}-{{{\mathit{q}}_{\mathrm{v}}}^{\top }}_1{\mathit{q}}_{\mathrm{v}2}\\ q_{\mathrm{s}1}{\mathit{q}}_{\mathrm{v}2}+q_{\mathrm{s}2}{\mathit{q}}_{\mathrm{v}1}+{\mathit{q}}_{\mathrm{v}1}\times {\mathit{q}}_{\mathrm{v}2}\end{array}\right)$$

(19)

The kinematic equation of a given unit quaternion is given as in: [50,51].

$$\dot{\mathit{q}}={\displaystyle \frac{1}{2}}\mathit{q}\otimes \mathit{\omega }={\displaystyle \frac{1}{2}}\mathbf{\Omega }\left(\mathit{\omega }\right)\mathit{q}$$

(20)

$$\mathbf{\Omega }\left(\mathit{\omega }\right)=\left(\begin{array}{ccc}0& & -{\mathit{\omega }}^{\top }\\ \mathit{\omega }& & -\mathit{S}\left(\mathit{\omega }\right)\end{array}\right)$$

where a skew symmetric matrix is represented with $S\left(·\right)\in SS\left(3\right)$ (e.g., $S\left({\mathit{z}}_1\right){\mathit{z}}_2={\mathit{z}}_1\times {\mathit{z}}_2$ where ${\mathit{z}}_1,{\mathit{z}}_2\in {\mathbb{R}}^3$). The conjugate of a quaternion is given as follows:

$${\mathit{q}}^{★}=\left(\begin{array}{c}{q}_{\mathrm{s}}\\ -{\mathit{q}}_{\mathrm{v}}\end{array}\right)$$

(21)

Additionally, the Euler angles are given as follows, with $\varphi$ = roll, $\theta$ = pitch, and $\psi$ = yaw:

$$\mathbf{\Theta }={\left(\varphi ,\theta ,\psi \right)}^{\top }$$

(22)

The true attitude quaternion is denoted with $\mathit{q}$, while the attitude error quaternion is denoted with $\delta \mathit{q}$. $\delta \widehat{\mathit{a}}$ represents the 3D attitude error and is used to calculate the attitude error quaternion $\mathit{\delta }\widehat{\mathit{q}}\left(\delta \widehat{\mathit{a}}\right)$. The attitude error quaternion $\mathit{\delta }\widehat{\mathit{q}}\left(\delta \widehat{\mathit{a}}\right)$ is determined from the attitude change $\delta \widehat{\mathit{a}}$ as follows: [52].

$$\delta \tilde{\mathit{q}}\left(\delta \widehat{\mathbf{a}}\right)={\displaystyle \frac{1}{16+{\left(\delta \widehat{\mathit{a}}\right)}^{\top }\left(\delta \widehat{\mathit{a}}\right)}}\left(\begin{array}{c}16-{\left(\delta \widehat{\mathit{a}}\right)}^{\top }\left(\delta \widehat{\mathit{a}}\right)\\ 8·\delta \widehat{\mathit{a}}\end{array}\right)$$

(23)

6.1.3. Inertial Measurement Unit

An IMU can be described using the following simplified measurement model. Specific forces and Angular Rate Sensor (ARS) measurements are given as follows: [50]

$${\mathit{f}}_{\mathrm{IMU}}^{\mathrm{b}}={\mathit{f}}_{\mathrm{ib}}^{\mathrm{b}}+{\mathit{b}}_{\mathrm{ACC}}^{\mathrm{b}}+{\mathit{\epsilon }}_{\mathrm{ACC}}^{\mathrm{b}}$$

(24)

$${\mathit{\omega }}_{\mathrm{IMU}}^{\mathrm{b}}={\mathit{\omega }}_{\mathrm{ib}}^{\mathrm{b}}+{\mathit{b}}_{\mathrm{ARS}}^{\mathrm{b}}+{\mathit{\epsilon }}_{\mathrm{ARS}}^{\mathrm{b}}$$

(25)

where ${\mathit{f}}_{\mathrm{ib}}^{\mathrm{b}}$ represents the specific force relating to the acceleration of the IMU. The variable ${\mathit{\omega }}_{\mathrm{ib}}^{\mathrm{b}}$ represents angular velocity, while ${\mathit{v}}_{\mathrm{ib}}^{\mathrm{b}}$ and ${\mathit{a}}_{\mathrm{ib}}^{\mathrm{b}}$ represent linear velocity, respectively. Acceleration (ACC) in the body frame is $\left\{\mathrm{b}\right\}$.

$$\begin{array}{cc}\hfill {\mathit{f}}_{\mathrm{ib}}^{\mathrm{b}}& ={\mathit{R}}_{\mathrm{e}}^{\mathrm{b}}·{\dot{\mathit{v}}}_{\mathrm{ib}}^{\mathrm{e}}-{\mathit{R}}_{\mathrm{e}}^{\mathrm{b}}·{\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}\hfill \\ \hfill & ={\mathit{a}}_{\mathrm{ib}}^{\mathrm{b}}+\mathit{S}\left({\mathit{\omega }}_{\mathrm{ib}}^{\mathrm{b}}\right)·{\mathit{v}}_{\mathrm{ib}}^{\mathrm{b}}-{\mathit{R}}_{\mathrm{e}}^{\mathrm{b}}·{\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}\hfill \end{array}$$

(26)

The gravity vector is represented by ${\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}$

$${\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}={\mathit{\gamma }}_{\mathrm{ib}}^{\mathrm{e}}+{\omega }_{\mathrm{ie}}\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 0\end{array}\right){\mathit{r}}_{\mathrm{eb}}^{\mathrm{e}}$$

(27)

where ${\omega }_{\mathrm{ie}}$ in (27) is the angular rotation rate of the earth (see [36,50]) and $\mathit{S}\left({\mathit{\omega }}_{\mathrm{ib}}^{\mathrm{b}}\right)·{\mathit{v}}_{\mathrm{ib}}^{\mathrm{b}}$ describes the centripetal acceleration. The accelerometer biases are described by ${\mathit{b}}_{★}^{\mathrm{b}}$. The ARS biases are described by ${\mathit{\omega }}_{★}^{\mathrm{b}}$, and ${\mathit{\gamma }}_{\mathrm{ib}}^{\mathrm{e}}$ describes the gravitational acceleration [36].

6.1.4. Kinematics—Strapdown Equations

The position and velocity of the body frame relative to the ECEF frame is represented by ${\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}\in {\mathbb{R}}^3$ and ${\mathit{v}}_{\mathrm{eb}}^{\mathrm{e}}\in {\mathbb{R}}^3$. The superscript $\left\{\mathrm{e}\right\}$ indicates that these vectors are decomposed in the ECEF frame. The attitude between the body and the ECEF frame is given as the unit quaternion ${\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}$, while the angular velocity of the body frame $\left\{\mathrm{b}\right\}$ relative to the ECEF frame $\left\{\mathrm{e}\right\}$ is given as ${\mathit{\omega }}_{\mathrm{eb}}^{\mathrm{e}}\in {\mathbb{R}}^3$. The strapdown equations following the explanations mentioned above are:

$$\begin{array}{cc}\hfill & {\dot{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}={\mathit{v}}_{\mathrm{eb}}^{\mathrm{e}}\hfill \end{array}$$

(28)

$$\begin{array}{cc}\hfill & {\dot{\mathit{v}}}_{\mathrm{eb}}^{\mathrm{e}}=-2·\mathit{S}\left({\mathit{\omega }}_{\mathrm{ie}}^{\mathrm{e}}\right){\mathit{v}}_{\mathrm{eb}}^{\mathrm{e}}+{\mathit{R}}_{\mathrm{b}}^{\mathrm{e}}\left({\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}\right)·{\mathit{f}}_{\mathrm{eb}}^{\mathrm{b}}+{\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill & {\dot{\mathit{q}}}_{\mathrm{b}}^{\mathrm{e}}={\displaystyle \frac{1}{2}}·\mathbf{\Omega }\left(\begin{array}{c}{\mathit{\omega }}_{\mathrm{eb}}^{\mathrm{b}}\end{array}\right)·{\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}\hfill \end{array}$$

(30)

where the specific force acting on the UAV is:

$${\mathit{f}}_{\mathrm{eb}}^{\mathrm{b}}={\left({\mathit{R}}_{\mathrm{b}}^{\mathrm{e}}\right)}^{\top }\left({\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}\right){\dot{\mathit{v}}}_{\mathrm{eb}}^{\mathrm{e}}-{\left({\mathit{R}}_{\mathrm{b}}^{\mathrm{e}}\right)}^{\top }\left({\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}\right){\mathit{g}}_{\mathrm{b}}^{\mathrm{e}}$$

(31)

6.1.5. Time Conventions

The payload raw data were timestamped by a dedicated hardware, (Section 2.2). The timestamps $t_k$ were converted to UNiplexed Information Computing System (UNIX) time. The absolute time measurement to convert $t_k\to t_{\mathrm{UNIX}}$ was provided by the Global Positioning System (GPS) time $T_{\mathrm{GPS}}$, which is derived by the “interval time of week” $iTow$, its fractional part $fTow$ and “weeks since GPS epoch” $GP{S}_{\mathrm{week}}$. All these signals are transmitted by the GPS satellite configuration and received by the GNSS receivers on the UAV. The GPS time $T_{\mathrm{GPS}}$ is calculated as follows [53].

$$T_{\mathrm{GPS}}=iTow·{10}^{-3}+fTow·{10}^{-9}+GP{S}_{\mathrm{week}}·T_{\mathrm{s}}$$

The duration of a GPS week $T_{\mathrm{s}}$ is defined as $T_{\mathrm{s}}\equiv \mathrm{604,800} \mathrm{s}$ [54]. The International Atomic Time $T_{\mathrm{TAI}}$ and the UNIX time $T_{\mathrm{UNIX}}$ are calculated as follows:

$$\begin{array}{cc}\hfill T_{\mathrm{TAI}}& =T_{\mathrm{GPS}}+\mathsf{\Delta }{T}_{\text{TAI-GPS}}\hfill \\ \hfill T_{\mathrm{UNIX}}& =T_{\mathrm{TAI}}+\mathsf{\Delta }{T}_{\text{UNIX-GPS}}-T_{\mathrm{leap}\text{TAI-UTC}}\hfill \end{array}$$

The leap seconds $T_{\mathrm{leap}\text{TAI-UTC}}$ consist of a constant offset $\mathsf{\Delta }{T}_{\text{TAI-GPS}}$ and the leap seconds between $T_{\mathrm{GPS}}$ and $T_{\mathrm{UTC}}$, $T_{\mathrm{leap}\text{GPS-UTC}}$ [55].

$$T_{\mathrm{leap}\text{TAI-UTC}}=\mathsf{\Delta }{T}_{\text{TAI-GPS}}+T_{\mathrm{leap}\text{GPS-UTC}}$$

The epochs for the GPS satellite configuration $T_{\mathrm{GPS}}$ and the UNIX time system $T_{\mathrm{UNIX}}$ are as follows:

$$\begin{array}{cccc}\hfill & T_{\mathrm{GPS}}\left(T_0\right)\hfill & \hfill & =\text{06.01.1980}0:00:00UTC+0\hfill \\ \hfill & T_{\mathrm{UNIX}}\left(T_0\right)\hfill & \hfill & =\text{01.01.1970}0:00:00UTC+0\hfill \end{array}$$

The offsets between $T_{\mathrm{GPS}}$, $T_{\mathrm{TAI}}$ and $T_{\mathrm{UNIX}}$ are therefore as follows (see Table 10):

$$\begin{array}{ccccccc}\hfill & T_{\mathrm{leap}\text{GPS-UTC}}\hfill & \hfill & =18 \mathrm{s}\hfill & \hfill & & \hfill \left[\text{01.01.2017-present}\right]\\ \hfill & \mathsf{\Delta }{T}_{\text{TAI-GPS}}\hfill & \hfill & \equiv 19 \mathrm{s}\hfill & \hfill & & \\ \hfill & \mathsf{\Delta }{T}_{\text{UNIX-GPS}}\hfill & \hfill & \equiv \mathrm{315,964,800} \mathrm{s}\hfill & \hfill & \end{array}$$

The value for $T_{\mathrm{leap}\text{GPS-UTC}}$ is given for the time period during which these experiments were conducted.

6.2. Multiplicative Error-State Kalman Filter—Prediction

The UAV’s Inertial Navigation System (INS) state vector is described as follows:

$$\widehat{\mathit{x}}=\left({\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}},{\widehat{\mathit{v}}}_{\mathrm{eb}}^{\mathrm{e}},{\widehat{\mathit{q}}}_{\mathrm{eb}}^{\mathrm{e}},{\widehat{\mathit{b}}}_{\mathrm{ACC}}^{\mathrm{b}},{\widehat{\mathit{b}}}_{\mathrm{ARS}}^{\mathrm{b}}\right)$$

(32)

The UAV kinematic is propagated as follows (Figure 17) [36,50].

$$\begin{array}{cc}\hfill {\widehat{\mathit{\omega }}}_{\mathrm{eb}k}^{\mathrm{b}}& ={\mathit{\omega }}_{\mathrm{IMU}k}^{\mathrm{b}}-{\widehat{\mathit{b}}}_{\mathrm{ARS}k}^{\mathrm{b}}-{\widehat{\mathit{R}}}_{\mathrm{e}}^{\mathrm{b}}{\mathit{\omega }}_{\mathrm{ie}}^{\mathrm{e}}\hfill \end{array}$$

(33)

$$\begin{array}{cc}\hfill \mathsf{\Delta }{\mathit{q}}_{\mathrm{b}k}^{\mathrm{e}}& =\left(\begin{array}{c}cos\left({\displaystyle \frac{T_{\mathrm{IMU}}}{2}}·{\parallel {\widehat{\mathit{\omega }}}_{\mathrm{eb}k}^{\mathrm{b}}\parallel }_2\right)\\ sin\left({\displaystyle \frac{T_{\mathrm{IMU}}}{2}}·{\parallel {\widehat{\mathit{\omega }}}_{\mathrm{eb}k}^{\mathrm{b}}\parallel }_2\right)·{\displaystyle \frac{{\widehat{\mathit{\omega }}}_{\mathrm{eb}k}^{\mathrm{b}}}{\parallel {\widehat{\mathit{\omega }}}_{\mathrm{eb}k}^{\mathrm{b}}{\parallel }_2}}\end{array}\right)\hfill \end{array}$$

(34)

$$\begin{array}{cc}\hfill {\widehat{\mathit{q}}}_{\mathrm{b}k}^{\mathrm{e}-}& ={\widehat{\mathit{q}}}_{\mathrm{b}k-1}^{\mathrm{e}}\otimes \mathsf{\Delta }{\mathit{q}}_{\mathrm{b}k}^{\mathrm{e}}\hfill \end{array}$$

(35)

$$\begin{array}{cc}\hfill {\mathit{R}}_{\mathrm{b}k}^{\mathrm{e}}& =\left({q_{\mathrm{s}k}}^2-{{\mathit{q}}_{\mathrm{v}k}}^{\top }{\mathit{q}}_{\mathrm{v}k}\right)·{\mathit{I}}_3\times 3+2q_{\mathrm{s}k}\mathit{S}\left({\mathit{q}}_{\mathrm{v}k}\right)+2{\mathit{q}}_{\mathrm{v}k}{{\mathit{q}}_{\mathrm{v}k}}^{\top }\hfill \end{array}$$

(36)

$$\begin{array}{cc}\hfill {\mathit{R}}_{\mathrm{b}\mathrm{avg}k}^{\mathrm{e}}& ={\displaystyle \frac{1}{2}}\left({\mathit{R}}_{\mathrm{b}k}^{\mathrm{e}}+{\mathit{R}}_{\mathrm{b}k-1}^{\mathrm{e}}\right)\hfill \end{array}$$

(37)

$$\begin{array}{cc}\hfill {\widehat{\mathit{a}}}_{\mathrm{eb}k}^{\mathrm{e}}& =-2\mathit{S}\left({\mathit{\omega }}_{\mathrm{ie}}^{\mathrm{e}}\right){\widehat{\mathit{v}}}_{\mathrm{eb}k-1}^{\mathrm{e}-}+{\mathit{R}}_{\mathrm{b}avgk}^{\mathrm{e}}\left({\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}\right){\widehat{\mathit{f}}}_{\mathrm{ib}k}^{\mathrm{b}}+{\mathit{g}}_{\mathrm{b}k}^{\mathrm{e}}\hfill \end{array}$$

(38)

$$\begin{array}{cc}\hfill {\widehat{\mathit{v}}}_{\mathrm{eb}k}^{\mathrm{e}-}& ={\mathit{v}}_{\mathrm{eb}k-1}^{\mathrm{e}-}+{\widehat{\mathit{a}}}_{\mathrm{eb}k}^{\mathrm{e}}·T_{\mathrm{IMU}}\hfill \end{array}$$

(39)

$$\begin{array}{cc}\hfill {\widehat{\mathit{p}}}_{\mathrm{eb}k}^{\mathrm{e}-}& ={\widehat{\mathit{p}}}_{\mathrm{eb}k-1}^{\mathrm{e}-}+{\widehat{\mathit{v}}}_{\mathrm{eb}k}^{\mathrm{e}-}·T_{\mathrm{IMU}}+{\displaystyle \frac{T_{\mathrm{IMU}}^2}{2}}·{\widehat{\mathit{a}}}_{\mathrm{eb}k}^{\mathrm{e}}\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill {\widehat{\mathit{b}}}_{\mathrm{ACC}k}^{\mathrm{e}-}& =e^{-}{\displaystyle \frac{{\mathrm{T}}_{\mathrm{IMU}}}{{\mathrm{T}}_{\mathrm{ACC}}}}·{\widehat{\mathit{b}}}_{\mathrm{ACC}k-1}^{\mathrm{e}-}\hfill \end{array}$$

(41)

$$\begin{array}{cc}\hfill {\widehat{\mathit{b}}}_{\mathrm{ARS}k}^{\mathrm{e}-}& =e^{-}{\displaystyle \frac{{\mathrm{T}}_{\mathrm{IMU}}}{{\mathrm{T}}_{\mathrm{ARS}}}}·{\widehat{\mathit{b}}}_{\mathrm{ARS}k-1}^{\mathrm{e}-}\hfill \end{array}$$

(42)

where the subscript k indicates the discrete time increment of the MEKF. ${\mathit{\omega }}_{\mathrm{IMU}k}^{\mathrm{b}}$ is given by (2) and ${\widehat{\mathit{f}}}_{\mathrm{ib}k}^{\mathrm{b}}$ by (26). Furthermore, for every new IMU measurement, the covariance matrix ${\widehat{\mathit{P}}}_k$ is updated as follows ([36] (Ch. 3)) and ([50] (Ch. 4)):

$$\begin{array}{cc}\hfill {\mathbf{\Phi }}_k& ={\mathit{I}}_{15}\times 15+{\mathit{F}}_k·T_{\mathrm{IMU}}\hfill \end{array}$$

(43)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_k^{-}& ={\mathbf{\Phi }}_k{\widehat{\mathit{P}}}_k^{+}{\mathbf{\Phi }}_k^{\top }+{\mathit{Q}}_{\mathrm{d}k}\hfill \end{array}$$

(44)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_k^{-}& \leftarrow \left({\widehat{\mathit{P}}}_k^{-}+{\widehat{\mathit{P}}}_k^{{-}^{\top }}\right)/2\hfill \end{array}$$

(45)

where the matrices ${\mathit{F}}_k$ and ${\mathit{Q}}_{\mathrm{d}k}$ are given in Appendix C.1.

6.3. Multiplicative Error-State Kalman Filter—Update

The error-state vector $\delta \widehat{\mathit{x}}$ expresses the correction to $\widehat{\mathit{x}}$ after an aiding measurement has been received:

$$\delta \widehat{\mathit{x}}=\left(\delta \widehat{\mathit{p}},\delta \widehat{\mathit{v}},\delta \widehat{\mathit{a}},\delta {\widehat{\mathit{b}}}_{\mathrm{ACC}},\delta {\widehat{\mathit{b}}}_{\mathrm{ARS}}\right)$$

(46)

The error-state vector is calculated for each aiding measurement (Figure 17), and the Kalman filter update with a new aiding measurement is performed as follows:

$$\begin{array}{cc}\hfill {\mathit{K}}_k& ={\mathit{P}}_k^{-}{\mathit{H}}_k^{\top }{\left({\mathit{H}}_k{\mathit{P}}_k{\mathit{H}}_k^{\top }+{\mathit{R}}_k\right)}^{-}1\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_k^{+}& =\left(\mathit{I}-{\mathit{K}}_k{\mathit{H}}_k\right){\widehat{\mathit{P}}}_k^{-}{\left(\mathit{I}-{\mathit{K}}_k{\mathit{H}}_k\right)}^{\top }+{\mathit{K}}_k{\mathit{R}}_k{\mathit{K}}_k^{\top }\hfill \end{array}$$

(48)

$$\begin{array}{cc}\hfill {\widehat{\mathit{P}}}_k^{+}& \leftarrow \left({\widehat{\mathit{P}}}_k^{+}+{\widehat{\mathit{P}}}_k^{{+}^{\top }}\right)/2\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill \delta {\widehat{\mathit{x}}}_k& ={\mathit{K}}_k·\left({\mathit{y}}_k-{\widehat{\mathit{y}}}_k\right)\hfill \end{array}$$

(50)

where ${\mathit{R}}_k$ is the measurement noise covariance matrix. Update of the state vector $\widehat{\mathit{x}}$ with error states in $\delta \widehat{\mathit{x}}$ is described in (51)–(55).

$$\begin{array}{cc}\hfill {\widehat{\mathit{p}}}_k^{\mathrm{e}+}& ={\widehat{\mathit{p}}}_{k-1}^{\mathrm{e}}+\delta {\widehat{\mathit{p}}}_k^{\mathrm{e}}\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill {\widehat{\mathit{v}}}_k^{\mathrm{e}+}& ={\widehat{\mathit{v}}}_{k-1}^{\mathrm{e}}+\delta {\widehat{\mathit{v}}}_k^{\mathrm{e}}\hfill \end{array}$$

(52)

$$\begin{array}{cc}\hfill {\widehat{\mathit{q}}}_k^{\mathrm{e}+}& ={\widehat{\mathit{q}}}_{k-1}^{\mathrm{e}}\otimes \delta {\widehat{\mathit{q}}}_k^{\mathrm{e}}\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill {\widehat{\mathit{b}}}_{\mathrm{ACC}k}^{\mathrm{e}+}& ={\widehat{\mathit{b}}}_{\mathrm{ACC}k-1}^{\mathrm{e}}+\delta {\widehat{\mathit{b}}}_{\mathrm{ACC}k}^{\mathrm{e}}\hfill \end{array}$$

(54)

$$\begin{array}{cc}\hfill {\widehat{\mathit{b}}}_{\mathrm{ARS}k}^{\mathrm{e}+}& ={\widehat{\mathit{b}}}_{\mathrm{ARS}k-1}^{\mathrm{e}}+\delta {\widehat{\mathit{b}}}_{\mathrm{ARS}k}^{\mathrm{e}}\hfill \end{array}$$

(55)

After the INS states have been corrected, the error state vector $\delta \widehat{\mathit{x}}$ is reset to 0.

6.3.1. Position Correction Using GNSS

As mentioned in Section 4, the GNSS receivers stored RAW-GNSS observables, allowing PPK-GNSS post-processing (Figure 8). Since there are two GNSS receivers (Figure 4) with position accuracies smaller than the distance between the two sensors (see Figure 4) the MEKF position updates using GNSS measurements also result in UAV heading corrections. The subscript $\mathrm{j}\in \{1,2\}$ in (56) indicates the GNSS antennae location, as shown in Figure 4. The MEKF position correction using GNSS measurements is based on the following:

$$\begin{array}{cc}\hfill {\mathit{y}}_j& ={\mathit{p}}_{\mathrm{e}{\mathrm{g}}_j}^{\mathrm{e}}+{\widehat{\mathit{R}}}_{\mathrm{b}}^{\mathrm{e}}·{\mathit{p}}_{{\mathrm{g}}_j\mathrm{b}}^{\mathrm{b}}\hfill \end{array}$$

(56)

$$\begin{array}{cc}\hfill \widehat{\mathit{y}}& ={\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill \mathit{H}& =\left(\begin{array}{cc}{\mathit{I}}_3& {\mathbf{0}}_3\times 12\end{array}\right)\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill \mathit{R}& ={\widehat{\mathit{R}}}_{\mathrm{e}}^{\mathrm{n}}\left(\begin{array}{ccc}{\sigma }_{\mu }^2& 0& 0\\ 0& {\sigma }_{\lambda }^2& 0\\ 0& 0& {\sigma }_{\mathrm{h}}^2\end{array}\right){\widehat{\mathit{R}}}_{\mathrm{e}}^{\mathrm{n}\top }\hfill \end{array}$$

(59)

6.3.2. Position Correction Using Barometer

To use the barometer as an aiding sensor in the MEKF, the altitude difference between earth’s geoid, here modeled using the EGM2008 [56], and the WGS-84 ellipsoid height must be determined. Using Vermeille’s [57] approach, the latitude, $\widehat{\mu }$, longitude $\widehat{\lambda }$, and height above the WGS-84 ellipsoid ${\widehat{h}}_{\text{wgs-84}}$ are calculated from ${\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}$ [56]

$$\widehat{\mu };\widehat{\lambda };{\widehat{h}}_{\text{wgs-84}}\leftarrow {\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}$$

(60)

where $\widehat{\mu }$ and $\widehat{\lambda }$, the geoid’s height over the WGS-84 ellipsoid, $h_{\mathrm{geo}}$ can be calculated using EGM2008 [56]. As a result, the geoid height at the take-off position of the UAV $h_{\mathrm{geo}0}$ can be determined and an offset between the measured GNSS altitude $h_{\text{wgs-84}}$ and the corrected barometer height $h_{\mathrm{b}}$ can be calculated. The value $h_{\mathrm{cal}}$ was used to eliminate the remaining offset between the geoid and ellipsoid at the take-off position. To calibrate the barometer to the flight’s altitude (see Appendix A, Table 10), a scaling factor $k_{\mathrm{sc}}$ was used. The absolute flight altitude for the calibration was obtained from GNSS sensor 1 (Figure 4). The values for $h_{\mathrm{cal}}$ and $k_{\mathrm{sc}}$ vary depending on atmospheric conditions and are listed in

Table A2. Calibration values for the UAV barometer.

	${\mathit{h}}_{\mathbf{cal}}$ $\left[\mathbf{m}\right]$	Scaling ${\mathit{k}}_{\mathbf{sc}}$ $[-]$
Flight 1	$38.5$	$0.9533$
Flight 5	$39.4$	$0.9455$
Flight 8	$39.5$	$0.9455$

. The calibration procedure for the barometer is more thoroughly explained in Appendix C.2.

The measurement update with the calibrated barometer altitude $h_{\mathrm{b}}$ is based on the following ([36] (Ch. 6)):

$$\begin{array}{cc}\hfill \mathit{y}& =h_{\mathrm{b}}\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill \widehat{\mathit{y}}& ={\widehat{h}}_{\mathrm{b}}={\parallel {\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}-{\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}\parallel }_2\hfill \end{array}$$

(62)

$$\begin{array}{cc}\hfill \mathit{H}& =\left(\begin{array}{cc}{\displaystyle \frac{{\left({\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}-{\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}\right)}^{\top }}{\parallel {\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}-{\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}{\parallel }_2}},& {\mathbf{0}}_1\times 12\end{array}\right)\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill \mathit{R}& ={\sigma }_{\mathrm{b}}^2\hfill \end{array}$$

(64)

where ${\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}$ is calculated as:

$$\begin{array}{cc}\hfill {\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}& =\left(\begin{array}{c}\left(R_{\mathrm{N}}+{\widehat{h}}_{\mathrm{geo}}\left(\widehat{\mu },\widehat{\lambda }\right)\right)cos\left(\widehat{\mu }\right)cos\left(\widehat{\lambda }\right)\\ \left(R_{\mathrm{N}}+{\widehat{h}}_{\mathrm{geo}}\left(\widehat{\mu },\widehat{\lambda }\right)\right)cos\left(\widehat{\mu }\right)sin\left(\widehat{\lambda }\right)\\ \left(R_{\mathrm{N}}\left(1-e^2\right)+h_{geo}\right)sin\left(\widehat{\mu }\right)\end{array}\right)\hfill \end{array}$$

(65)

$$\begin{array}{cc}\hfill R_{\mathrm{N}}& =a/\sqrt{1-e^{2}sin{\left(\widehat{\mu }\right)}^2}\hfill \end{array}$$

(66)

$R_{\mathrm{N}}$ is given as in (66) ([36] (Ch. 2.4.2)), with a and e given in Appendix Calculation of Earth Eccentricity.

6.4. Multiplicative Error-State Kalman Filter—Tuning and Performance

The measurement uncertainties for the two PPK-GNSS receivers and the barometer used in the MEKF are listed in

Table 7. Measurement uncertainties.

	PPK-GNSS	Barometer
${\sigma }_{\mu }$ [$\mathrm{m}$]	0.2	—
${\sigma }_{\lambda }$ [$\mathrm{m}$]	0.2	—
${\sigma }_{\mathrm{h}}$ [$\mathrm{m}$]	0.9	1

. These values were determined by tuning and are in line with the datasheet for the GNSS receivers [35,53] and the barometer [58].

The process noise matrix is given by:

$${\mathit{Q}}^{\mathrm{b}}=\left(\begin{array}{cccc}{q}_{\mathrm{ACC}}& 0& 0& 0\\ 0& q_{\mathrm{ARS}}& 0& 0\\ 0& 0& q_{\mathrm{ACC},\mathrm{bias}}& 0\\ 0& 0& 0& q_{\mathrm{ARS},\mathrm{bias}}\end{array}\right)$$

(67)

These values were selected by calculating the ARS and ACC sensor noise and bias models for the IMU [34] according to [36,50] and are listed in

Table 8. Process noise.

Variable	Unit	Value
$q_{\mathrm{ACC}}$	[$\mathrm{m}$2/$\mathrm{s}$]	$8.6044·{10}^{-6}$
$q_{\mathrm{ARS}}$	[$\mathrm{rad}$2/$\mathrm{s}$]	$6.8539·{10}^{-6}$
$q_{\mathrm{ACC},\mathrm{bias}}$	[$\mathrm{m}$2/$\mathrm{s}$5]	$2.1443·{10}^{-9}$
$q_{\mathrm{ARS},\mathrm{bias}}$	[$\mathrm{rad}$2/$\mathrm{s}$3]	$4.0500·{10}^{-9}$

.

With these settings, the MEKF is able to estimate position ${\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}$ and altitude ${\mathit{q}}_{\mathrm{b}}^{\mathrm{e}}$ with high accuracy, so that direct georeferencing (Section 7) is possible. As an example, Figure 18a,b show the MEKF acceleration bias $\delta {\widehat{\mathit{b}}}_{\mathrm{ACC},\mathrm{bias}}$ and the MEKF gyroscope bias $\delta {\widehat{\mathit{b}}}_{\mathrm{ARS},\mathrm{bias}}$ for flight 8, respectively.

6.5. Time Synchronization between Sensor Logs and MEKF Output

The MEKF output maintains the time granularity of $\mathsf{\Delta }t=10 \mathrm{n}\mathrm{s}$ of the IMU timestamps. Following data cleanup and correction for SentiBoard drift with $T_{\mathrm{GPS}}$, the converted $T_{\mathrm{UNIX}}$ (Section 6.1.5) is used to synchronize the payload and UAV-autopilot logs.

The concluding state-estimation processing step is packing the data into chunks of $N=2000$ imaging spectrometer frames ${\mathit{F}}_{\mathrm{IS}}$, together with the corresponding RGB images and MEKF output. Splitting the data into sizable packages ensures that file sizes are manageable. The data packages created in this way serve as input for the georeferencing step described in Section 7 and are referred to below as transects.

7. Georeferencing

The georeferencing approach used in this paper was based on the open-source gref4hsi python library (gref4hsi-GitHub: https://github.com/havardlovas/gref4hsi (accessed on 14 June 2024) PyPI: https://pypi.org/project/gref4hsi/) (see also [59] (accessed on 14 June 2024)), which is an expansion of the georeferencing approach used in [60]. This library is targeted towards the georeferencing (ray–terrain intersection), orthorectification (resampling), and coregistration (geometric calibration) of hyperspectral push-broom imagery. The data in this paper were georeferenced and orthorectified with default settings except that a geoid model was used as terrain model, and we orthorectified the datacube to a 1 × 1 $\mathrm{m}$ Universal Transverse Mercator (UTM) 33N grid. The input for the software is the imaging spectrometer data, navigation data, a push-broom camera model, and a terrain model. The output are orthorectified datacubes, RGB composites, and ancillary data.

7.1. State-Estimation Data—Time Interpolation and Pose Conversions

The gref4hsi software performs a ray–terrain intersection in ECEF $\left\{e\right\}$ for each imaging spectroscopy capture. The closest position ${\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}$ and orientation ${\mathit{q}}_{\mathrm{n}}^{\mathrm{b}}$ of the state-estimation output to a capture are found. ECEF positions, ${\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}$, and NED quaternions, ${\mathit{q}}_{\mathrm{b}}^{\mathrm{n}}$, are used as input to the software.

At first, the software performs linear interpolation of ${\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}$ and Spherical Linear Interpolation (SLERP) for orientations ${\mathit{q}}_{\mathrm{b}}^{\mathrm{n}}$ to the timestamp of each imaging spectrometer frame, ${\mathit{F}}_{\mathrm{IS}}$. The interpolated orientation, ${\widehat{\mathit{q}}}_{\mathrm{b}}^{\mathrm{n}}$, is then converted to the ECEF frame, $\left\{e\right\}$, by:

$$\begin{array}{cc}\hfill & {\mathit{R}}_{\mathrm{n}}^{\mathrm{e}}\leftarrow \left[\widehat{\mu },\widehat{\lambda },\widehat{h}\right]\leftarrow {\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}\hfill \end{array}$$

(68)

$$\begin{array}{cc}\hfill & {\widehat{\mathit{R}}}_{\mathrm{b}}^{\mathrm{n}}\leftarrow {\widehat{\mathit{q}}}_{\mathrm{b}}^{\mathrm{n}}\hfill \end{array}$$

(69)

$$\begin{array}{cc}\hfill & {\widehat{\mathit{q}}}_{\mathrm{b}}^{\mathrm{e}}\leftarrow {\widehat{\mathit{R}}}_{\mathrm{b}}^e={\mathit{R}}_{\mathrm{n}}^{\mathrm{e}}·{\widehat{\mathit{R}}}_{\mathrm{b}}^{\mathrm{n}}\hfill \end{array}$$

(70)

The rotation matrices ${\mathit{R}}_{\mathrm{n}}^{\mathrm{e}}$ depend on the UAV’s current location and are determined for each time step dependent on the latitude $\widehat{\mu }$, longitude, $\widehat{\lambda }$ and altitude $\widehat{h}$, which are calculated from ${\mathit{p}}_{\mathrm{eb}}^{\mathrm{e}}$ (68). The origin of the local NED coordinate frame used for georeferencing is determined by the first N frames, where N is the number of spatial–spectral imaging spectrometer frames grouped into one transect, as defined in Section 6.5. The time-interpolated ECEF positions of the UAV can then be used in the georeferencing. Notably, the ECEF referred to here is WGS-84 with EPSG code 4978.

7.2. Push-Broom Camera Model

We hereunder include parameters describing the camera projection model as well as the relative pose of the imager on the UAV, sometimes referred to as boresight angles (three angles) and lever arms (three lever arms). For the low $\mathrm{FOV}$ imager, we omitted lens distortions and simply modeled the camera using a 1D pinhole model. We can calculate the angular field across the track, $\zeta$, from the slit length, $S_{\mathrm{l}}$, and the focal length f of the objective lens (see Figure 6):

$$\zeta =2·{tan}^{-1}\left({\displaystyle \frac{S_{\mathrm{l}}}{2·f}}\right);\left[\mathrm{rad}\right]$$

(71)

Based on the principle of conformal triangles, we can calculate the equivalent focal length in pixels using the number of illuminated pixels along the spatial axis of the camera sensor, $s_{\mathrm{w}}$:

$$f_{\mathrm{pix}}={\displaystyle \frac{n_{\mathrm{p}}}{tan\left(\frac{\zeta }{2}\right)}};\left[\mathrm{pix}\right]$$

(72)

The second parameter needed to define pixels as rays is the principal point, $u_{\mathrm{c}}$ (defining the optical axis). It is commonplace to set it as the midpoint, i.e., $u_{\mathrm{c}}\approx n_{\mathrm{pix}}/2$, where $n_{\mathrm{pix}}$ is the number of pixels in the region of interest $s_{\mathrm{w}}$.

Using these camera parameters, we can compute a ray, given the pixel index. If we define pixel indices as $i\in \{1,\dots ,n_{\mathrm{pix}}\}$, a pixel can be defined by the ray, ${\mathit{r}}_{\langle i\rangle }^{\mathrm{s}}$, as:

$${\mathit{r}}_{\langle i\rangle }^{\mathrm{s}}=\left(\begin{array}{c}{\displaystyle \frac{u_i-u_{\mathrm{c}}}{f_{\mathrm{pix}}}}\\ 0\\ 1\end{array}\right)$$

(73)

where we should note that the x-axis of the imager is approximately aligned with the y-axis of the UAV due to different conventions. For further illustrations of the pinhole model see Figure 6. Moreover, since the MEKF estimates the UAV’s pose, we need to express the rays in the UAV’s coordinate reference system (CRS). The boresight angles describing the rotation from the imaging spectrometer frame $\left\{s\right\}$ to the UAV body frame $\left\{b\right\}$ are Euler angles parametrized ${\mathit{R}}_{\mathrm{s}}^{\mathrm{b}}$ (defined in Appendix D). With ${\mathit{R}}_{\mathrm{s}}^{\mathrm{b}}$, the ray direction in the ECEF frame $\left\{e\right\}$ is:

$${\widehat{\mathit{r}}}_{\langle i\rangle }^{\mathrm{e}}={\widehat{\mathit{R}}}_{\mathrm{b}}^{\mathrm{e}}·{\mathit{R}}_{\mathrm{s}}^{\mathrm{b}}·{\mathit{r}}_{\langle i\rangle }^{\mathrm{s}}$$

(74)

The position of the imaging spectrometer in the ECEF frame $\left\{e\right\}$, i.e., ray origin, is determined by:

$${\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}={\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}+{\mathit{R}}_{\mathrm{b}}^{\mathrm{e}}·{\mathit{p}}_{\mathrm{bs}}^{\mathrm{b}}$$

(75)

The lever arm ${\mathit{p}}_{\mathrm{bs}}^{\mathrm{b}}$, here meaning the translation from body frame $\left\{b\right\}$ to imaging spectrometer frame $\left\{s\right\}$, can be determined from Figure 4. For the configuration used in the work, we have ${\mathit{p}}_{\mathrm{bs}}^{\mathrm{b}}={\left(0.08,0,0\right)}^{\top }$.

7.3. Digital Elevation Model

A raster file representing the EGM2008 geoid [56] is used to represent the sea surface, and it was cropped to the survey area and used as a terrain model. The DEM is used to create a point cloud, which in turn is used to create a 3D-mesh, representing the geoid locally. The 3D mesh was then converted to ECEF and used for ray–terrain intersection.

7.4. Ray Tracing Methodology for 3D Mesh Intersection

Ray tracing is used to find the intersections between the rays extending from the imaging spectrometer ${\mathit{r}}_{\langle i\rangle }^{\mathrm{e}}$ and the 3D mesh generated in Section 7.3. The intersection search is limited to a length of $500 \mathrm{m}$ (see $l_{\langle i\rangle }$ in (77)). The intersection point of a ray with the 3D mesh, decomposed using the ECEF coordinates $\left\{e\right\}$, is represented by ${\mathit{p}}_{\mathrm{em}}^{\mathrm{e}}$. The relationship between the 2D mesh intersection point ${\widehat{\mathit{p}}}_{\mathrm{em}}^{\mathrm{e}}$, the vector from the imaging spectrometer to the 2D mesh ${\widehat{\mathit{p}}}_{\mathrm{sm}}^{\mathrm{e}}$, and ${\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}$, respectively, ${\widehat{\mathit{r}}}_{\langle i\rangle }^{\mathrm{e}}$ and $l_{\langle i\rangle }$, is given by:

$$\begin{array}{cc}\hfill {\widehat{\mathit{p}}}_{\mathrm{em}}^{\mathrm{e}}& ={\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}+{\widehat{\mathit{p}}}_{\mathrm{sm}}^{\mathrm{e}}\hfill \end{array}$$

(76)

$$\begin{array}{cc}\hfill {\widehat{\mathit{p}}}_{\mathrm{em}}^{\mathrm{e}}& ={\widehat{\mathit{p}}}_{\mathrm{es}}^{\mathrm{e}}+{\widehat{\mathit{r}}}_{\langle i\rangle }^{\mathrm{e}}·l_{\langle i\rangle }\hfill \end{array}$$

(77)

Since the MEKF outputs and thus ${\widehat{\mathit{p}}}_{\mathrm{sm}}^{\mathrm{e}}$ are timestamped, each mesh intersection point ${\widehat{\mathit{p}}}_{\mathrm{em}}^{\mathrm{e}}$ corresponds to a reflectance spectrum ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ generated in Section 5. This results in a 3D point cloud where each point represents a reflectance spectrum ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$.

7.5. Orthorectification

The point cloud generated in the geocentric coordinate system $\left\{e\right\}$, or ECEF, is transformed into a desired projected CRS, here UTM 33N. The projected point cloud is used to find the extent of the transect and to generate a shapefile describing the footprint, which in turn is used to generate a regular grid with a resolution of $1 \mathrm{m}$ with the CRS coordinates x and y aligned with east and north. By determining the nearest neighbor of each grid point and each point in the point cloud, we get a resampling map. This allows for resampling of the recorded datacube into the orthorectified datacube, which spatially corresponds to a grid. The datacube is masked by the footprint and invalid pixels are given a “no-data” value. The metadata of each non-georeferenced datacube are supplemented with additional data related to the coordinate system and saved as a Band Sequential (BSQ) image file. An RGB composite of the datacube is also orthorectified, as it is easier to use for visualization, (Figure 19). The square in Figure 19 represents the area that is of interest for further analysis in Section 8 (see Appendix F).

7.6. Direct Georeferencing Accuracy

The presented payload was designed without a gimbal. An attitude or position error in the state-estimation step has therefore a direct impact on the georeferenced position of each spectrum. Georeferencing errors can mainly be attributed to the following factors:

$$\mathsf{\Delta }{\widehat{\mathit{p}}}_{\mathrm{em}}^{\mathrm{e}}=\mathsf{\Delta }{\widehat{\mathit{p}}}_{\mathrm{eb}}^{\mathrm{e}}+h·tan\left(\mathsf{\Delta }\mathbf{\Theta }\right)$$

(78)

where $\mathsf{\Delta }\mathsf{\Theta }={\left(\mathsf{\Delta }\varphi ,\mathsf{\Delta }\theta ,\mathsf{\Delta }\psi \right)}^{\top }$ is the attitude error. As (78) shows, the georeferencing accuracy becomes highly sensitive to attitude errors with increased altitude.

We rely on landmarks to determine the accuracy of the imaging spectrometer georeferencing approach discussed in this paper. Figure 20 shows the Ny-Ålesund pier (visible in the lower half of Figure 19). The distance between the base map and the directly georeferenced image gives an accuracy between $2.0$ $\mathrm{m}$ and $4.1$ $\mathrm{m}$ for this part of the flight. Over Ny-Ålesund (and the pier in Figure 20 the UAV flew at an altitude of $h=200 \mathrm{m}$. According to (78), for this flight altitude, the 1° attitude error corresponds to a $3.5$ $\mathrm{m}$ georeferencing error.

When looking at Figure 21a, it is evident that the accuracy is not always as good as Figure 20 would suggest. The UAV was flying at an altitude of 300 $\mathrm{m}$ over the area shown in Figure 21a. As (78) shows, at higher altitudes h, attitude errors $\mathsf{\Delta }\mathsf{\Theta }$ translate to higher georeferencing errors. At an altitude of 300 $\mathrm{m}$, an attitude error of 1° already causes a georeferencing error of $5.24$ $\mathrm{m}$. The decrease in accuracy in Figure 21a compared to Figure 20 can be partly attributed to this, and partly to the highly dynamic maneuvering of the UAV flying over the area in Figure 21a,b. The overflight of the island shown in Figure 21a corresponds to the time points 900 $\mathrm{s}$ and 2950 $\mathrm{s}$ in Figure 18. It is visible in these two plots that the MEKF bias estimated $\delta {\widehat{\mathit{b}}}_{\mathrm{ACC},\mathrm{bias}}$ and $\delta {\widehat{\mathit{b}}}_{\mathrm{ARS},\mathrm{bias}}$ increased temporarily.

Figure 21b shows the Ny-Ålesund airport area. This image illustrates that the direct georeferencing accuracy degrades during dynamic UAV maneuvers such as the final approach or the climb after takeoff. Suggestions for improving image quality in these flight conditions are discussed in Section 9.2.

8. Ocean Color Estimation

8.1. Sun-Glint Removal

The low sun elevation during the UAV flights in Ny-Ålesund caused patches in the georeferenced ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes where considerable sun glint is visible. This is clearly seen in Figure 19 and Figure A5. For further processing, these images were therefore corrected for sun glint. The software hytools [61] with the method from [62] was used to correct the ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes for sun glint. An iceberg-free deep water sample in the middle of the fjord was selected for each flight and used to correct the ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacube for one complete flight. The method presented in [62] is particularly suited for data gathered with UAVs or aircraft, since it performs well with high-resolution data and does not require masking of land areas, icebergs, ships, or areas with other high ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ values. Figure 22 shows the transect with the “deep water/sun glint” sample (red box) used for the sun glint correction for flight 8. Similar samples were chosen for the other analyzed flights. The longest wavelength ($\lambda =797 \mathrm{n}\mathrm{m}$), in the Near Infrared spectrum (NIR), which is assumed to not be heavily influenced by second-order optical effects, was used in this algorithm to correct for sunlight.

8.2. Extraction and Recombination of Imaging Spectroscopy Data

The imaging spectroscopy data gathered inside the sampling area visible in Figure 19 and Figure A5 are of particular interest for data analysis, since this box corresponds to the area defined for robotic missions and ground sampling in the overarching mission in Ny-Ålesund/Svalbard (see Appendix F).

For further analysis,

Table A4. The coordinates for the sampling box defined for observational pyramid.

	EPGS:4326 WGS-84		EPGS:32633WGS-84/UTM Zone 33N
	Longitude $\mathit{\mu }$	Latitude $\mathit{\lambda }$	Northing [m]	Easting [m]
1	78°57′41.888″	11°57′34.84″	9,261,245.442	2,877,896.203
2	78°56′10.691″	11°57′34.84″	9,255,900.023	2,875,579.055
3	78°56′56.264″	12°1′32.665″	9,252,422.498	2,890,882.088
4	78°56′56.264″	11°53′37.015″	9,264,696.813	2,862,566.410

, of the imaging spectroscopy data, the transects located inside the defined coordinates were merged into one connected datacube. Overlapping areas were avoided by removing transects for areas that were overflown previously in the same flight. The datacubes were clipped at the boundaries of the sampling area. Transects with corrupted or faulty data were removed. The extracted data cubes for flight 1, flight 5, and flight 8 are shown in Figure 23. These ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes form the basis for further analysis in this section.

8.3. Exploratory Data Analysis Approaches

A Principal Component Analysis (PCA) was performed on all three datacubes shown in Figure 23. The first Principal Component (PC-1) was used to derive a mask excluding non-water areas. The non-water areas are mainly icebergs/icefloes from the surrounding glaciers, boats, and robotic agents operating in the area. Two exploratory data analysis algorithms were applied to the ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes. The spectral angle mapper (SAM) and a nonlinear unmixing method based on Non-negative Matrix Factorization (NMF) were used.

8.3.1. Spectral Angle Mapper

A spectral angle ${\theta }_{\mathrm{SA}}$ as defined in:

$${\theta }_{\mathrm{SA}}=arccos\left({\displaystyle \frac{{\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)·{\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)}{\parallel {\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right){\parallel }_2·{\parallel {\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)\parallel }_2}}\right)$$

(79)

where ${\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)$ is the reference spectrum. The SAM allows for visualizing small spatial variations in the ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacube. The masked datacube, created with PC-1, was further processed with the SAM published in [63] and implemented in [64].

In the first attempt, the mean spectrum ${\mathit{\rho }}_{\mathrm{RS}\mathrm{mean}}\left(\lambda \right)$ of the datacube was used as the reference spectra ${\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)$. The mean spectrum was calculated as shown in (80). The results for this analysis are shown on the left-hand side of Figure 24 for flight 8 and in Figure A7 and Figure A8 for flights 1 and 5. The right-hand side in these figures shows the SAM results with the Chl-A reference spectrum ${\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)$ shown in Figure 25.

The Chl-A reference spectrum, ${\mathit{\rho }}_{\mathrm{RS}\mathrm{ref}}\left(\lambda \right)$, was derived with the Ecolight-S ocean model [49,65] from the weight-specific in vivo absorption spectrum of the main Chl-A pigment, ${\alpha }_{\mathrm{sol}\mathrm{i}}\left(\lambda \right)$, $\left[{\mathrm{m}}^2\mathrm{m}{\mathrm{g}}^{-1}\right]$ published in [66]. The simulation results for $5 \mathrm{m}\mathrm{g}{\mathrm{m}}^{-3}$ Chl-A in pure water are shown in Figure 25. In the simulation, a concentration of $5 \mathrm{m}\mathrm{g}{\mathrm{m}}^{-3}$ Chl-A was used for the surface layer up until a depth of 15 $\mathrm{m}$. Infinitely deep water was used as a bottom boundary condition because we are simulating Chl-A in pure water. To simulate the water status of the fjord during the UAV mission, the bottom reflections should be included, as the waters were clear. In addition, other constituents should be modeled, for example, color Dissolved Organic Matter (cDOM), the different phytoplankton species and their scattering properties, and minerals. However, setting up this type of simulation is out of scope for this work. Figure 26 shows the deviation of the spectral angle ${\theta }_{\mathrm{SA}}$ from a reference Chl-A spectrum for flight 8, the right-hand side is a zoomed-in area of the ${\theta }_{\mathrm{SA}}$ variation. It can be seen that the white patches in the transect in the original image on the left are masked-out areas. When comparing Figure 26 with Figure A6 it becomes clear that these masked-out areas are mainly icebergs. Most of the icebergs in Kongsfjorden originate from glacier calving in that area (see Figure 10).

It is hypothesized that the higher Chl-A concentration, which increases with the density of icebergs, is related to the icebergs themselves. Icebergs originating from glaciers carry minerals and nutrients into the ocean, which could promote algal growth and thus lead to higher Chl-A concentrations. The freshwater inflow from these icebergs could also lead to ocean stratification, resulting in less ocean water mixing and thus a more stable Chl-A layer on the ocean surface, which can be detected by remote sensing applications such as the payload presented in this work ([67,68] (Ch. 4)).

Figure 27 shows the spectra with maximal and minimal Chl-A content (according to SAM), as well as the mean spectra ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ as calculated using (80) for flight 8.

$${\mathit{\rho }}_{\mathrm{RS}\mathrm{mean}}\left(\lambda \right)={\displaystyle \frac{1}{c·r}}\sum _{i=1}^c\sum _{j=1}^r{\mathit{X}}_{\mathrm{IS}i,j}$$

(80)

8.3.2. Nonlinear Spectral Unmixing, and Data Reduction

Water that leaves a radiance reaching the imaging spectrometer is typically consists of multiple materials, such as pure water molecules or Chl-A molecules. One cannot expect that there is a pixel with light that has only interacted purely with one material (e.g., Chl-A). Therefore, the assumptions required for a linear spectral unmixing algorithm are not met ([69,70] (10.9.2)). A more complex non-linear spectral unmixing approach, such as NMF, could be chosen. The NMF algorithm used here is thoroughly documented in [71].

NMF divides the flattened hyperspectral datacube ${\tilde{\mathit{X}}}_{\mathrm{IS}}\in {\mathbb{R}}^{\mathrm{P}\times \mathrm{b}}$ into two positive semidefinite matrices, a matrix containing the spectral endmembers $\mathit{H}\in {\mathbb{R}}^{\mathrm{P}\times \mathrm{e}}$ and a matrix containing the abundance of these endmembers $\mathit{W}\in {\mathbb{R}}^{\mathrm{e}\times \mathrm{b}}$ (81).

$${\tilde{\mathit{X}}}_{\mathrm{IS}}\approx \mathit{W}·\mathit{H}$$

(81)

Solving this problem requires finding the minima of the non-convex cost function shown in (82).

$$\begin{array}{cc}\hfill & min{\parallel {\tilde{\mathit{X}}}_{\mathrm{IS}}-\mathit{W}·\mathit{H}\parallel }_{\mathrm{F}}^2\hfill \end{array}$$

(82)

$$\begin{array}{cc}\hfill & \mathrm{s}.\mathrm{t}.\mathit{W}\ge 0,\mathit{H}\ge 0\hfill \end{array}$$

(83)

The flattened hyperspectral datacube ${\tilde{\mathit{X}}}_{\mathrm{IS}}$ is a two-dimensional representation of the three-dimensional datacube ${\mathit{X}}_{\mathrm{IS}}$, where $\mathrm{P}$ stands for the number of pixels and $\mathrm{b}$ for the number of spectral band. The superscript $\mathrm{e}$ stands for the number of extracted endmembers.

NMF can be used to find different spectral categories (endmembers) in the investigated data. A subsample of $\mathrm{10,000}$ ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ spectra of the flight 8 data was used to train the NMF algorithm. The resulting NMF transformation was applied to the complete ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes of flight 1, flight 5, and flight 8. This approach ensures the comparability of the NMF output between the different flights.

The NMF algorithm was configured to result in $\mathrm{e}=4$ spectral endmembers. The abundances of these endmembers are shown in Figure 28 for flight 8 and in Figure A9 and Figure A10 for flight 1 and flight 5, respectively.

These endmembers represent the most likely different concentrations of components dispersed or dissolved in the water. The exact chemical composition leading to the obtained spectra must be determined by in situ sampling. However, similar patterns are visible in the SAM results for flight 8. This strengthens the hypothesis that the ocean color variation in these images is heavily influenced by iceberg meltwater.

8.4. Satellite-Based Imaging Spectroscopy

On 28 May 2022, the HYPSO-I satellite was passing over Kongsfjorden and the area surveyed by the UAV. HYPSO-I was equipped with a comparable imaging spectrometer to the UAV payload presented in this paper [72].

For the following analysis, we utilized data from the satellite pass occurring on 28 May 2022 11:25 UTC+0. Spectral and radiometric calibration and atmospheric correction were done using the software hypso-package [73]. The radiometric transfer code 6SV1 [74] was chosen as the atmospheric correction algorithm. Corrupted bands were removed from the datacube and a PCA was used to create a land and cloud mask.

The resulting datacube formed the basis for further analysis, consisting of a SAM with the same settings as the UAV imaging spectrometer and a NMF where all spectra were used in the optimization function. Figure 29 shows the georeferenced RGB composite overlaid over the Svalbard coastline. Figure 30 and Figure 31 represent the SAM results with the endmember shown in Figure 25 and the ${\theta }_{SA}$ variation from the mean spectrum in the datacube, respectively. Figure 32 shows the NMF endmembers. The color coding for these figures represents the maximal and minimal value for the corresponding data product.

When comparing Figure 32c,d with the RGB image in Figure 29 it becomes evident that the patterns which can be seen in these two last NMF plots correlate to the cloud distribution in the RGB image. Further analysis would have to be done to reduce the cloud impact in Figure 30 and Figure 31 further. However, Figure 30 and Figure 31 show that there is significant ocean color variation in the wider area of north Svalbard, the local variations seen in the UAV remote sensing images Figure 24 and Figure 26 are lost in this larger view.

Figure 33 shows the SAM results in the Ny-Ålesund area on the left and the K3 sampling area shown in Figure 19 is shown as a red box. On the right side of Figure 33 the K3 sampling area is zoomed out. When comparing the right-hand side of Figure 33 with Figure 26 it is visible that both images show a similar trend (more Chl-A to the upper right). In Figure 34 the spectrum with maximal Chl-A concentrations from the UAV mission (blue) and the HYPSO satellite (red) are compared. It is visible that the oxygen (O₂) absorption band around $\lambda =759 \mathrm{n}\mathrm{m}$ for the satellite spectra goes in the wrong direction. This likely has to do with an inaccurate atmospheric correction in the satellite data.

The peak values of the spectrum and the shape of the spectrum match. At higher wavelengths, however, the two spectra deviate from each other. This can most likely also be attributed to inaccurate atmospheric correction, as well as second-order optical effects above 750 $\mathrm{n}$$\mathrm{m}$. Icebergs and other highly reflective materials that are smaller than a satellite pixel are not removed by the land mask. However, due to their higher reflectivity, these materials have an above-average influence on the measured ${\rho }_{RS}\left(\lambda \right)$. To validate this hypothesis, it would be necessary to create a more refined landmask and employ various subpixel hyperspectral target detection algorithms to find the spectral endmembers resulting in the measured ${\rho }_{\mathrm{RS}}\left(\lambda \right)$.

8.5. In Situ Measurements

In situ measurements are a crucial part of remote sensing and imaging spectroscopy missions that should not be underestimated. These measurements provide essential ground truth data needed for validating remote sensing data. They furthermore contribute vital reference points for calibrating remote sensing data parameters.

In situ measurements were conducted during the field campaign described in Appendix F. These measurements were limited to several sampling areas. Among these is the sampling area shown in Figure 19, (Table A4). The in situ Chl-A measurements relevant to this mission are listed in

Table 9. In situ measurements (credit: Glaucia Moreira Fragoso, Sanna Kristiina Majaneva, Janina Osanen, Nathalie Summers).

Date/Time [dd Month yyyy HH:mm]	Chl-A Concentration [μg/L]	Depth [$\mathbf{m}$]	Repetition [-]
22 May 2022	1.52569	0.0	1
22 May 2022	1.91727	0.0	2
22 May 2022	1.81959	0.0	3
26 May 2022	0.09681	0.0	1
26 May 2022	0.11151	0.0	2
26 May 2022	0.08817	0.0	3
27 May 2022 18:45	0.57906	0.0	1
27 May 2022 18:45	0.60344	0.0	2
27 May 2022 18:45	0.69489	7.0	1
27 May 2022 18:45	0.70043	7.0	2
27 May 2022 18:45	0.49050	7.0	3

. Measurements for the dates 22 May 2022 and 26 May 2022 were derived from surface probes using a fluorometer as described in ([75] (Ch. 2.2)) or ([76] (Ch. 2.5)). For 27 May 2022, water samples from various depths were analyzed. These water samples were analyzed using the High-Performance Liquid Chromatography (HPLC) method [77].

Analysing the values in Table 9, it is evident that the ocean during the field campaign was in “pre-bloom” condition (Chl-A concentration < $40 \mathsf{\mu }\mathrm{g}/\mathrm{L}$) and that the biological conditions varied over the duration of the mission. From Figure 24, Figure A7 and Figure A8, it is evident that the Chl-A concentration varied not only on a large scale within the fjord but also locally, forming distinct patches even within the sampling area. The ocean color variations observed in the NMF plots—Figure 28, Figure A9 and Figure A10—exhibit similar patchiness.

Closer temporal and spatial alignment of in situ measurement and remote sensing data are essential to map the Chl-A concentration derived from in situ measurements to remote sensing data values. Local variability in the measured in situ Chl-A concentration, such as the low Chl-A concentration on 26 May 2022, could partly be explained by the patchiness of the Chl-A concentration within the sampling area. This underscores the fact that a comprehensive understanding of biological and physical conditions can be achieved neither through remote sensing techniques nor in situ measurements alone. Both approaches are necessary for a truly comprehensive perspective.

9. Summary and Conclusions

A total of eight flights were performed in Ny-Ålesund/Svalbard over Kongsfjorden and all flights are listed in

Table 10. List of all UAV flights (local time: UTC+1). The marked flight numbers are the ones relevant in this papers data analysis Section 8.

Flight No.:	Date [dd month yyyy]	Take-Off Time [Local] [HH:mm]	Flight Duration [min. ]	Altitude (Harbour) [m.a.s.l]
1	22 May 2022	13:11	32.31	300 (200)
2	23.05.2022	09:44	61.06	300 (150)
3	24 May 2022	18:02	52.08	300 (200)
4	24 May 2022	19:19	32.40	300 (200)
5	26 May 2022	10:19	48.37	300 (200)
6	26 May 2022	17:22	58.96	300 (200)
7	27 May 2022	10:51	55.95	300 (200)
8	27 May 2022	17:31	59.71	300 (200)

. The UAV took off and landed at the Ny-Ålesund airport and all flights took place during favorable weather conditions. A UAV control station was set up at the Ny-Alesund airport in the building in the background of Figure 35. More about this mission is explained in Appendix F. The flights that were analyzed for this paper are marked in blue in Table 10.

The payload constructed with COTS components which is presented in this paper functioned successfully during the Ny-Ålesund test campaign. The payload and the processing pipeline produced georeferenced ${\mathit{\rho }}_{\mathrm{RS}}\left(\lambda \right)$ datacubes with a high data quality, both spatially and spectrally. The direct georeferencing algorithm presented in Section 7 yields good results in steady flight conditions. However, further improvements are desirable to improve image quality during turbulent flight conditions (see Section 9.2). Section 8 shows that the datacube’s spectral data quality allows for retrieving ocean properties for relevant scientific applications, such as Chl-A detection.

When comparing the flights with the in situ measurements in Table 9 it becomes obvious that the conditions in the fjord changed significantly between flight 1 and flight 8. This is reflected in the spectral angle mapper data as well as the NMF data shown in Section 8. The comparison of satellite and UAV data shows promising results. Section 8.4 shows that hyperspectral images from a wider area add valuable information to an imaging spectroscopy mission conducted with a UAV. Information gathered this way helps to put the UAV remote sensing data in a bigger context. However, as mentioned in the last section of Section 8.4, a more thorough investigation of the spectral endmembers would be required. Furthermore, when studying a dynamic environment, such as the biological system in a fjord, better temporal alignment of satellite and UAV data would be desirable.

Furthermore, the results in Section 8.4 show that atmospheric correction of satellite data might be inaccurate, while on the other hand, UAV data apparently provide more correct ${\rho }_{\mathrm{RS}}\left(\lambda \right)$ spectra (when compared with [30]) as well as an obviously better spatial resolution.

9.1. Conclusions

Based on the results presented in this paper, it is evident that push-broom imaging spectrometers built with COTS components and data processing pipelines as presented in this paper are eligible for scientific use. With further design enhancements and an improved post-processing pipeline, such devices can compete with commercial imaging spectroscopy solutions. To achieve this goal, further enhancements in the imaging spectrometer and payload design are necessary, as well as further improvements in the post-processing pipeline (Section 9.2).

The SAM and NMF approaches presented in Section 8.3.1 and Section 8.3.2 are not able to quantify physical Chl-A units within the water. To derive physical Chl-A units an Ocean Color (OC) algorithm such as that described in [78] is needed. Such an algorithm, however, requires multiple field campaigns with in situ measurements to find the sensor-specific calibration coefficients used in these algorithms.

9.2. Future Work

The field-testing of the presented payload not only proved its capability of generating useful results but also revealed weaknesses in both payload design and post-processing techniques. The payload design would benefit greatly from an industrial, high-resolution monochrome or RGB camera which allows for tighter control of its system parameters. It would furthermore be very beneficial if its images could be timestamped by the SentiBoard. Additional sensors, such as a pressure, temperature, and humidity sensors could furthermore contribute to a more holistic picture of the sensed environment.

The imaging spectrometer design would also benefit from a wider $\mathrm{FOV}$ and a better smile and keystone calibration. Direct downwelling irradiance measurements would increase the versatility of the payload by allowing direct remote sensing reflectance calculation and the possibility to adapt to different lighting conditions.

While the georeferencing algorithm results in good direct georeferencing results during long, straight transects, its accuracy decreases rapidly when the UAV flies using dynamic maneuvers. This can become problematic when remote sensing operations are conducted in unfavorable wind conditions. The state-estimation algorithm has been identified as a major reason for this degradation. A state-estimation algorithm that better handles non-linearities and error sources during dynamic maneuvers is expected to yield better results. Smoothing using Factor Graph Optimization (FGO) or similar means would increase the state-estimation accuracy in post-processing, especially in states that are affected by nonlinear flight maneuvers. Further improvements could be made in the georeferencing algorithm with non-rigid pixel registration, which allows for the co-registration of imaging spectrometer spectra and the RGB camera pixels, similar to [79].