2.1. System Design and Sensors
A detailed discussion on the theoretical principles for underwater HI applications can be found in [
44] and an extension of such theory from an under-ice perspective can be found in [
4]. Here we only discuss the aspects that have driven the design of the under-ice system.
Depending on the camera settings and the desired aims, HI sensors can capture features at different scales ranging from millimeter close-range imagery to continuous swaths of data at the mesoscale. The mapping scale is determined by the sensor distance from the target and the mounting platform. Hyperspectral images are required to be orthorectified to enable extraction of meaningful and accurate metric information of the feature of interest (e.g., distances, shapes, and areas). This is ultimately necessary to compute the biochemical properties of the target [
28], and to allow for accurate repeat surveys and co-registration with other datasets.
The modality in which the frame is acquired can be in either pushbroom or as 2D snap-shot imagers. Pushbroom HI line scanners are optimal when it comes to cover large surfaces under dynamic conditions as spectral and spatial information are acquired at the same instance. Pushbroom HI also comes at the best compromise with respect to fundamental sensor properties such as image quality, sensitivity, spectral coverage, and spectral, spatial, and radiometric resolutions [
28,
45]. However, in order to compose a rectified pushbroom orthoimage, sensors are required to be moving relative to the imaged surface at precisely matched speeds, imaging frequencies (or frame rates), all whilst acquiring a highly stable attitude (pitch, roll, and heading) and distance from the target [
28,
37,
46]. Consequently, pushbroom HI is particularly sensitive when integrated onto dynamic platforms surveying under real environmental conditions and requires the full set of six-position (X, Y, Z) and orientation (pitch, roll, and heading) parameters (pose) assigned for every scan-line. An additional suite of sensors is therefore required to be integrated, and/or additional data products need to be included post-processing for robust HI geometric correction. These include highly precise Global Navigation Satellite Systems (GNSS)/inertial measurements units (IMUs), digital elevation models (DEMs), and orthomosaics of the imaged surface and/or a series of ground control points (GCPs) [
28].
Considering that light levels beneath sea-ice are typically very low, ranging from 0.1 to 10% of the incoming solar radiation, HI scans are forced to move at reasonably low speeds so that the signal to noise ratio (SNR) is maximized, requiring integration times and imaging frequency to be optimized (resulting in relatively long integration times and slow imaging frequency required for low-light levels). This makes HI imaging of transmitted under-ice radiance challenging for dynamic underwater conditions and future deployment onto platforms (e.g., ROVs) that are susceptible to continuous buoyancy, speed, drag, and currents adjustments. Also, under-ice navigation and positioning is far from trivial and/or comes at high costs.
The developed approach here aims instead to scan relatively smooth under-ice surfaces by sliding or “skiing” at a predefined fixed distance from the ice at precisely controllable speeds (
Figure 1). This enables the scanning movement to remain considerably stable, reducing some of the requirements aforementioned. The transect is prepared to be a pre-defined straight-line between 10 to 40 meters in length, limited in this prototype by the length of tether (
Figure 2). Ideally, the set-up is expected to permit stable scanning speeds matched to the low-light levels experienced and the need of pushbroom HI orthorectification to be suppressed (or minimized). To achieve a steady, slow, and controllable movement, two WG1500 manual worm gear winches (Dutton Lainson, NE, USA) were established at each end-point of the surveyed transects (
Figure 2). Stainless steel wires were attached from each winch to the respective end of the aluminum frame legs of the payload rig, which allowed the system to precisely slide back and forth through controlled winch rotations (
Figure 2).
Such a sliding concept is only possible on under-ice surfaces which are relatively flat—a common feature of land-fast sea ice in both the Arctic [
20,
47] and in Antarctica [
48]. Fast-ice is not only a relevant target for first tests of the technology, but also provides a relatively simpler optical set-up where algae are mostly residing at the bottom of the ice, at least during spring [
14]. Under rougher under-ice surfaces (e.g., pack ice, platelet ice, ice fissures, and cracks caused by pressure ridges or medium to large brinicles) the scanning advancement of the system result could be impeded with such a skies-based concept.
Figure 3 displays the core components of the internal payload that were fitted in the system enclosure. An overview of all sensors, equipment specifications, and their purpose for this first test can be found in
Table 1. Detailed information of technical design and software employed to operate the payload can be found
Appendix A. The
appendix also includes a schematization of power supply and data transmission paths from the surface elements to the enclosure interior and the external payload (
Figure A1).
To select an appropriate distance between the imaging sensors and the ice, we considered the trade-off between HI and RGB imaging specifications together with a series of environmental and logistical constraints (see [
4] for a trade-offs overview). For example, spatial resolution and image footprint are inversely correlated since increased distances from the ice yields a larger footprint at the cost of pixel size. Increasing the distance from the ice also enlarges the depth of field (DOF), which is an important factor to consider for close-range optical HI and RGB imaging applications. The DOF should be large enough to cover at least the sea-ice skeletal layer where most of the algal biomass is concentrated. Nonetheless, while gaining distance from the ice seems appealing to increase survey area, it increases logistical and technical problems which are relevant to the deployment of a large sliding platform beneath thick ice cover. Such problematics add up to the known effects of the water column on measured light intensity and spectral composition in the visible range [
49]. Overall, the increased costs of deploying optical sensors underwater need to be considered together with the additional challenges of geometric and chromatic correction of underwater images associated to the diverse refractive indices across the seawater-glass-air interface [
50,
51,
52]. Such aberrations are not trivial to correct and depend on multiple factors such as the sensors optical parameters and settings, deployment mode (e.g., distance from the ice and field of view (FOV) inclinations), water optical properties and the underwater housing lens design (e.g., flat versus dome), and material (e.g., thickness of the acrylic window).
For this prototype test, we found that an enclosure with a flat-port fitted with sensors separated approximately one meter from the ice would be a good compromise considering our equipment, deployment capabilities, and the spatial variability of the target (sea-ice algae) that we were surveying (
Figure 3). The custom-built and low-cost aluminum frame that set the distance from the sensor to the ice was approximately 1.20 ± 0.10 m in length (variable by changing the angle of the legs and steel clamps position). It also allowed the legs to be modified to any desired length if required (
Figure 1 and
Figure 2). The span between the 1.48-m-long skies ranged from 0.82 to 1.2 meters. It was confirmed that no components of the frame or skis interfered with the sensors FOVs and that FOVs of both sensors largely overlapped for coherent HI and 3D data interpretation.
Since the system travels at a fixed distance from the target, the horizontal and vertical footprint of the sensors can be estimated for the entire transect using standard imaging formulas (e.g., see Appendix B in [
24]). Nonetheless, a flat-port causes magnification of images due to the multiple refractions at the air-acrylic-water interfaces, thus reducing the apparent FOV [
51]. The amount of magnification is generally ≤1.33 and can be theoretically obtained using Snell’s law. However, such calculations are not straightforward and require a series of sensor optical parameters and sensor specifications, not always easily retrievable. Some include entrance pupil distance relative to the port and imaging object, underwater focus distance and port thickness, among others. To precisely calculate the sensor footprint on the ice, a simpler way is to image objects of known length from which we can retrieve pixel size and derive horizontal and vertical footprint thereafter.
Finally, it is important to consider that miniaturization of remote sensing payloads is always preferable but is inevitably associated with increased cost and/or complexity [
28,
29]. We must then consider logistical and technical constraints as significant factors that could impede the deployment of a cost-effective solution. It was also preferable to use commercially available and off-the-shelf components when possible, to foster ease of replicability. For example, it was considered mandatory for the system to be surface powered and to be able to stream data to operator and change sensors acquisition parameters based on observed circumstances in real-time. The latter is not straightforward, considering a large amount of data is generated over the multiple high-frequency imaging processes. Costly underwater fiber optic connectors and tethers were avoided by allocating an internal digital processing unit (DPU) within the enclosure, which directly interfaced with the multiple sensors and allowed for on-board data storage (
Figure 3). Power and communication with the surface was enabled through an ethernet/power cable permitting for virtual network computing (VNC). Altogether, these design features come at the cost of payload volume, and the entire payload was fitted into a cylindrical enclosure with an internal diameter of 0.23 m and a length of 0.6 m (
Figure 3).
The overall height of the system (including the frame legs and skis—
Figure 1) was approximately 2 m which required a well-regulated buoyancy to keep the system vertical and pushing against the ice with moderate upward pressure to allow for smooth scanning. This was achieved through modular buoyancy and ballast units that regulated the system’s vertical buoyancy and stability based on local conditions as displayed in
Figure 1.
One benefit of the system’s frame size was that it allowed the incorporation of external sensors in the future. For example, for our first tests, we included an upward-looking TriOS Ramses ACC-VIS spectrally resolved irradiance sensor near the ice-water interface to measure light directly exiting the sea-ice matrix (seen in
Figure 2 and specified in
Table 1).
2.3. Deployment and Data Acquisition
The AK10 only allows for manual focus, and the system does not currently have the capability for remote focusing. The focus distance was required to be set to the predefined scanning distance of the system of approximately 1.2 m. Nonetheless, we need to consider that the focal distance and DOF have the potential to change underwater under a flat port set-up to ultimately affect image sharpness. We, therefore, used an underwater focusing target immerged in the ice hole together with a dummy acrylic glass port to focus the camera under dry conditions while mimicking the underwater optical set-up. The Sony a6300 interface allowed for remote autofocus.
We selected sunny and completely cloud-free days for our deployments to maximize under-ice transmitted light (and thus HI SNR). Before deployment, the enclosure was vacuumed using a standard vacuum pump and PREVCO vacuum kit manifold assembly to an internal pressure of −15 in.-Hg in gauge for leak testing and to reduce internal condensation risks. Although the air in Antarctica is typically very dry, this process is important to avoid any condensation within the enclosure due to the considerable heat produced by sensors and equipment compared to the exterior temperature.
Due to its voluminous shape and weight, the system required two to three people to be manually deployed into the ice hole. The system was then manually pushed below the 1.8 m thick sea ice by two people using rods inserted into the incorporated cradles (see
Figure 1 and
Figure 4b). The system can then be rotated into the desired transect direction (e.g., western).
Once under-ice, the system was winched three to four meters away from the hole and the tent to avoid interference in the light conditions beneath the ice. We were able to speed up the worm gear winches (designed to be slow for data acquisition) using a winch adapted electric drill as seen in
Figure 4d to move the system into the right position for data collection. An initial assessment of the HI signal intensity from directly under-ice was then performed. The optimal traveling speed and HI and RGB imaging settings were then maximized for both SNR and image quality.
The AK10 data storing and imaging settings, including integration time, imaging frequency, spatial, and spectral binning were controlled in real-time using the Lumo Recorder software (Specim Spectral Imaging, Oulu Finland). For HI, the spatial and spectral dimensions were binned to 1024 spatial pixels across track, and a spectral resolution of 3.5 nm (178 bands), respectively. Whilst the spectral dimension could have been further binned to 7.5 nm for increasing the signal; this was avoided as too coarse spectral resolutions are known to hamper the application of some of the HI processing methods for ice algae [
27]. The HI frequency was set to 10 Hz and an exposure time of 99 ms (maximum setting available). The ideal sled system speed for these settings was found to be around 0.008 ms-1 corresponding roughly to one rotation of our worm gear winch per second. The read-out frequency of the IMU was also set to 10 Hz aiming for HI and IMU data time-stamp synchronization at the decisecond (ds) level. The survey distance of 1.18 m between the HI sensor and the ice resulted in a HI footprint width on the ice of approximately 0.61 m and a pixel size of 0.00625 m. The Lumo Recorder software was programmed to acquire 100 samples of a dark frame image with the shutter closed at the end of each acquired hyperspectral image. Dark frame images were taken for the subsequent radiometric correction of the imagery through the removal of dark current noise.
The Sony a6300 is operated through the Sony Imaging Edge software “Remote” feature. The software allows live streaming the camera view and permits exposure control, ISO, time-lapse shooting interval, and AF settings to be modified. We found that at the selected winch speed, an imaging interval of 0.1 Hz was sufficient to guarantee abundant forward overlap (>90%). This relatively large sampling interval, together with the slow movement allowed the camera to be set to AF, which resulted in sharp and focused images. The ISO was set to 250; aperture maximized to f/2.8 and shutter speed set to 1/250 sec for most of the circumstances. The altitude of the camera was around 1.2 m, which yielded an estimated footprint width of 0.586 m in water and a resolution of 0.0001 m. All images were captured in the Sony RAW format (.ARW) to allow for any eventual image pre-processing approaches (e.g., see appendix in [
24]).
The radiometrically calibrated Ramses ACC-VIS was synchronized to acquire an under-ice irradiance sample at the same time as each Sony a6300 RGB image (0.1 Hz) was taken. In this way, it is possible to link every image to a Ramses ACC-VIS radiometric irradiance sample and locate images spatially across the transect through the retrieved camera positions following SfM digital photogrammetry.
The STS-VIS radiometer was set-up to acquire a measurement of incoming downwelling solar irradiance every minute considering the highly stable conditions during the surveys and the relatively low variability in sun angle.
Following system retrieval, HI, RGB imagery, and IMU navigation data files were downloaded directly from the SATA SSD within the DPU. VNC allows for direct data transfer from the payload to the surface, but the operation is time-consuming for large files such as the HI imagery data files.
2.4. Data Processing
Both hyperspectral image analysis and SfM photogrammetry are active research topics for many land-based applications. The adaptation of established terrestrial procedures to novel under-ice applications requires targeted studies aiming to identify, test, and evaluate their performance in an under-ice context. Here we present only preliminary data outputs of the developed system and assess their quality and potentials from a biophysical perspective. We do this by looking exclusively at the western transect and selecting a successful subsample for hyperspectral image analysis and processing (
Figure 5), namely block B. For the RGB imagery and photogrammetry, we retrieve for the first time a high-resolution orthomosaic and DEM of the under-ice using commercially available software. For HI, we adapt some of the known methods in under-ice bio-optical literature to the hyperspectral images and illustrate potential new ones.
2.4.1. RGB Imagery and SfM Digital Photogrammetry
It is well known that image quality and poor camera network geometries can considerably affect SfM model’s reconstruction and the extraction of accurate metric information. Image quality in non-metric cameras is influenced by the camera sensor, lens quality, mechanical stability, and the overall image acquisition process under dynamic conditions. Poor camera network geometry refers to the lack of forward or side overlap in the imagery and/or lack of oblique imagery. Underwater, SfM photogrammetry is further challenged when using flat-ports due to the multiple refraction processes that magnify FOV, affect the focal length and produce a series of geometrical (e.g., radial distortion) and chromatic aberrations in the images directly affecting camera calibration algorithms in SfM, which ultimately affect the reconstructed model.
While image quality, per se, was not considered problematic in our transect dataset, the flat port did cause non-negligible effects on the imagery (e.g., noticeable pincushion distortion). To solve such aberrations and obtain an accurate camera calibration one can formulate the complex mathematical models of the imaging process in water [
53,
54] or perform a rigorous camera calibration using underwater targets with precisely known geometry [
55]. Another option is to rely on camera self-calibration, which refers to the calibration process using only image point correspondences for large and well-composed datasets [
56,
57]. However, self-calibration is challenging in our dataset as camera network geometry is particularly weak when dealing with elongated strips with only nadir images and no side overlap and/or oblique imagery [
57]. Systematic errors produced in such datasets can cause bending and non-linear deformations in the photogrammetric models as confirmed by our tests [
57,
58]. Here we apply a simple preliminary solution to the camera calibration problem using a constrained self-calibration approach by taking advantage of the flat under-ice surface, the known transect lengths and a series of identifiable reference points that were also measured from above the surface.
Prior to photogrammetric processing, 733 Sony RAW images acquired for the western transect were first imported into Adobe Lightroom where an initial lens correction and manual batch compensation for pincushion distortion was performed. Lightroom considers camera lens profiles into its corrections, and this empirical “trial-and-error” approach is simply to partially reduce bending of the model to a near straight level. Duplicate images were discarded as labeled repetitions during sled idle times, and the remaining images were exported from Lightroom as .JPG files for further SfM processing.
The 3D reconstruction of the under-ice surface was created using Agisoft Metashape (previously Photoscan), is a software package which has been extensively used for 3D modeling and photogrammetry over a wide range of geoscience applications [
59,
60]. The workflows for under-ice DEM and orthomosaic generation are described here. Photo alignment accuracy was selected as medium (for computational reasons) and provided a first estimate of camera calibration parameters and the reconstructed scene. The produced sparse point cloud model at this stage was noticeably bent and deformed. We proceeded to filter outlier’s and low accuracy points using the gradual selection tools. Due to the smooth nature of the surface (
Figure 4c), we assumed that all the surface areas with little algal cover were level with a reference height of 0.0 m, and created a dense and well-distributed network of reference level markers with a Z position (altitude) 0.0 m. We also added the known transect length as a scale bar length reference together with a series of points that were identifiable and could be referenced to above surface positions whose relative position could be measured with a measuring tape. For our entire western transect, we allocated 32 of these reference points, termed ground control points (GCPs) [
61,
62].
All these level reference GCPs are assigned with a high marker accuracy of 0.002 m in Metashape reference settings options. The model is then processed using the optimization of camera alignment feature where non-linear deformations can be removed by optimizing the estimated point cloud and camera calibration parameters based on these known reference marker coordinates [
59]. During this optimization, Metashape adjusts estimated point coordinates and camera parameters minimizing the sum of reprojection error and reference coordinate misalignment error.
The Metashape workflow is then followed by dense cloud reconstruction (medium quality and aggressive depth filtering), 3D mesh from the dense cloud (Arbitrary surface type, medium quality, enabled interpolation, and aggressive depth filtering), texture mapping (orthophoto mapping mode and mosaic blending mode), and finally DEM and Orthophoto production. The scaled orthomosaic and DEM were exported in .TIF format to QGIS and the DEM was processed with a hillshade function for visualization purposes.
2.4.2. Hyperspectral Imaging and Radiometer Data
The retrieved HI images of block A and B consisted of a three-dimensional (x, y, λ) data cube where x and y represent the spatial dimensions, and λ the spectral dimension. The first two steps of the HI processing workflow include radiance conversion of digital numbers (DN) and pushbroom image rectification. The system was designed so that little to no geometric rectification and IMU data integration is required. This was the case for block A and B of the analyzed transect (
Figure 5).
Per-pixel radiance conversion was done using Specim Caligeo PRO software (Spectral Imaging, Specim Ltd., Finland) which addresses noise and geometric aberrations inherent to the sensor and performs the conversion of DN into downwelling spectral radiance Ld (λ, mW m2 sr−1 nm−1) using the in-situ acquired dark current frames and the associated calibration files. For the present study, spectral bands <400 nm and >700 nm were considerably noisy and outside the range of interest, therefore spectral subsetting was applied reducing the data to a total of 89 bands.
The block B HI subsamples are then smoothed using a Savitzy-Golay low-pass filter with a polynomial order of three and frame length of nine aiming to reduce noise in the transmitted signals without hindering the retrieval of fine spectral features [
63,
64].
Following this procedure, we adapted methodologies previously applied to track biomass variability from under-ice spectra such as normalized difference indices (NDIs) and principal component analysis (PCA) (also known as EOF) [
5,
17,
19,
27]. Every pixel within the HI subsample was integral-normalized to reduce the amplitude component of spectral variability and to focus on differences in spectral shape, a pre-processing standardization method previously applied in sea-ice bio-optical literature [
8,
19,
48].
PCA for hyperspectral remote sensing is typically employed for dimensionality reduction, to reveal complex relationships among spectral features or for the identification of prevalent spectral characteristics. PCA has been widely used in optical oceanography for extracting information about seawater constituents from spectral data (e.g., [
65,
66]). In our case, PCA was applied to the spectral dimension of block B data cube to explore and highlight the most variable features and relationships across all pixels in the block B image [
27,
67].
Spectral indices, such as NDIs, have been linearly correlated to the logarithm of sampled chl-a in multiple sea-ice studies [
5,
19,
48]. Since we have not developed a specific spectra-biomass relationship for our site that applies to the developed HI payload yet, a couple of identified optimal NDIs from the land-fast sea-ice of Davis Station and McMurdo Sound, Antarctica by [
48] were selected and utilized as a proxy of biomass. Before index implementation, block B was spatially binned to two by two pixels, reducing the spatial resolution from 0.624 mm to 1.2 mm, but boosting per pixel signal. The following NDI equation was then applied to every pixel in the image:
where λ
1 and λ
2 are wavelength bands selected across the sensor spectral range and L
d (λ, mW m
2 sr
−1 nm
−1) is the solar downwelling radiance transmitted through the ice. From [
48], we selected 441:426 nm and 648:567 nm as two different NDIs in different areas of the spectrum and applied the NDI equation to every pixel in the block B image. In this study, we used radiance to compute the indexes rather than under-ice radiance normalized to surface irradiance (or transflectance [
68]). Changes in above surface illumination conditions (e.g., solar geometry and atmospheric effects) within the block A and B image subsample were considered negligible.
In addition to adapting PCA and NDIs to under-ice HI, we also tested for the use of an index called Area under curve Normalized to Maximal Band depth between 650–700 nm (ANMB
650–700) of the continuum removed spectrum [
69]. ANMB
650-700 has been successfully applied for chl-a and chl-b mapping using HI of Norwegian spruce trees [
69] and Antarctic moss beds [
70], and here we use it as a proxy of chl-
a or ice algal biomass.
For this index, we applied the same Savitzky-Golay low-pass filter and the two by two spatial binning factor, but no integral normalization is performed. Instead, the entire image is normalized by the highest spectrum intensity within the block, which corresponds to an algal free cavity in the ice visible in the image (shown later in the results section). This provides a proxy of light transmittance over roughly the last 5 to 15 cm of ice bottom and enhances visibility of the absorption peak of chl-a at 670 nm of each pixel spectrum. The continuum removal transformation on the spectrum is a fundamental pre-processing step to enhance and standardize the specific absorption features of biochemical constituents [
71]. It allows for the normalization of the transmittance spectra so that individual absorption features can be compared from a common baseline. Following a localized continuum removal, we can calculate the Area Under Curve in the range between 650 and 700 nm (AUC
650–700) where chl-a attains one of its absorption peaks:
where
and
are values of the continuum-removed transmitted spectra at the j and j+1 bands,
and
are wavelengths of the j and j+1 bands, and n is the number of the used spectral bands. We can then calculate the ANMB
650-700 index as:
where MBD
650–700 is a Maximal Band Depth of the continuum-removed reflectance, generally at one of the spectrally stable wavelengths of strongest chl-a absorption around 670–680 nm. Normalization of AUC
650–700 by MBD
650–700 is a crucial step for strengthening the relationship between ANMB
650–700 and the chl-a content for higher chl-a concentrations. The logic behind this spectral index is exploiting well-known changes of the transmittance signature shapes produced within these wavelengths mainly by the changes in algal chl-a content.
In order to validate the robustness of the HI data compared to traditional means of acquiring under-ice spectra, hyperspectral irradiance variability measured with the Ramses ACC-VIS across the entire transect (samples shown as black dots in
Figure 5) was computed and compared with spectra of every pixel in block B. The Ramses ACC-VIS data further allow us to gain an estimate of downwelling irradiance intensity exiting the ice-water interface and was used to gain an insight of the light levels experienced under-ice. These can then be used to baseline the signal quality of the data achieved using our HI system under those specific conditions. The TriOS Ramses ACC-VIS was radiometrically calibrated using the factory provided calibration files (traceable within international standards) during the data acquisition process.