Coral Reef Monitoring by Scuba Divers Using Underwater Photogrammetry and Geodetic Surveying

Abstract

Underwater photogrammetry is increasingly being used by marine ecologists because of its ability to produce accurate, spatially detailed, non-destructive measurements of benthic communities, coupled with affordability and ease of use. However, independent quality control, rigorous imaging system set-up, optimal geometry design and a strict modeling of the imaging process are essential to achieving a high degree of measurable accuracy and resolution. If a proper photogrammetric approach that enables the formal description of the propagation of measurement error and modeling uncertainties is not undertaken, statements regarding the statistical significance of the results are limited. In this paper, we tackle these critical topics, based on the experience gained in the Moorea Island Digital Ecosystem Avatar (IDEA) project, where we have developed a rigorous underwater photogrammetric pipeline for coral reef monitoring and change detection. Here, we discuss the need for a permanent, underwater geodetic network, which serves to define a temporally stable reference datum and a check for the time series of photogrammetrically derived three-dimensional (3D) models of the reef structure. We present a methodology to evaluate the suitability of several underwater camera systems for photogrammetric and multi-temporal monitoring purposes and stress the importance of camera network geometry to minimize the deformations of photogrammetrically derived 3D reef models. Finally, we incorporate the measurement and modeling uncertainties of the full photogrammetric process into a simple and flexible framework for detecting statistically significant changes among a time series of models.

1. Introduction

Underwater photogrammetry has increasingly become a popular technique for three-dimensional (3D) mapping of subaquatic environments at different spatial scales and resolution. In underwater scenarios, several application domains can be identified, such as deep-sea exploration [1]), archeology [2], marine ecology [3,4,5] and sub-sea metrology [6], each of them with specific requirements and constraints. In marine ecology, photogrammetry acts as a game changer, allowing ecologists to obtain non-invasive, objective measurements of the underwater habitat structure and complexity, crucial in understanding the health and temporal changes of benthic communities. Numerous examples in the literature describe research and experiments that have utilized photogrammetry carried out by scuba divers [7,8,9], as well as by remotely operated or autonomous underwater vehicles [1]. Metric characterizations and monitoring of coral reefs have greatly benefitted from the growing availability of affordable and easy-to-use hardware and software tools [10]. This has permitted researchers to expand the range of studies and opened new research opportunities in marine ecology. However, because many systems are used in a black-box mode, there is the risk that the results obtained with photogrammetry may provide unreliable evidence if the technique itself is not properly implemented and quality control procedures are not properly executed.

Several studies have examined factors that influence photogrammetry in underwater environments. First, water turbidity may significantly affect the image quality and light absorption in water, thus influencing the colors present in images and leading to un-sharpness [11]. Reflections of the sunlight on the water surface may influence the image quality by producing a moving light pattern on the bottom, which varies from image to image. Other factors which may degrade image quality include motion blur caused by the scuba diver moving too quickly over the target substrate and the presence of floating vegetation on the water’s surface that blocks the sunlight and can cause color shifts. The presence of an underwater optical port (spherical, hemispherical or flat) in front of the lens alters the image formation geometry, introducing optical aberrations [12] and, in the case of flat ports, also introduces refractions/distortions, which translates into a departure from the classic photogrammetric mathematical model [13]. The establishment of highly accurate geodetic networks underwater to serve as known references is crucial for monitoring and change detection tasks, and still represents a demanding challenge underwater [14,15,16,17]. Recognizing the importance of assessing the precision and accuracy potential of photogrammetrically derived models, empirical approaches have been employed to characterize measurement errors [18], reliability over time [19] and accuracy [3] of natural and artificial coral reef models of different sizes and structural complexity.

This paper collects the experience gained in coral monitoring within the Moorea Island Digital Ecosystem Avatar (IDEA) project [20]. Based on preliminary studies [9,14] and expanding the results presented in [21], here, we critically revise and expand all the steps of the developed photogrammetric approach for coral reef temporal monitoring. In the authors’ knowledge, it is the first time that rigorous surveying methods are adopted in this domain. The critical need for accurate and stable reference points for assessing changes in the range of a few centimeters per year for environmental monitoring purposes is stressed, and a thorough analysis of several photographic systems for underwater photogrammetric monitoring of coral reefs is presented. Moreover, moving from previous studies by [22,23,24], we propose a simple and flexible framework to characterize the quality of the derived reef models and to assess the confidence or significance level of the changes detected through time. The method is founded on formal principles of surveying and error propagation theory and is implemented adopting state-of-the-art algorithmic solutions.

The paper is divided into four main sections, which cover the main steps of the implemented photogrammetric procedure described in Figure 1. Section 2 describes the Moorea IDEA project, its motivation and requirements, with an overview of the procedure developed for an accurate four-dimensional (4D) monitoring and change detection of Moorea coral reefs. Section 3 focuses on a comparative quantitative analysis of different underwater camera systems, with the aim of investigating the accuracy potential of high-quality off-the-shelf (i.e., digital single-lens reflex—DSLR and mirrorless interchangeable lens cameras—MIL cameras) as well as low-cost systems (action cameras). It also includes tests on color reproduction and the analysis of image quality underwater. The results from the photogrammetric bundle adjustment are also discussed. Section 4 delves into the critical topic of camera network to control the photogrammetric model deformation. Finally, the issue of surface model generation and comparison of 3D models for four-dimensional (4D) monitoring or change detection are addressed in Section 5. Although the last block, surface texturing and orthophoto generation, is still an integral part of our procedure, it is not of interest for the current investigation and therefore not further investigated here.

2. Four-Dimensional (4D) Monitoring and Change Detection of Moorea Coral Reefs

The current study is a component of the larger Moorea IDEA project, undertaken by an inter-disciplinary and international team of researchers, aiming at digitizing an entire island ecosystem at different scales, from island topography to microbes [25,26]. Within this broad context, underwater photogrammetry is carried out at different epochs to provide not only a digital representation of the underwater ecosystem, but also to add time as the fourth dimension to the classic 3D representation. The multi-temporal modeling approach constitutes the base to study how physical, chemical, biological, economic and social processes interact.

The IDEA project is linked to the Moorea Coral Reef Long-Term Ecological Research (MCR LTER) program, established by the US National Science Foundation in 2004 as a model system to better understand factors that mediate coral community structure and function. In particular, an essential objective of the MCR LTER program is to investigate the impacts of eternal drivers such as the influence of global environmental change on the viability of coral reefs. This objective is accomplished through long-term observations, process-based experimentation and bio-physical modeling efforts.

The MCR LTER project has established several permanent sampling locations around the island of Moorea in the three primary coral reef habitats, i.e., fringing reef, back reef and fore reef. The underwater photogrammetry monitoring project focuses on sites in the fore reef and fringing reef habitats only. Here, we report studies on plots of varying dimensions (ranging from 5 × 5 m to 16 × 8 m and 50 × 10 m) in the fringing reef and on the fore reef (Figure 2), at different depths, ranging from 3 m on the fringing reef up to 15 m on the fore reef.

Primary and Secondary Control Networks for Coral Reef Monitoring

Accurate reference networks are required for environmental change detection and monitoring. They are crucial when the variations to be measured are in the range of a few centimeters per year, typical of highly dynamic environments such as oceanic coral reefs, where 3D landscape elements are continuously changing over time. Corals may grow or shrink, sand can be deposited or dispersed and nonliving hard substrates can be eroded. Scuba divers (experts or tourists) and underwater vehicles may themselves cause changes to the reef architecture, for example dislodgement of coral or other substrates, while operating.

For the Moorea IDEA coral reef monitoring project, each plot is equipped with a permanent network of reference points, which serves the purpose of providing a datum or reference frame to the photogrammetric models measured over time. This permanent network consists of a primary and a secondary network. The points forming the primary control network (primary reference points, PRPs) are measured through classic geodetic techniques (trilateration and leveling). Their coordinates define the datum for the first photogrammetric epoch (or baseline epoch) and, also, provide an independent accuracy check. Conceptually, they should not be subject to change over time. The coordinates of the points belonging to the secondary network (secondary reference points, SRPs) are estimated in the baseline photogrammetric processing (first epoch) and, together with the photogrammetrically estimated primary reference points, are used to transform the successive photogrammetric models in a common reference system. Typically, in a plot of 25 m², five primary and four secondary reference points are installed. The number and distributions of primary reference points are often not optimal from a surveying point of view but are dictated by measurement equipment and environmental constraints. Figure 3 shows two larger plots with more reference points. The required accuracy of the permanent reference network is quite high. To enable the measurement of temporal changes on the order of cm-range, the point accuracy (especially in height) should be a few mm.

The reference points, for both the primary and secondary control networks, consist of stainless-steel, threaded, expansion anchors inserted and cemented into the coral reef matrix to a vertical depth of approximately 40 mm to assure stability over time. Plastic bolts are normally placed into the anchors to prevent the growth of marine organisms from covering the reference points between photogrammetric measurement epochs. When measurements are performed, 9.5 mm diameter, 30 cm high stainless-steel poles are screwed into the anchors. The height of the poles is such that the visibility between the points is guaranteed, notwithstanding the reef topography. Special headers are mounted on top of the poles to perform point-to-point direct and reverse distances measurements using a metal tape with millimeter graduation. The headers are replaced with planar photogrammetric coded targets during the image acquisition step. The geometric levelling process requires a millimeter-scale graduated rod (also known as level staff) that is held vertically on the point anchors using a specifically designed adapter. An underwater green laser pointer is mounted onto a surveyor’s tripod equipped with a standard tribrach bubble level and an ad-hoc mounting head which allows the laser to be rotated around both the vertical and laser optical axes. The tripod is positioned halfway between two primary points, and for each point, on the same staff, two elevation readings are recorded with the laser rotated 180° around its axis between the two readings. The mean value is retained, thus removing any systematic angular offset between the laser optical axis and the mounting head. Further details on the design and fabrication of the specialized equipment can be found in [24].

Geodetic network adjustment can be performed following a minimal constraint [24] or free network solution approach. Here, we opt for the free network solution approach, which provides optimal results in terms of inner coordinate accuracy, minimizing the mean variance of point coordinates. The computational procedure follows three steps: (i) a first approximate solution is computed with minimal constraints [14], (ii) the free network solution is computed and (iii) a rigid 3D Helmert transformation is estimated between solutions (ii) and (i) to define a consistent datum. The raw observations, i.e., point-to-point direct and reverse distances and mean height readings, are first checked to detect outliers and then are input in the adjustment process. We used two software solutions, which provided statistically comparable results, Trinet+ [27] and GAMA V. 1.12 [28]. Some problems involved in underwater geodetic network establishment and their relation to photogrammetry are addressed in more detail in [14].

Table 1. Moorea Island Digital Ecosystem Avatar (IDEA) coral reef monitoring. Shown are average standard errors in planimetry (σXY) and height (σZ) for the primary control network in free network solutions. The number of measured primary reference points (#PRP), degree of freedom (DOF), average depth and depth range values are also reported.

	16 × 8 m Plot—Fringing Reef	5 × 5 m Plots—Fore Reef 1	20 × 5 m Plot—Fore Reef	50 × 10 m Plot—Fore Reef
#PRP	13	5	11	32
DOF	78	16	51	175
σXY	3.2 mm	2.3 mm	4.4 mm	5.4 mm
σZ	2.6 mm	2.0 mm	3.0 mm	4.2 mm
Average Depth	5 m	10 m	10 m	12 m
Depth Range	3 m	2 m	1 m	4 m

¹ The average values for five 5 × 5 m plots are shown. #PRP means the number of PRP.

reports the obtained average standard errors in planimetry and height for plots of different dimensions, e.g., five 5 × 5 m, one 20 × 5 m and one 50 × 10 m in the fore reef, and one 16 × 8 m in the fringing reef. Figure 4 shows the graph of the most challenging primary control network, i.e., for the 50 × 10 m fore reef plot.

3. Comparison of Diver-Operated Underwater Photogrammetric Systems

One of the core activities in the Moorea IDEA coral reef monitoring project has been the evaluation and comparative analysis of the accuracy and metric performances of different underwater photographic systems, with the aim of identifying the best solution for use in coral reef documentation and monitoring purposes. In this context, we were aiming at two different systems: (a) a high-end solution that will yield the best possible attainable accuracy, and (b) a low-cost components-based system that can be handled easily and is affordable for everyone.

In underwater photogrammetry, the imaging system is composed of four main components: (i) the camera with its sensor, (ii) the optical lens, (iii) a waterproof housing and (iv) an underwater port installed on the housing. Each of these components plays an important role in the achieved quality of the acquired images and, as a whole, contributes to the overall system stability, something which is vitally important photogrammetrically.

With this in mind, we tested both high-quality commercial off-the-shelf (COTS) and low-cost (action cam) underwater camera systems on one of our 5 × 5 m fore reef plots (Figure 5). We used both digital single-lens reflex (DSLR) and mirrorless interchangeable lens (MIL) cameras in our tests of the high-quality COTS digital camera systems. Ultimately, we chose to include MIL cameras in this study as these types of cameras have gained a lot of attention in the underwater photographic world for their ability to fill the gap between the heavier and more expensive DSLRs and the lighter, lower quality, compact digital cameras. As a low-cost alternative, we experimented with GoPro^®, which have become very popular among marine ecologists for their affordable price, ease of use and portability. Lastly, we investigated both single and multi-camera systems.

Images were collected by scuba divers, operating the following camera systems (Figure 6,

Table 2. Characteristics of underwater photogrammetric systems tested.

System Acronym	N750	N300	PL51-PL52	PL41	5-GoPro (GoPro41 to GoPro45)
Camera Type	DSLR	DSLR	MIL	MIL	Action cam
Camera Body	Nikon D750	Nikon D300	Panasonic Lumix GH5S	Panasonic Lumix GH4	GoPro Hero 4 Black edition
Sensor Type (Dimensions (mm))	Full frame (35.9 × 24)	APS-C (23.6 × 15.8)	Four thirds (17.3 × 13)	Four thirds (17.3 × 13)	1/2.3 inch (6.17 × 4.55)
Pixel Size (um)	6.0	5.6	4.6	3.8	1.5
Image Size (pixel)	6016 × 4016	4288 × 2848	3680 × 2760	4608 × 3456	3840 × 2160
Lens (For zoom lenses, focused at focal length)	Nikkor 24 mm f/2.8 D	Nikkor 18–105 mm f/3.5–5.6 (at 18 mm) with +4 diopter 1	Lumix G 14 mm f/2.5	Olympus M. 12–50 mm f/3.25–6.3 (at 22 mm)	3 mm
Underwater Pressure Housing (material)	NiMAR NI3D750ZM (polycarbonate)	Ikelite 6812.3 iTTL (polycarbonate)	Nauticam NA-GH5 (aluminum)	Nauticam NA-GH4 (aluminum)	GoPro housing (polycarbonate)
Port lens (material)	NiMAR NI320 dome-port (acrylic)	Ikelite 5503.55 dome-port (acrylic)	Nauticam N85 3.5” wide-angle dome-port (acrylic)	Nauticam N85 Macro Port (glass) with Wet Wide-Lens 1 (WWL-1) dome-port (glass)	Flat with red filters (glass)

¹ The +4 diopter allowed the camera to properly focus underwater at the shortest focal length (i.e., at 18 mm).

):

• PL41: Panasonic Lumix GH4 (MIL)
• PL51-PL52: stereo system with two Panasonic Lumix GH5 (MIL)
• N750: Nikon D750 (DSLR)
• N300: Nikon D300 (DSLR)
• 5-GoPro: 5-head camera system with GoPro cameras named GoPro41 to GoPro45, where GoPro45 is the nadir looking camera.

All the camera systems were tested at two different heights, or working distances, above the reef at 2 and 5 m (except for the D300, which was used at a working distance of 2 m only) with the additional goal of investigating distance-dependent-induced errors and assessing the accuracy of the photogrammetrically derived products.

Table 3. Characteristics of underwater photogrammetric systems tested.

	N750	N300	PL41	PL51	PL52	GoPro45
Working distance: 2 m
GSD (mm)	0.5	0.6	0.6	0.8	0.7	1.2
Number of images	304	581	451	523	523	431
Working distance: 5 m
GSD (mm)	1.2	-	1.4	1.4	1.4	2.2
Number of images	101	-	139	166	166	430
Mean intersection angle (degrees)	27.9	-	30.4	24.9	25.5	27.8

summarizes some camera network parameters for the different camera systems and the two working distances. Although slight variations in camera networks (ground sample distance - GSD as well as exterior orientation parameters) were present, we observed a high degree of consistency in our metrics of camera performances among the different photogrammetric systems used. Figure 7 shows a camera network typically implemented over the plot with cross-strips and oblique views at 2 m working distance.

3.1. Camera Systems’ Set-Up

The camera settings were selected to maximize the image quality for the different cameras, while taking into consideration that the images had to be taken by scuba divers. Tests were carried out underwater using resolution and standard, commercially available photographic color reference cards for a quantitative evaluation of image quality [29]. Our goal was to verify that the image quality was homogenous across the different sensor format or, failing that, to provide the information necessary to allow the formulation of a proper stochastic model that would weigh the image observations according to their quality. The standard photography parameters to set are the three components of the so-called exposure triangle (Figure 8a), i.e., aperture, shutter speed and sensor sensitivity - ISO. In air, when dealing with close-range photogrammetry, the aperture setting can be extremely critical to properly adjust the depth of field (DoF) to the subject of interest so that all parts of the subject image are reproduced with an acceptable degree of sharpness. In water, especially when using dome ports, the DoF is significantly increased [12], making this aspect less critical. However, image quality is still heavily influenced by the chosen aperture (f number) and, consequently, the proper value must be selected based on the chosen combination of lens and housing port to minimize the introduced optical aberrations (diffraction, field curvature, spherical aberrations, etc., [29]). When acquiring the images from a moving platform, such as a swimming diver, shutter speed controls the motion blur: according to the expected motion (swimming) speed, v, the shutter speed, t, needs to be set so that the displacement, s, during the exposure time is less than the ground sample distance (GSD, Figure 8b). The ISO value controls the sensor’s sensitivity to light: the higher the number, the less light is needed to achieve a correctly exposed image for a given aperture and shutter speed. In other words, increasing the ISO will amplify the light signal allowing faster shutter speeds under lower light conditions, but at the cost of increasing the image noise or grain.

In practice, setting an adequate shutter speed is often problematic, because divers often operate underwater in low-light conditions, a red filter may be used to lower the portion of the bluish wavelengths which further reduces the amount of light available and the platform (diver) may make uncontrolled, fast movements with the camera.

With these premises, we chose to use the values summarized in

Table 4. Characteristics of underwater photogrammetric systems tested.

		N750	N300	PL51 | PL52	PL41	5-GoPro
Acquisition Mode		Single shot	Single shot	Time lapse @ 2 s shooting interval	Single shot	Video @ 30 frame per seconds (Field of view = wide)
Original Images/Video Format		raw	raw	raw	raw	MP4
Exported Images/Extracted Frames Format		JPG @ highest quality	JPG @ highest quality	JPG @ highest quality	JPG @ highest quality	PNG (then converted to JPG @ highest quality)
Shooting Mode		Aperture priority	Aperture priority	Aperture priority	Shutter priority	-
Aperture Value		f/8	f/5.6	f/5.6	-	f/2.8
Shutter Speed		-	-	-	1/250	1/120
Minimum Shutter Speed		1/250	1/125	1/250	-	-
Shutter Mode		mechanical	mechanical	Mechanical | electronic	Mechanical	Electronic
Iso Mode		AUTO	AUTO	AUTO	AUTO	MAX 1
Iso lower | Upper auto Limit		100 | 3200	200 | 1600	100 | 1600	200 | 1600	400 | 1600
Focus	First Shot	Auto focus	Auto focus	Auto focus	Auto focus continuous	-
	Entire Acquisition	Manual	Manual	Manual		-

¹ The camera automatically adjusts the ISO up to the maximum specified.

for our tests after considering the specific characteristics of each camera system and the predominant environmental conditions.

Lastly, there is the consideration of how best to set the camera’s focus. To ensure the stability of the interior camera parameters throughout the image acquisition, and, whenever possible, we selected the focusing distance as follows: first, the automatic focus option is used to focus the camera system at the proper acquisition distance; then, the focus is switched to manual to avoid changing the focus throughout the entire shooting session.

The three single camera systems (PL41, N750 and N300) were configured in single shot mode. To collect synchronized data, the PL51-PL52 stereo and 5-GoPro systems were used in time lapse photo and video mode, respectively.

The stereo system synchronization for the PL51-PL52 camera system was achieved by manually and simultaneously initializing the image acquisition for the two cameras.

For the 5-GoPro systems, the video mode was selected for the flexibility provided in recording the plots and because a waterproof multi-camera hardware-based synchronization approach would have required the modification of the factory pressure housing and the development of a special in-house system. Video synchronization was achieved via cross-correlation of external audio signals. The nadir looking camera (GoPro45) was selected as the master and the delays of the other four cameras were estimated. Frames were extracted from each video stream at a fixed time rate (1 fps) in the lossless PNG format. The PNG frames were then converted to JPG at the highest possible quality. Relevant exchangeable image file format (EXIF) tags were also embedded, allowing photogrammetric software applications to automatically recognize images coming from different cameras and estimate the initial values for camera calibration [30].

The analysis of the multi-camera systems did not show significant differences from the single camera systems and are consequently omitted in the following discussion. Interested readers should refer to [21].

3.2. White Balance, Color Correction and Image Quality

Color-corrected images are important for a proper interpretation of the recorded scene. Underwater, it is crucial to understand the state of health of the marine environment.

Analyzing the quality of the acquired images provides insights on the used imaging system, e.g., introduced optical distortions and aberrations, critical factors that may negatively influence the attainable accuracy.

Water acts as a selective filter. This means that not only a great amount of light entering water is absorbed, but that the different wavelengths composing the visible spectrum are absorbed differently, according to their spectral power distribution (SPD). Underwater, after only a few meters’ depth, and depending on water clarity (e.g., due to local characteristics such as dispersed particles, presence of soil, algae, plankton, etc.), the SPD of light reaching the subject of interest mostly lacks the longer wavelengths corresponding to red and orange colors. They are absorbed quickly, causing the greenish or bluish appearance of images acquired underwater, even at shallow depths.

Restoring the proper color balance from a picture that has been acquired underwater while not taking into account this phenomenon is equivalent to trying to color balance a picture taken above the water under a very different source of illumination than natural light (e.g., a picture taken under domestic, tungsten-filament lighting using a color camera setting for sunlight). In this situation, if the image has not been taken in raw format, a proper reproduction of colors is very difficult to achieve in post-processing, as the color content recorded by the RGB sensor lacks the signal in the corresponding wavelengths or might saturate in another channel. Attempts to recover the color corresponding to that specific missing wavelength would result only in the amplification of the sensor noise.

Many algorithmic solutions have been proposed [11,31], using extended physical models to recover the color information from bluish color-shifted images also considering distance-dependent effects.

We adopt a different approach, which relies on the acquisition of images in raw format. Raw files contain uncompressed and minimally processed data captured by the image sensor, making it possible to perform white balance adjustment before converting the images to the JPG format. When possible, we introduce standard color reference cards directly into the plots to verify that color reproduction is consistent over the entire acquired scene.

An example of the tests carried out during the 2019 Moorea campaign using color reference cards distributed along a straight path at about 1 m distance from each other, up to a maximum distance from the camera of about 5 m, is shown in Figure 9. Figure 9a shows the bluish color cast caused by the automatic in-camera white balancing, while Figure 9b displays the image after using a white balance performed on site at the operative depth of 12 m with a white card before starting the image acquisition. Despite an overall brightness reduction as function of the camera-to-color reference card distance and local color casts reflected off the reddish algae on the dead corals, the overall colors are reproduced consistently, even for distances from the camera of up to 3–4 m. Such large differences are not encountered during a nadir-like camera network where the camera-to-object distances remain almost the same, but can occur when acquiring oblique images. Oblique images also are more affected by color shifts in the observed scenery, although, as shown in Figure 9b, color variations are not critical in Moorea. We performed color processing and analyses of oblique images by automatically masking out the areas of the images that are more distant than 3 m using a depth map approach.

To improve the color appearance and contrast of the frames acquired with the GoPro camera system, a red filter was used during the image acquisition (Figure 10). The red filter modifies the SPD by lowering the amount of dominant blue light available under the water, resulting in a more balanced SPD that GoPro cameras can automatically white balance at the cost of a slightly higher noise due to the reduced overall light.

In Figure 11, we show some details of sample images (center and bottom right corner) acquired with the different camera systems used in this study. Generally, image quality degradation can be observed moving from the center to the corner, as expected due to the presence of water and underwater optical ports. The full-frame N750 shows pronounced spherical aberrations towards the corners due to the dome port. The N300 displays a halo close to the center of the image, likely due to local defects of the optical elements (lens, port or diopter, or some combination of these elements). The GoPro reveals a chromatic aberration, increasing towards the image borders due to the flat port. Image pre-processing steps can be applied to improve the GoPro image quality and reduce the chromatic aberration effects. However, the improvements achieved with these methods, e.g., collocation model or a separate calibration approach for three color channels, while able to be observed visually, are difficult to quantify computationally [32].

3.3. Photogrammetric Processing

The collected image datasets were processed following a free network self-calibrating bundle adjustment approach, using both Agisoft Metashape (V 1.6, [33]) and DBAT (V. 0.8.5., [34,35]). The two software tools produced results that were not significantly different from each other. Eight different cases for two working distances, 2 and 5 m, were considered, i.e., the five high-quality off-the-shelf cameras, nadir looking GoPro, stereo and 5-GoPro systems.

3.3.1. Residual Systematic Patterns

An interesting analysis concerns the average image residual patterns for the different camera systems (Figure 12).

Image observations’ residuals (or reprojection errors), r, are computed as:

$$r_{x_i}=x_{i_{ }}- {\overline{x}}_i$$

(1)

$$r_{y_i}=y_i- {\overline{y}}_i$$

(2)

$$r_i= \sqrt{r_{x_i}{}^2+r_{y_i}{}^2}$$

(3)

where ($x_{i_{ }}{y}_i$) represent the image observation coordinates in the image plane and (${\overline{x}}_i$, ${\overline{y}}_i$) are the re-projections of the 3D coordinates estimated within the bundle adjustment procedure (image coordinate residuals).

A similar systematic pattern is observed for N750 and PL41, with higher residuals arranged in a circular shape around the image center and towards the corners. The residual systematic effect for the N750 was already reported in [36] and is also confirmed by the visual analysis of the images (Figure 11). The behavior is assumed to be related to optical effects introduced by the dome port. These effects are not modeled by the standard functions of self-calibration and can therefore not be compensated.

Although PL51 and PL52 are nominally the same camera system, they show very different residuals maps, with higher values for the PL52. This performance is also consistently observed in the values reported in

Table 5. Independent check: comparison between the photogrammetric solutions and primary control network.

	N750	N300	PL41	PL51	PL52	GoPro45
Working distance: 2m
RMSr (pixel)	0.9	0.8	0.6	0.6	1.5	1.6
RMSEXY on PRPs (mm)	3.8	3.9	3.8	2.9	3.5	4.0
RMSEZ on PRPs (mm)	2.6	5.0	2.3	2.1	3.6	2.6
RMSEXYZ on PRPs (mm)	3.5	4.3	3.4	3.2	3.5	3.6
3D_RMSEXYZ on PRPs (mm)	6.0	7.4	5.8	5.5	6.1	6.3
MAX ERROR [on PRP] (mm)	9.4 [2]	10.3 [2]	9.3 [2]	8.4 [2]	7.7 [2]	9.6 [2]
σX | σY | σZ (mm)	0.9|0.9|1.3	0.6|0.6|1.1	1.5|1.5|1.8	0.8|0.8|1.1	1.6|1.6|3.7	2.2|2.1|3.8
Working distance: 5m
RMSr (pixel)	0.8	-	0.4	0.4	1.2	1.7
RMSEXY on PRPs (mm)	3.8	-	3.9	3.5	4.1	3.5
RMSEZ on PRPs (mm)	3.1	-	2.9	2.6	6.2	3.1
RMSEXYZ on PRPs (mm)	3.6	-	3.6	3.2	4.9	3.4
3D_RMSEXYZ on PRPs (mm)	6.2	-	6.2	5.6	8.4	5.8
MAX ERROR [on PRP] (mm)	10.1 [2]	-	9.1 [2]	7.4 [2]	11.5 [5]	8.6 [1]
σX | σY | σZ (mm)	0.8|0.8|1.8	-	0.5|0.5|1.2	0.6|0.5|1.2	1.7|1.7|3.7	3.2|3.4|5.8

,

Table 6. Comparison of photogrammetric bundle adjustment (BA) solutions for different camera systems at the two working distances: RMSE_XY|RMSE_Z|3D_RMSE_XYZ, of differences computed on the nine PRPs + SRPs 3D coordinates (values are in mm).

5 vs. 2m
N750	N300	PL41	PL51	PL52	GoPro45
0.4|1.0|1.2	-	0.8|2.5|2.7	0.6|1.7|1.9	2.1|4.3|5.2	1.2|3.3|3.8

and

Table 7. Comparison of photogrammetric bundle adjustment (BA) solutions between the different camera systems: RMSE_XY|RMSE_Z|3D_RMSE_XYZ, of differences computed on the nine PRPs + SRPs 3D coordinates (values are in mm).

	N750	N300	PL41	PL51	PL52	GoPro45
	Working distance: 2 m
N750	-
N300	1.1|3.4|3.7	-
PL41	0.4|0.6|0.8	1.0|3.8|4.1	-
PL51	1.0|1.4|2.0	0.8|2.9|3.1	0.8|1.8|2.2	-
PL52	1.6|1.4|2.7	1.5|3.4|4.0	1.6|1.4|2.6	1.0|2.5|2.9	-
GoPro45	0.9|1.3|1.8	0.7|2.5|2.7	0.8|1.9|2.1	0.8|1.0|1.5	1.3|2.2|3.0	-
	Working distance: 5 m
N750	-	-
N300	-	-
PL41	0.6|3.6|3.8	-	-
PL51	0.6|1.9|2.1	-	1.0|5.0|5.2	-
PL52	1.1|4.2|4.5	-	1.4|7.5|7.8	1.2|3.3|3.7	-
GoPro45	1.1|3.1|3.5	-	1.1|4.2|4.5	1.1|3.8|4.1	1.0|6.0|6.2	-

. The worst performances are to be ascribed to the use of the silent mode in the PL52 camera system which enables the electronic shutter instead of the mechanical shutter. This introduces effects that are not properly modelled in the classic photogrammetric camera model (including self-calibration).

The distinctive systematic effect visible for N300 confirms the visual analysis in Figure 11; there, a halo is visible to the right of the central part of the image where the target is located.

The image residuals are quite high in magnitude for the GoPro. This is not surprising due to poorer image quality caused by a combination of the cheaper sensor and lens and the presence of a flat port [36]. Comparing the reprojection errors, the GoPros produced values that were greater than the higher quality systems by a factor of 2. This agrees with the results in [8].

3.3.2. Object Space Analysis

The coordinates of the primary reference points PRPs (five) are used a-posteriori to define the datum and serve as an independent check to empirically compute the errors in object space (root mean square error, RMSE), as seen from Formulas (4) to (9):

$$RMS{E}_X = \sqrt{\frac{1}{n}·{\displaystyle \sum }_{i=1}^n{\left(X_{Phot{o}_i} - X_{PR{P}_i}\right)}^2}$$

(4)

$$RMS{E}_Y = \sqrt{\frac{1}{n}·{\displaystyle \sum }_{i=1}^n{\left(Y_{Phot{o}_i} - Y_{PR{P}_i}\right)}^2}$$

(5)

$$RMS{E}_Z = \sqrt{\frac{1}{n}·{\displaystyle \sum }_{i=1}^n{\left(Z_{Phot{o}_i} - Z_{PR{P}_i}\right)}^2}$$

(6)

$$RMS{E}_{XY} = \sqrt{\left(RMS{E}_X{}^2 + RMS{E}_Y{}^2\right)/2}$$

(7)

$$3D\_RMS{E}_{XYZ} = \sqrt{RMS{E}_X{}^2 + RMS{E}_Y{}^2 + RMS{E}_Z{}^2}$$

(8)

$$RMS{E}_{XYZ} = 3D\_RMS{E}_{XYZ}/\sqrt{3}$$

(9)

where the subscripts Photo and PRP indicate the photogrammetrically derived and primary control network point coordinates, respectively. X and Y define the horizontal plane, while Z is along the vertical direction.

Table 5 summarizes the results of the independent check, i.e., the comparison between the free network self-calibrating bundle adjustment for the different camera systems and the primary control network at the two working distances. The horizontal errors are larger than the vertical component at both working distances, except for the PL52 and N300. This is not in accordance with theory (see the standard deviations in Table 3) and can be attributed to the fact that there are still, even after self-calibration, small systematic errors (see Figure 12). The maximum error is consistently, except in two cases, on the same reference point. Standard deviations of the object space points (σX, σY, σZ) are also reported in Table 5. As expected, σZ is generally larger than σX and σY and the highest values are observed for the GoPro nadir camera. Interestingly, the values are roughly the same for the two working distances across all of the camera systems. This is against expectations and is still under investigation.

In Table 6, the intra-comparison between each camera system for the working distances of 2 and 5 m is reported, while in Table 7, the inter-comparison is summarized. In this case, the analysis is performed on the photogrammetrically derived coordinates of all the reference points, primary plus secondary (PRPs + SRPs, Figure 5). The RMSEs are then computed according to Equations (4)–(9), where the point coordinates from the same camera systems at the two working distances (Table 6) or two different camera systems (Table 7) are introduced.

As expected, greater differences are observed in the vertical direction and the differences are smaller at the shorter working distance. However, a high degree of consistency is observed among the photogrammetric systems, especially for the higher quality camera systems (N750, PL41 and PL51). Surprisingly, the GoPro system performs well in comparison with the other higher quality cameras. The different systems all perform within the accuracy required for quantifying the growth of several species of corals commonly found on coral reefs in the South Pacific, under favorable imaging configurations, i.e., highly redundant networks of images acquired with nadir and oblique optical axes, and on a relatively small area, which mitigate the effects of systematic errors caused by water refraction and flat ports, still visible in the residual patterns described in Section 3.3.1.

In summary, while these tests demonstrate the potential for very high photogrammetric accuracy, even underwater, it should be noted that our empirical error measurements are based on the use of only five reference points.

4. Camera Network Analysis

As highlighted in the previous section, under very special conditions, i.e., those with robust and reliable camera networks and when the surveyed area is not very large, the accuracy performances of different camera systems, both high and low quality, can be comparable. Here, we want to stress the importance of the camera network or imaging configuration, particularly in critical situations, such as, for example, the survey of a very elongated plot, which is very common when AUVs (autonomous underwater vehicles), ROVs (remotely operated vehicles) or underwater scooters are used.

The analysis is performed on the 50 × 10 m fore reef plot, whose geodetic graph and topography are shown in Figure 4. The site was surveyed photogrammetrically in three different dives with the same camera system, PL51, which was disassembled after each dive to download the acquired data and recharge the batteries. For this reason, even if all the images are processed together, three different sets of camera calibration parameters are considered. The final network comprises 2600 nadir images, arranged in cross-strips, and 700 oblique images.

Two self-calibrating free-network BA solutions are computed (Figure 13): pure nadir imaging configuration (Figure 13, upper image) and the full image block, i.e., nadir plus oblique images (Figure 13, lower image). The coordinates of 32 primary reference points (PRPs) are considered a-posteriori to provide the empirical independent check for the photogrammetric solutions.

From

Table 8. Self-calibrating free-network BA solutions for the 50 × 10 m fore reef plot.

	Nadir	Nadir + Oblique
# images	2600	3300
Average GSD (mm)	0.69	0.74
RMS reprojection error (pixel)	0.74	0.71
RMSEXY | RMSEZ | 3D_RMSEXYZ (mm)	16.5 | 107.3 | 109.8	13.4 | 12.8 | 22.9

, the RMSE of the complete network shows that the accuracy is worsening for the pure nadir camera network by about 400%. This finding is in accordance with empirical evidence underwater [7] as well as in other operative conditions (unmanned aerial vehicle—UAV, and terrestrial photogrammetry, [37,38] and results from simulations [39].

5. Level of Detection or Significance of Changes in Coral Reef Monitoring

In classic geodetic monitoring, a displacement field between different measurement epochs is computed on well-recognizable, often signalized homologous points. Another approach very commonly employed in geosciences to monitor the evolution of natural surfaces (e.g., landslides, rockfalls, coastal cliff, riverbanks, etc.) entails the comparison of two 3D models in the form of gridded digital elevation models (DEMs), point clouds or meshes. In this case, three methods are adopted to compute the distance between the two models: DEM of difference (DoD), cloud-to-cloud and cloud-to-mesh distance, and each of these methods present specific advantages and drawbacks, as discussed in [40]. Regardless of the method used, propagating measurement and modeling uncertainties throughout the process up to the final comparison and analyses is crucial to determine whether the computed differences highlight significant temporal changes or are mainly due to measurement and modeling errors. The concept of level of detection at a required confidence interval x% (LoD_x%) [22,40] is adopted in monitoring activities: measured changes smaller than the specified LoD_x% should be considered as effects of random errors and disregarded. The estimated LoD_x% depends on the uncertainties of the compared models, relative registration error (or the residuals of the Helmert transformation between the two epochs) and the required confidence level [40].

Comparison between point clouds have been long preferred over comparison between meshes, on the grounds that the meshing operation introduces additional uncertainties and approximations in the modeling process, also often accompanied by a loss of high-frequency details (smoothness). Here, we take advantage of multi-view stereo (MVS) algorithms which incorporate the meshing step in the photogrammetric workflow and produce a mesh as final output [41]. This has two benefits. First, the photo-consistency check incorporated in the meshing process allows recovery of more high-frequency details visible in the images [42]. Second, the error budget propagation is theoretically achievable throughout the full photogrammetric workflow, from the BA downstream to the mesh generation. Our implemented approach is depicted in Figure 14.

The procedure starts with the photogrammetric processing of two epochs to evaluate the coral growth over the investigated time span. This step follows the procedure described in Section 3, where each survey is processed through a self-calibrating BA. The coordinates of the primary control points, which are assumed to be stable over time (Section 2), are introduced a-posteriori to check the accuracy of the two photogrammetric solutions and highlight any residual systemic errors. If the independent check for the two epochs does not show any significant residuals or differences, the photogrammetrically derived coordinates of the primary and secondary points from the first epoch (the older one) are used to define the reference datum and perform the relative registration of the second epoch to the first. The final transformation error, as well as the tie points’ standard deviations, are retained for the LoD_x% estimation, as better detailed hereinafter. Tie points’ standard deviations can be estimated from the covariance matrix in a rigorous approach, such as the one we follow, or via Monte Carlo simulation, as shown in [22]. The mesh models for the two epochs are generated at a similar resolution. While the dense matching uncertainty should be theoretically available for each mesh vertex [23,43], it is computationally demanding and would require massive amounts of memory usage, particularly in the case of the high-resolution models required for coral reef monitoring. A strict solution is offered by the multi-image geometrically constrained least squares matching technique, where not only the matching parameters of all images involved are computed but also simultaneously the object space coordinates, including their standard deviations [44]. Unfortunately, this solution is not available to us in the form of implemented software. As a less intensive alternative, we consider the total number of stereo pairs per vertex as the confidence parameter, with zero corresponding to interpolated vertices, one to vertices reconstructed from a single stereo pair, two reconstructed from two stereo pairs and so on. The tie points’ standard deviations are also transferred to the mesh vertices of the same epoch through nearest neighbor linear interpolation; analogously, the registration error is associated with the vertices of the mesh from the second epoch. The distance between the two mesh models is finally computed only for those vertices with a number of stereo-pairs greater than one and compared against the estimated LoD_x%. Considering a confidence interval of 95% and based on established error analysis [22,40,45], Equation 10 is adopted:

$$Lo{D}_{95\%} = \pm t·\left(\sqrt{\frac{{\sigma }_1{}^2}{n_1} + \frac{{\sigma }_2{}^2}{n_2}} + E_{reg}\right)$$

(10)

where

• $t$ is computed for each vertex according to the t-statistics, which replaces the standard normal distribution when $n_1$ and $n_2<30$, with a confidence level of 95% and a degree of freedom (DoF) computed as (Borradile, 2003; Langue et al., 2013):

  $$DoF = {\left( \frac{{\sigma }_1{}^2}{n_1} + \frac{{\sigma }_2{}^2}{n_2}\right)}^2/\left(\frac{{\sigma }_1^4}{n_1{}^2}/\left(n_1 - 1\right) + \frac{{\sigma }_2^4}{n_2{}^2}/\left(n_2 - 1\right)\right)$$

  (11)
• ${\sigma }_1$ and ${\sigma }_2$ are the tie points’ standard deviations from the covariance matrix and are transferred to the mesh vertices for epoch 1 and epoch 2,
• $n_1$ and $n_2$ are the number of stereo pairs for each vertex in the mesh for epoch 1 and epoch 2, respectively. The distances are disregarded if $n_1$ or $n_2$ < 2,
• $E_{reg}$ is the registration error from epoch 2 to epoch 1.

The adopted approach is showcased for the 5 × 5 m fore reef plot presented in Section 3. The results of the photogrammetric processing for the two epochs is summarized in

Table 9. Statistics from the photogrammetric processing for epoch 1 (2018) and epoch 2 (2019) on the same plot. The same camera system was used (PL51).

	Epoch 1 (2018)	Epoch 2 (2019)
Working distance (m)	2.0	2.0
GSD (mm)	0.8	0.9
Number of Images	523	318
RMS Reprojection Error (pixel)	0.6	0.6
RMSEXY | RMSEZ | RMSEXYZ on PRP (mm) from Geodetic Network	2.9 | 2.3 | 3.2	2.3|1.8|3.8
RMSEXY | RMSEZ | RMSEXYZ on PRP + SRPs (mm) from Epoch 1	-	1.3|0.9|1.6
σX | σY | σZ | σXYZ (mm)	0.8|0.8|1.1|1.6	0.8|0.7|1.5|1.9
Mesh Resolution (mm)	0.1	0.2

. We used the mesh model generation from depth maps implemented in MetaShape v.1.6 [33], a method based on the work by [46].

The sequential steps of the outlined procedure (Figure 14) for the detection of significant changes between the two epochs are illustrated in Figure 15. In particular, the last row reports the distance computation without and with test of significance. Oriented distances between the two 3D models were calculated using the Cloud to Mesh (C2M) algorithm implemented in the CloudCompare software suite [47]. According to the common color scale, the warm colors represent positive distances and suggest a growth of investigated colonies, and cold colors represent negative distances. Values exceeding 50 mm and –30 mm are shown in magenta and full blue, respectively. Grey is used to represent computed distances that do not exceed the computed LoD_95%; thus, the areas in grey display differences that are not significant. Significant positive distances are mainly associated with the outer surface of coral colonies where the coral growth is greatest. Negative distances are sparser than positive ones and appear to be restricted to area of little coral cover and valleys between coral colonies. Figure 16 visually shows the temporal changes between the two epochs on a small area, changes that are quantified as significant differences using a color-coded representation.

6. Conclusions and Future Developments

We presented a thorough study to demonstrate the ability of underwater photogrammetry for coral reef monitoring applications. The study was conceived within a wider project aiming at modeling the complex ecosystem of a remote tropical island. The innovative aspects of our research lie in the approach implemented to assure that the strict requirements demanded by the application are fully met.

The first crucial step is the establishment of reference points, stable over time in the reef structure, measured using underwater geodetic methods. These reference points are not only necessary to establish a common reference system for successive surveys, but they are also needed as an independent check to verify the accuracy of the photogrammetric models. The method implemented and equipment specifically designed have been described in detail. This step still represents a big challenge, especially if the monitoring activity is extended over a large area. To overcome this demanding activity, we are currently investigating an alternative solution based on a prototype device developed by the authors (Figure 17) which will allow the recording of inertial as well as pressure measurements that are then included as observations into the bundle adjustment to reduce 3D model deformations over long transects. Preliminary tests anticipate a reduction of the error in the Z coordinate of more than 50% on a 50 m long transect with only nadir looking images.

The second aspect concerns the investigation of different camera systems for underwater photogrammetry and coral monitoring. We have presented a systematic study intended to identify best practices for image acquisition strategies and protocols. We have shown that, when the surveying area is small, lower-cost cameras, such as GoPro, can produce results similar to those produced by higher-cost custom camera systems, provided that the proper camera settings are selected. When surveying large areas, image quality and, even more, camera network become crucial to reduce deformations in the photogrammetric model, due to uncompensated systematic errors. Some of the authors are currently investigating computational methods for mitigating the unmodeled systematic errors in a self-calibrating bundle adjustment approach. One method [48] uses radial weighting of image observations which proportionally penalizes those observations with larger radial distances and is designed mainly to work with flat ports. The second approach [39] estimates lookup table corrections from systematic residual patterns iteratively and is targeted at camera systems using dome ports. After the first self-calibrating bundle adjustment solution, the image plane is subdivided into a grid, where for each cell, the median image residual is computed for x and y image coordinates. The correction, equal to the computed median residual error, is applied to the x and y image observations and a new self-calibrating bundle adjustment step is run. The procedure is iteratively repeated until the convergence criterion is reached (difference in the solution vector). This method is well-known in photogrammetry as the Masson d’Autume technique for systematic error removal [49]. While the radial re-weighting of image observations for flat ports considerably improves the accuracy in object space, especially in the presence of weak camera network geometry, the lookup table correction method for dome ports does not seem to further improve the accuracy in object space due to the low absolute values of the corrections.

We also stressed how an imaging geometry comprising both nadir and oblique views can reduce model deformations up to four times compared to a camera network with only nadir images.

The investigation concluded with a discussion of a framework we have developed to characterize the quality of the photogrammetrically derived reef models and assess the confidence or significance level of the changes detected through time. Measurement and modeling uncertainties are propagated downstream to the photogrammetric workflow and exploited to identify changes over time which can be considered statistically significant. The images in the last row in Figure 15 and Figure 16 stress the importance of conducting a test of significance which is critical for providing marine ecologists with more reliable results for analysis of temporal changes, thus avoiding misinterpretation that might occur when ignoring the uncertainty propagation through such a complex process.

We are currently working on incorporating automatic semantic segmentation approaches [50] into our photogrammetric pipeline to identify coral and non-coral components within the reef and restrict the change detection analysis only to the elements of interest.