Semantic Segmentation of Coral Reefs Using Convolutional Neural Networks: A Case Study in Kiritimati, Kiribati

Highlights

What are the main findings?

• We developed and trained ResNet101-SCSE-U-Net for semantic segmentation of 16 benthic reef classes, achieving 84% overall accuracy, and modifications to the model were able to handle the highly unbalanced training data.
• The trained model was successfully applied to 27 unlabeled plots from 2023, demonstrating strong generalizability and providing a tool for automated monitoring of new reef surveys.

What are the implications of the main findings?

• We provide a reproducible, step-by-step pipeline for fully supervised deep learning-based annotation of coral reef imagery, reducing the reliance on manual labeling by experts.
• The pipeline and trained models can be transferred across reef environments, capturing ecological variation and enabling fully automated, high-resolution classification of benthic communities.

Abstract

Habitat complexity plays a critical role in coral reef ecosystems by enhancing habitat availability, increasing ecological resilience, and offering coastal protection. Structure-from-motion (SfM) photogrammetry has become a standard approach for quantifying habitat complexity in reef monitoring programs. However, a major bottleneck remains in the two-dimensional (2D) classification of benthic cover in three-dimensional (3D) models, where experts are required to manually annotate individual colonies and identify coral species or taxonomic groups. With recent advances in deep learning and computer vision, automated classification of benthic habitats is possible. While some semi-automated tools exist, they are often limited in scope or do not provide semantic segmentation. In this investigation, we trained a convolutional neural network with the ResNet101 architecture on three years (2015, 2017, and 2019) of human-annotated 2D orthomosaics from Kiritimati, Kiribati. Our model accuracy ranged from 71% to 95%, with an overall accuracy of 84% and a mean intersection of union of 0.82, despite highly imbalanced training data, and it demonstrated successful generalizability when applied to new, untrained 2023 plots. Successful automation depends on training data that captures local ecological variation. As coral monitoring efforts move toward standardized workflows, locally developed models will be key to achieving fully automated, high-resolution classification of benthic communities across diverse reef environments.

1. Introduction

Habitat complexity plays a vital role in ecosystem function and diversity, offering numerous benefits, including enhanced habitat provisioning, increased resilience, and effective coastal protection [1,2,3,4]. Complex and dynamic habitats promote resilience by providing greater niche availability and refugia across multiple spatial scales [5,6,7,8,9]. Increased habitat complexity is positively associated with species richness, abundance, and diversity across most benthic ecosystems [1]. For example, fish abundance in coral reefs has been closely linked to structural complexity [10,11,12,13,14].

To quantify reef complexity, many monitoring programs now incorporate outputs such as rugosity, curvature, and vertical relief derived from photogrammetric surveys. However, while the physical complexity of reefs can be semi-automatically processed using tools like digital surface models (DSM) and orthomosaics, the classification of benthic communities still depends heavily on manual expert annotation. This reliance on expert input is both time consuming and resource intensive, contributing to persistent undersampling of benthic communities [15,16,17,18]. Recent advances in computer vision and machine learning, specifically semantic segmentation using neural networks, offer a promising path forward by automating benthic classification and helping bridge the gap between data collection and ecological interpretation [19,20].

Semantic segmentation, which assigns a class to every pixel in an image, is a widely used method in image analysis [21]. While other techniques are also used, such as random point identification, object detection, and instance segmentation, semantic segmentation may be a more suitable option for marine applications owing to the complexity of underwater environments and their organisms. In this context, automation refers to the use of algorithms to perform tasks, such as identifying and classifying benthic features, that would otherwise require manual annotation. It has been applied in various contexts, including invertebrate detection in Antarctica, mapping sponge behaviors, and assisting with megafauna species identification and population counts [22,23,24,25,26].

One of the earliest applications of neural networks for benthic classification was CoralNet, a tool that remains widely used [27]. CoralNet facilitates the semi-automated annotation of benthic imagery, using human-identified random point classifications to estimate the percentage composition of each class [27,28]. While CoralNet has a strong training dataset for point-based analysis, it lacks image segmentation capabilities and cannot integrate higher dimensional data such as DSMs or orthomosaics. TagLab was the first semi-automated program designed for orthomosaic imagery and 3D models, combining interactive, pixel-wise segmentation with an automated learning pipeline to develop robust semantic segmentation models for benthic mapping [29]. More recently, there has been an increase in the diversity of neural networks for benthic classification. Hopkinsons et al. [30] integrated 3D models with imagery acquired via SfM photogrammetry, leveraging the combined dimensionality to improve classification outcomes. Other recent tools, like DeepReefMap, focus on speed and versatility, enabling the automated classification of video data collected by drones and autonomous underwater vehicles (AUVs). While these systems provide practical solutions for large-scale surveys, they typically achieve moderate classification accuracy, ranging from 60% to 85%, and coarse benthic classes, such as coral, algae, substrate, and sand [31]. By contrast, an imaging spectroscopy (hyperspectral) approach that combines spectroscopy with 3D models offers finer taxonomic resolution and reported accuracies of 90–96%. However, this method requires highly specialized equipment called the HyperDiver [18], greatly limiting its availability.

Even more recently, the exponential rate of advancements in computer vision and machine learning has led to an increase in automated approaches for benthic habitat classification, particularly through the use of convolutional neural networks (CNNs) and semantic segmentation techniques. Semantic segmentation has emerged as the dominant method due to its ability to capture fine-scale spatial heterogeneity in complex environments like coral reefs. Several recent studies have demonstrated the applicability of CNN-based approaches in this context, including those that leverage point annotations [29,30], pretrained general-purpose models [32], or voxel-based architectures [33]. These studies often rely on relatively small training datasets (typically 3–8 plots), point-based annotations, minimal to no applications on new datasets, or limited temporal coverage, which can constrain the ecological generalizability of models. For example, Remmers et al. [32] introduced the RapidBenthos pipeline, which combines a pre-trained segmentation model (SAM) with multi-viewpoint annotations to classify reef imagery with minimal manual effort. While highly efficient and capable of classifying over 40 taxa within individual plots, the method relies on sparse point labels from ReefCloud, limiting its ability to generalize to fully labeled datasets or across time. It also cannot measure growth or mortality, nor fully capitalize on the fine spatial resolution of data. Therefore, there is a trade-off between taxonomic resolution and ecological metrics, where point-based classification offers higher taxonomic specificity while pixel-level segmentation enables the measurement of ecological processes such as growth and mortality and temporal changes. One group presented a middle ground approach by combining sparse point-based annotations with superpixel-based delineations, enabling improved semantic segmentation performance while reducing labeling effort [34,35].

Despite the recent increase in the number of studies attempting to automate the classification process, many coral monitoring programs still heavily rely upon manual digitization for benthic classification [28]. This process is labor intensive, time consuming, and costly, yet it has generated an extensive repository of annotated vector layers with high-resolution 3D models and 2D orthomosaics. These datasets remain underutilized for the development of automated classification systems, representing a valuable resource for training next-generation neural network models.

Here, we developed a CNN containing the U-Net architecture [36] with a ResNet101 encoder and a Spatial and Channel Squeeze & Excitation (SCSE) block, specifically designed to automate the digitization of coral reef communities from 2D Red, Green, Blue (RGB) orthomosaic photos. The model was trained using complete pixel-level annotations across 77 plots surveyed in 2015, 2017, and 2019. We had between 16 and 27 benthic classes and one background, including living coral morphologies, macroalgae, and non-living substrate types. We then ran the fully trained model on 28 unlabeled sites from 2023. We included significantly more classes than most comparable pixel-based annotation studies and had one of the most temporally resolved datasets for automated benthic classification [31,34,35]. Our objective was not only to improve classification accuracy but to provide open-source, user-friendly code that is easy to implement and adaptable to local reef contexts. Given the complexity and variability of coral reef ecosystems, scalable automation depends on diverse and well-labeled training data. As the field continues toward standardization, developing locally trained models remains essential for building robust and transferable pipelines for benthic community classification.

2. Methods

2.1. Data Acquisition: Study Site, Plots, and Mega Photo Quadrats

To quantify the change in reef complexity, annual expeditions to Kiritimati (Christmas Island), Republic of Kiribati, were conducted before the onset of (May 2015) and following the mass coral bleaching event caused by the 2015–2016 El Niño-induced marine heat wave, which resulted in 89% mortality of all living hard coral around the island [37]. Thirty permanent 4 × 4 m plots (labeled as mega photo quadrats (MPQs)) were marked with rebar for relocation and resurveyed annually. These MPQs were established using structure-from-motion (SfM) photogrammetry techniques at 10 shallow forereef sites (3 MPQs per site) on Kiritimati (Christmas Island; Republic of Kiribati) and were revisited in subsequent years to monitor temporal changes within the same spatial footprint (Figure 1). The 10 sites were categorized into three levels of human disturbance gradients: low, medium, and high (as detailed in [17,37]). At each site, the three permanent MPQs were approximately 10 m apart along a 10 m to 12 m isobath. The MPQs were surveyed annually from 2015 to 2023, except for the period from 2020 to 2022 due to COVID-19 travel restrictions. Data from 2015, 2017, 2019, and 2023 were used for this investigation. Some sites could not be surveyed in certain years due to accessibility issues; for example, sites 15 and 19 were not surveyed in 2019 because of south swells generated by three consecutive hurricanes over the Hawaiian Islands. A full description of the sites is available in [38].

2.2. Data Preprocessing: 3D Models, Benthic Community Digitization, and Data Conversion

Images were collected using SfM photogrammetry techniques described in [15], and a detailed description of these field methods from Kiritimati is provided in [17]. Between 400 and 800 images were collected for each MPQ (plot). Ground control points and scale bars were placed at the corners of the MPQs (plots) for scaling and orthorectification. A diver collected overlapping (70–80%) planar images while swimming roughly 1 m to 2 m above the benthos in a boustrophedonic or lawn-mower pattern. Images were captured using a Nikon D750 DSLR with a 24 mm Sigma lens, housed in an Ikelite underwater system with an 8-inch hemispheric dome port. The images were automatically processed using Agisoft PhotoScan software 2.2.1 [39] to produce 3D models with a resolution of <0.01 cm and then scaled up to a resolution of 1 cm. The outputs included a DSM, an orthomosaic photo, a 3D mesh, and a 3D point cloud for each plot. We manually annotated the benthic surface composition on each 2D orthomosaic using the digitization tool in ArcMap 10.8.2 [40] to create a benthic cover map for each plot in 2015, 2017, and 2019. The benthic cover map categorized the substrate into 27 morphological classes and one background class in the orthomosaics with NaN values (labeled class 127; Table S1). A trained technician used the digitization tool in ArcMap to manually delineate polygons around living coral colonies, dead coral, and inorganic substrate, assigning each polygon to 1 of the 27 predetermined morphology classes (Table S1). Digitizing each plot required 6 to 10 h of labor, with more time needed for plots with high live coral cover. The final output for each plot was a multi-polygon shapefile that delineated benthic cover, including individual coral colonies.

In total, we processed 77 plots (one orthomosaic per plot) from 2015, 2017, and 2019 and 27 from 2023 (Table S2). Using ArcMap, we clipped and trimmed the orthomosaics and benthic cover polygons to ensure consistent extents for each plot. In ArcMap, the classified polygons were then rasterized and converted into a mask (classes were assigned a number instead of a category; Table S1) to be used as labels for the training data. The mask rasters were generated at the same resolution and with the same pixel dimensions as the orthomosaics, ensuring that each pixel was assigned either a benthic class or the background class. Since the orthomosaics were very large (>10,000 by 10,000 pixels), we divided each image and mask into tiles of 512 by 512 pixels. Tiles at the right and bottom border were padded and labeled as background class 127. The masks were converted into a one-hot encoded vector, which was essential for transforming categorical data into a format suitable for multi-class semantic segmentation. Examples of tiles are shown in Figure S1, and a simplified example of this process is illustrated in Figure 2. For the CNN training, only 2D orthomosaics were used; we did not include the 3D outputs from the SfM pipeline to increase the potential for transferability of the model.

2.3. Data Processing: Training Pipeline, U-Net Modifications, and Building the Model

The CNN model was trained on 125,901 tiles from the subsetted orthomosaics (plots) and masks from 2015, 2017 and 2019, and it was subsequently applied to predict masks that aligned with the orthomosaics (plots) from 2023 (Table S2). While the architecture of the CNN model followed a standard U-Net framework [26,36], we made several modifications to improve training stability, efficiency, and performance. To enhance generalizability and address uneven class distributions, we incorporated data augmentation, class weighting, and tile boosting of rare classes. We reduced overfitting through regularization techniques. These adjustments stabilized training, decreased processing time, and improved the ability of the model to handle the uneven class distribution.

2.3.1. Training, Validation, and Testing Pipeline

A training, validation, and testing pipeline was developed to ensure efficient data loading and transformation, thereby improving model generalizability and performance. Training, validation, and testing were randomly split across all of the compiled tiles: 70%, 15% and, 15%, respectively. We pooled all 125,901 tiles, partitioned them into training, validation, and test sets using spatially disjoint splits along the x-axis and y-axis to minimize spatial autocorrelation and ensure non-overlapping regions across splits [41]. The training/validation/testing pipeline included data preparation and prefetching (loading the next set of data while the current batch is processing) using ‘autotune’ from TensorFlow, one-hot encoding of NumPy mask files, storage as TensorFlow record files [42,43], and concurrent parallel loading of image tiles with optimized batching [42,44]. To further improve generalizability, data augmentation and shuffling were integrated. Data augmentation applied random transformations such as rotation and flipping. These techniques increased dataset diversity, enabling the model to better adapt to unseen data. By combining random augmentations with shuffling, the pipeline also mitigated overfitting and improved performance, particularly for underrepresented classes. The validation and testing pipelines followed the same structural framework as the training pipeline but excluded augmentation and shuffling to conserve computational resources. The testing portion of the pipeline was only used for final model evaluation and predictive performance. All datasets were cached to local scratch storage to further reduce latency and ensure reproducibility across training runs.

2.3.2. CNN Model Architecture Implementation

We implemented the U-Net architecture using TensorFlow and Keras API with a ResNet101 encoder and Spatial and Channel Squeeze and Excitation (SCSE) attention blocks decoder [36,42,45]. The ResNet101 backbone, pretrained on ImageNet, was used as the encoder (contracting path), where residual blocks enabled deep feature extraction and mitigated vanishing gradients via identity skip connections [45]. The decoder (upsampling path) comprised transposed convolutional blocks concatenated with corresponding encoder features. Each decoder stage included a convolutional layer, max pooling, batch normalization, ReLu activation, and an SCSE block to recalibrate features spatially and channel-wise. SCSE blocks combine channel-wise attention via global average pooling (cSE) with spatial attention (sSE), enhancing the focus of the model focus on semantically important regions [45,46,47]. SpatialDropout2D was applied after each decoder block to reduce overfitting. A final 1 × 1 convolution with softmax activation produced class probabilities and predictions for multi-class semantic segmentation [48]. We used softmax activation because we had a multi-class classification in our semantic segmentation [48]. A full list of the hyperparameters can be found in

Table 1. Hyperparameters for model architecture, training settings, and data handling for the ResNet101-based segmentation model.

Category	Hyperparameter	Setting
Model Architecture	Encoder	ResNet101 (pretrained on ImageNet)
	Decoder	Transposed convolution layers
	Activation function	ReLu
	Dropout type	SpatialDropout2D
	Dropout rate	0.5
	Skip connections	Enabled
Training Configuration	Loss function	Combined Focal + Dice Loss
	Optimizer	AdamW (weight decay 1 × 10−5)
	Learning rate	1 × 10−7
	Batch size	16
	Epochs	50
	Early stopping patience	30
	LR scheduler	ReduceLROnPlateau
Data Handling	Tile size	512 by 512 pixels
	Stride (step size)	256 by 256 pixels
	Data augmentation	Crop, flip, rotate, brightness adjustment
	Shuffle buffer size	1000
	Prefetch buffer size	AUTOTUNE
Regularization	Weight decay	1 × 10−5
	Batch normalization	Enabled
Miscellaneous	Random seed	42
	Class weights	Computed from training data

, with descriptions provided in Section 2.5.

To address class imbalance in the training data, we calculated pixel-level class frequencies across all training masks prior to tiling. Rare classes were identified as those with a frequency below half the median class frequency. During tiling, we duplicated tiles containing at least one rare class by a factor inversely proportional to the frequency of that class. Specifically, the duplication factor was calculated as the ratio of the median class frequency to the frequency of the rarest class present in the tile, capped between 1 and 10 to prevent overrepresentation. This targeted oversampling strategy ensured that rare but important benthic classes contributed more meaningfully during model training. To further mitigate the class imbalance, we used a combined loss function of focal and Dice losses with integrated class weights and label smoothing [49,50]. Class weights were calculated using the inverse log-frequency of class occurrences across all training data and normalized to 1.0. Weights were used to scale the focal loss to reduce bias toward the dominant classes. Dice loss was computed per image and weighted more heavily to ensure shape and boundary alignment were maintained. A label smoothing factor of 0.1 was applied to soften hard class assignments and improve generalizability [51].

All masks were one-hot encoded (Figure 2). We used an AdamW optimizer (weight decay of 1 × 10⁻⁵) with gradient clipping to stabilize training and prevent exploding gradients (hyperparameters can be found in Table 1; [52,53]). The model was compiled using a combined focal and Dice loss, with accuracy and the Dice coefficient evaluated as performance metrics. The model fitting process involved defining the training and validation datasets, specifying the number of epochs, setting the steps per epoch and validation frequency, applying class weights, and integrating monitoring callbacks (Table 1). This was the most time-consuming and computationally expensive phase, as it determined the overall model performance. The model was trained for the full 50 epochs. Each mask output was reassembled into a raster.

We conducted multiple rounds of convolutional neural network (CNN) model development using high-performance computing clusters and cloud services offered through the Digital Research Alliance of Canada. On these high-performance cluster servers, we used 250 GB of RAM and 1 NVIDIA A100S × M4 GPU with 40 GB of memory.

2.4. Data Post-Processing: Validation, Overfitting, and Model Comparison

Model performance was assessed on the testing data across all 27 benthic classes and one background class using accuracy metrics, including overall accuracy, the Dice coefficient, and loss. The model was then applied to predict new masks for data from 2015 to 2019. Only 16 classes were meaningfully predicted, and these were used to evaluate pixel- and class-level performance, including precision, recall, F1 score, and mean intersection over union (mIoU), to assess mask accuracy. The performance of the CNN was evaluated using a combination of metrics to ensure its accuracy, robustness, and generalizability. Learning curves for training and validation loss, as well as accuracy, were analyzed over epochs to assess model convergence, underfitting, and overfitting. To evaluate classification performance, we used a confusion matrix to compare the ground truth classes with the predicted classes. Only classes that were predicted correctly at least once were included in the matrix, resulting in 16 out of the 27 original classes. The confusion matrix was row-normalized to show, for each ground truth class, the percentage of pixels predicted as each class. We calculated per-class and per-site performance metrics, including intersection over union (IoU), precision, F1 score, and recall.

To evaluate model performance, we benchmarked SCSE U-Net with the ResNet101 backbone architecture against three commonly used CNN baselines: a standard U-Net, a fully convolutional neural network (FCNN), and a DenseNet-based model [36,54,55]. Performance was assessed using four metrics: test accuracy, the Dice coefficient, loss, and mean intersection over union (mIoU).

2.5. Applications to New Data

The trained model was applied to the unlabeled 2023 orthomosaics to evaluate its performance on imagery not included in the training or validation sets. Predicted masks were generated for each plot, and to provide a basis for assessment, we also prepared a small number of independent example masks that were manually annotated but not used in model development. These example masks allowed for direct visual comparison with the model outputs, confirming that the predicted classes aligned with the expected cover types, such as living coral types, rubble fields, and sand areas. This procedure provided additional validation of model generalization and demonstrated its utility for generating classifications in datasets that lack complete manual annotation.

3. Results

3.1. Model Performance

We provide a workflow that outlines the main processing steps in Figure 2. We introduced key modifications to the standard U-Net architecture and other CNN components to ensure that the model was well suited for our large, multi-class, multi-year, and spatially variable dataset. Some of these changes included ensuring that segmentation masks were properly one-hot encoded, implementing effective data augmentation strategies to increase dataset variability, calculating class weights, and selecting an appropriate optimizer and loss function (Table 1). These modifications significantly improved the ability of the model to learn global patterns, generalize new data, and converge efficiently during training.

To evaluate model training and validation while running the model, we tracked accuracy, loss, and Dice coefficient curves and the learning rate, all of which showed a good fit. The training and validation values remained closely aligned throughout all 50 epochs, and the learning rate decreased steadily during training (Figure 3). The steady increase and agreement in training and validation accuracy scores and Dice coefficients, along with the steady decrease in losses, indicated that the model effectively captured key features of the input data, supporting its ability to generalize and make accurate predictions (Figure 3;

Table 2. Model performance metrics (loss, Dice coefficient, accuracy, mean IoU) across training, validation, and testing datasets.

	Loss	Dice Coefficient	Accuracy
Testing	2.34	0.64	0.64
Validation	1.94	0.78	0.78
Training	1.87	0.81	0.84

). We added a mixed-precision in-memory training function (faster training by using mixed datatypes), which almost doubled the training speed but introduced substantial instability and exploded gradients.

SCSE U-Net with the ResNet101 backbone showed strong training, validation, and testing performance, with Dice coefficients of 0.81, 0.78, and 0.64, and corresponding losses of 1.87, 1.94, and 2.34, respectively (Table 2). These metrics indicated good pixel-level generalizability, even on held-out test data. Class-level metrics, namely mIoU, precision, F1 score, and recall, were stronger than the pixel-level metrics at both the class and site levels (Tables S3 and S4). The overall mIoU was 0.82, reflecting strong model performance, particularly given the heavily imbalanced dataset (Table S3). Most class-level IoU scores were high, ranging from 0.71 to 0.92, with the exceptions of Live Massive Grooved (0.52), Soft Coral Nodular (0.03), and Dead Foliose (0.46), which were impacted by mislabeling in one or two plots (Table S3). At the site level, both mIoU and accuracy remained strong, with mIoU scores ranging from 0.70 to 0.80 and accuracy scores ranging from 0.83 to 0.95 (Table S4). One exception was Site 32, with an mIoU of 0.52 and an accuracy of 0.85, which was mostly Rubble and Dead Massive.

3.2. Model Outputs and Application to New Data

In general, plots with more living coral produced more accurate masks, potentially because the Dead Massive and Rubble classes exhibited greater internal variability, often including cover by crustose coralline algae, sand, or microalgae. Despite these challenges, the predicted masks remained robust, capturing key spatial patterns across most sites. For example, in Site 5, Plot 1 (Figure 4), where data were taken four years apart, the model captured ecological change following the 2015–2016 El Niño event. Coral cover and benthic diversity declined sharply, which showed an increase in the dead massive coral class in 2019. One error occurred in Site 5 MPQ1, where we observed a large Non-Coral Submassive cover on the far left of the true mask, but it was predicted as Non-Coral Submassive (Figure 4a,b).

The confusion matrix showed classification accuracy and misclassification patterns based on the 2015, 2017, and 2019 datasets (Figure 5 and Figure S3). Only 16 of the 27 benthic cover classes were consistently predicted (background class 127 was excluded), which matched the class distribution of the true masks. The best-predicted classes were primarily living coral types, except for Live Massive Grooved, which was often confused with Live SubMassive, and Foliose for Live Branching, because of similarities in morphology and color (Figure 5). Small coral colonies were also a challenge, likely due to their limited spatial extent and underrepresentation in the training dataset. In the non-living classes, Dead Massive, Rubble, and Dead Branching all had the highest accuracy (Figure 5). However, distinguishing between hard inorganic and dead substrates was more difficult than discerning living coral classes. These findings were consistent with the per-class performance metrics IoU, precision, and recall (Table S3).

We benchmarked the performance of three standard CNN architectures, U-Net, FCNN, and DenseNet, against our proposed ResNet101-SCSE-Unet model for benthic segmentation (

Table 3. Test accuracy, Dice coefficient, loss, and mean intersection over union (mIoU) are reported for each convolutional neural network (CNN) architecture evaluated on the benthic segmentation task.

CNN	Test Accuracy	Test Dice	Test Lost	mIoU
U-Net only	0.43	0.34	2.04	0.48
FCNN	0.39	0.27	2.06	0.48
DenseNet	0.64	0.66	1.99	0.48
SCSE-Unet-ResNet101	0.64	0.66	2.13	0.83

). ResNet101-SCSE-Unet achieved the highest mIoU score of 0.83, significantly outperforming all other models, which each reached an mIoU of only 0.48. While DenseNet matched our model in overall test accuracy (0.64) and Dice coefficient (0.66), it fell short in class-level agreement, as reflected in its lower mIoU score. The U-Net and FCNN models demonstrated noticeably lower test accuracy scores (0.43 and 0.39, respectively) and Dice coefficients (0.34 and 0.27), with higher loss values than DenseNet and our model. These results demonstrated that ResNet101-SCSE-Unet offers substantially improved class-wise prediction accuracy and segmentation consistency compared to commonly used CNN baselines.

The predicted and observed class proportions were highly consistent across years, with the vast majority of differences falling within ±0.01, underscoring the ability of the model to predict benthic cover with high accuracy (Table S5). This indicated that the model effectively captured both dominant and rare classes across multiple time points. Notable exceptions included an overestimation of Dead_Foliose in 2017 (+0.035) and rubble in 2019 (+0.026), as well as underestimation of Dead_Tabulate and SoftCoral_Plating (Table S5). Several live coral categories, such as Branching, Foliose, and Massive morphologies, were modestly overpredicted in 2015 but showed much smaller discrepancies by 2019, reflecting improved generalization over time. Despite small localized biases, the overall magnitude and direction of prediction errors were small or even negligible, and comparable to human error.

We evaluated generalization on 2023 plots with ground-truth masks held out from training (Site 8, Plot 2; Site 34, Plot 1; and Site 35, Plot 1; Figure 6). Despite minor tiling artifacts visible in some scenes, predictions closely matched the withheld annotations and, in several instances, delineated coral colonies that were absent from the human-labeled plots. Predicted boundaries were contiguous and smooth, reducing stair-step effects common in manual labels. Qualitative inspection indicated that the model captures fine-scale morphological detail and within-plot heterogeneity that can be difficult to annotate consistently, supporting its use for mapping in years lacking annotations.

4. Discussion

4.1. Automation of Annotated Coral Reef 3D Modeling

Here, we provide a step-by-step guide on automating the annotation of coral reef imagery using a fully supervised deep learning model. In contrast to traditional manual annotation, which is time consuming, labor intensive, and often inconsistent, our approach enables scalable, repeatable classification of complex benthic features at the pixel level. Our ResNet101-SCSE U-Net model achieved a strong mIoU of 0.83, demonstrating high classification accuracy across diverse reef habitats around Kiritimati and across a temporally resolved dataset. Our model proved to be effective for large-scale benthic monitoring. Moreover, the strong agreement between true and predicted masks gives us confidence in the accuracy of predictions on previously unlabeled 2023 data. When compared with benchmark CNN architectures, our model consistently outperformed others at both the pixel and class levels, highlighting its ability to handle a highly imbalanced dataset.

By automating classification across high-resolution 2D reef reconstructions, our approach addresses a major bottleneck in coral reef monitoring: the need to process large volumes of photogrammetric data with consistency and speed. This is particularly critical for repeat surveys aiming to detect ecological change over time. In reducing dependence on manual annotation, our framework increases scalability and reproducibility. We applied multiple checks and safeguards so that the data were not overfit. Our workflow offers a clear breakdown of the process, along with recommendations for modifying the code (Figure 2). For further details, readers can refer to our Zenodo v1 public archive, where we provide additional resources to help researchers apply this method to their own datasets.

4.2. Comparison with Recent Literature

Most recent studies on benthic classification rely on limited datasets, typically using only 3 to 8 plots, collected at a single time point, and trained on point annotations rather than full pixel-level labels. These models often include only a few or broad benthic classes and are rarely applied to new unlabeled data [30,31,32,56,57]. By contrast, our approach is built on a densely pixel-level annotated and available benthic dataset, comprising 77 fully labeled plots across five years (three time points 2015, 2017, and 2019), with dense pixel-level annotations across 27 benthic classes, later reduced to 16 after training.

Some more recent studies have developed innovative workarounds to overcome sparse annotations, for example, by using superpixels to propagate point labels, reporting improvements of 5–10% mIoU compared to other point-sampled methods [34]. Others have addressed specific technical challenges within the semantic segmentation pipeline: the authors in [58] applied dense labeling to video object segmentation; those in [59] benchmarked architectures for pixel-wise segmentation of two target species; and those in [60] combined improved color correction with colony-level classification and pixel-level labeling. While these studies contribute valuable technical advancements, they remain limited either in ecological scope, taxonomic breadth, or temporal resolution. Our study explicitly incorporates temporal dynamics by training models on multi-year data and applying them to 27 unannotated plots from 2023. Our dataset captures both spatial and temporal ecological gradients, supporting the development of models with greater generalizability and ecological relevance.

In parallel with technical advances, there has been a rise in the number of publications proposing automated classification pipelines for SfM-derived reef imagery. However, few offer a comprehensive, user-friendly workflow like the one presented here. Some recent studies eliminate the need for local training data by leveraging pre-trained models from large citizen science programs, while others focus on imagery from drones or UAVs [32,57]. Despite these innovations, manual annotation remains a key bottleneck for many organizations using SfM photogrammetry in routine monitoring [61]. Our workflow directly addresses this gap, as it is particularly well suited for research groups already generating human-labeled data, allowing them to train their models and automate future analyses. This could also be a limitation, as most coral annotations are made at the point level, while generating pixel-level labels for training is time consuming and labor intensive. While we designed our method with a specific audience in mind, it is adaptable, and we have provided sufficient technical details to make the pipeline accessible to non-specialists. Beyond benthic classification, this approach contributes to the broader effort to bridge the gap between quantitative computer vision methods and applied ecological research.

4.3. Model Improvements, Limitations, and Benchmark Comparisons

One of the most significant improvements we made was addressing the severe class imbalance. Unlike studies using fewer plots or coarser class groupings, our dataset had more variation across space and time. To improve the performance of predicting rare classes, we used rare class boosting during preprocessing by duplicating tiles containing underrepresented classes. We also applied data augmentation and used log-scaled class weights in a combined focal and Dice loss, helping the model to learn from both rare and dominant classes more effectively.

To prevent overfitting and improve generalizability, we added several regularization techniques, including spatial dropout (which disables random neurons during training) and weight decay (which penalizes large weights; Table 1). We tested multiple backbone architectures, namely ResNet50, ResNet152, and ResNet-101, and selected ResNet-101, which balanced depth and computational efficiency [45]. We also used early stopping and learning rate scheduling during training. Specifically, we monitored validation accuracy and loss, stopping training if no improvements were seen. The ReduceLROnPlateau callback lowered the learning rate when progress stalled [62]. These strategies prevented overfitting and resulted in stable convergence. Together, these adjustments, focused on class balancing, efficient data handling, and regularization, significantly improved training speed, model stability, and performance across all classes.

During training, we included all 27 classes and one background, but despite increasing the representation of rare categories, we reduced this to 16 classes to assess changes over time in benthic classes (Tables S1 and S3). We did not retrain the model on the 16 valid classes, which may have led to lower testing accuracy (0.64; Table 2). One challenge was inconsistent expert labeling, particularly in distinguishing rubble or other dead coral types from dead massive coral, which led to the overestimation of mortality and confusion during model training. When morphology was not distinguishable, it was classified as Rubble; otherwise, it was defaulted to Dead Massive or, to a lesser extent, Dead Branching when the colony morphology was still apparent. This may have led to an overestimation of Dead Massive and an underestimation of other dead classes. For example, Dead Soft Coral, Dead Columnar, Non-Coral Massive, and Non-Coral Branching were frequently misclassified as Dead Massive (Figure S3). All of the removed classes had an IoU of 0.0, which was due to their absence in the human annotations, such as three of the non-coral classes (branching, encrusting, and massive), or because they had too few examples (Table S3). Class-level IoU scores were generally high with a few exceptions, and sites with the highest coral cover performed best, suggesting that living coral taxa were much more distinct than dead or non-living substrates.

The three benchmark architectures performed worse than our model. Both U-Net only and FCNN showed low testing accuracy and mean IoU scores (0.43 and 0.39 for accuracy; 0.48 each for mean IoU; Table 3) [36,54]. This demonstrates their limited ability to capture the diversity and uneven distribution of benthic classes. Interestingly, DenseNet achieved the same testing accuracy as our model (0.64), which reflects pixel-level agreement. However, the mIoU scores differed significantly, where DenseNet had an mIoU of 0.48, compared to 0.82 for our model. As the mean IoU is calculated at the class level, this difference shows the ability of our model to outperform standard architectures and highlights the importance of modifying standard CNN models to the specific data characteristics, such as when dealing with highly imbalanced classes, as is common in benthic classification. DenseNet may have struggled with edge segmentation, leading to poor mask boundary overlap and explaining the discrepancy between its pixel accuracy and class-level performance (mIoU). By contrast, our model produced sharper, more confident predictions that better handled edges, resulting in improved overlap between predicted and true masks. This indicates that our model generalizes across classes, rather than only performing well at pixel-level accuracy.

4.4. Application to New Data and Ecological Implications

The temporal resolution of our dataset and the permanent nature of our plots allowed us to effectively classify 28 non-annotated plots from 2023. The differences between the predicted and true masks were negligible, giving us confidence in the accuracy of the newly predicted masks (Figure 6). Most recent studies in the emerging field of automated classification within the SfM pipeline do not include time series data or application to large unlabeled data. One notable exception is Coralscapes (currently a preprint), which shows promising results and strong performance on new, unlabeled data. Coralscapes captures 39 classes using semantic segmentation for generalizable performance with GoPro Hero 10 cameras. However, the timing of the surveys is not specified in the preprint, and the dataset does not appear to include any time series component.

Improving the performance of the model on new data may require additional manual annotations and training data that better reflect differences in data conditions, such as lighting, habitat types, or image quality. Annotating a small subset of new plots can help to assess model accuracy and guide decisions about broader application. Much like other machine learning algorithms, some local variability will always decrease the efficacy of the model when new data are introduced [63,64]. However, as models and machine learning algorithms improve, there may be simple adaptations to models such as this one. This model will be used in future expeditions for our group, and the level of detail left in this paper is meant to guide future colleagues but also inform other coral ecologists of the steps necessary to make benthic annotations faster.

Overall, the model predictions aligned well with ground-truth proportions for most benthic classes, with differences generally within ±0.01% across years (Figure S4; Table S5). This high level of accuracy indicates that the model was able to reliably reproduce community composition and capture the relative abundances of both dominant and rare groups. A small number of classes exhibited some minor divergence, reflecting systematic biases: Dead_Foliose was slightly overestimated in 2017, and rubble in 2019, while Dead_Tabulate and SoftCoral_Plating were consistently underestimated, likely due to their rarity and morphological similarity to other classes. Notably, predictions for major live coral groups (Branching, Foliose, Massive) were robust and converged closely with the observed values, demonstrating the ability of the model to generalize well across years and sites. Taken together, these results demonstrate that the model had strong predictive accuracy, with only a few systematic sources of error, reinforcing its suitability for quantitative assessments of benthic composition and its application in long-term monitoring frameworks.

Using the 2023 hold-out plots (not used in training), we found that the model works well on new data and adds value when labels are incomplete. Small tile artifacts are visible, but the predictions usually have smooth, continuous edges and, in several cases, outline colonies that human labeling missed (Figure 6). This shows that the model can pick up fine detail that is hard to capture by hand in crowded, mixed habitats. A practical use of the model is to let it pre-label new images and have experts review and correct the labels, rather than labeling everything from scratch. We should still guard against over-smoothing, which could merge small colonies or narrow gaps. Potential improvements could be to reduce tiling artifacts (more overlap with weighted blending, test-time augmentation) and sharpen edges (boundary-aware loss or post-processing). For measurement, we will pair mIoU with the boundary F-score, add colony-level detection/recall, and use uncertainty maps to flag low-confidence areas. Together, these steps support using the model to map years without labels while limiting false positives and smoothing errors.

5. Conclusions

There has been a rise in the number of CNN benthic classification method publications, each with its own advantages and disadvantages. Here, we argue that our method is advantageous for datasets like ours that already have a backlog of manual annotation, which we suspect is quite a big issue in the field. While our method requires a substantial amount of training data, it remains a feasible option due to the availability of previously annotated datasets. This approach was, in part, inspired by the knowledge that such datasets (previously annotated) exist at other institutions and programs. This CNN is directly transferable to coral or benthic classification with trained data; however, given the high accuracy, this method could be suitable for many ecological repositories with multiclass RBG (Red, Green, Blue) images and a strong backlog of annotations.

Our automated classification method yielded high performance for analyzing benthic cover and complexity time series data taken in Kiritimati, Kiribati. The success of this approach was largely driven by the substantial effort invested in high-quality field image collection, which was critical for producing high-resolution orthomosaics. While applying this classifier to other datasets will depend on factors such as image quality, taxonomic resolution, and the availability of training data, our step-by-step decomposition of the methodology enhances accessibility for field ecologists, biologists, and other researchers without a computer science background. Although unbalanced data posed challenges in model training and optimization, advances in deep learning are likely to simplify such refinements in the future. To facilitate broader adoption and further development, we have made the model publicly available in Zenodo (doi: 10.5281/zenodo.15238744).