Kelp forests support diverse and productive ecological communities throughout temperate and arctic regions worldwide. Environmental anomalies such as cyclones, storms, marine heat waves and climate change have a detrimental effect on benthic marine life including kelps [1
]. Significant declines in kelp bed were observed around the globe in recent decades, with the main drivers identified as eutrophication and climate change related environmental stressors. For instance, large-scale disappearance of kelp was observed in 2002 in the southern coast of Norway [2
]. In Spain, large scale reductions in two main species of kelp have also been observed since the 1980’s [3
Similarly, kelp populations in Australia have decreased as a consequence of climate change driven environmental stressors. In the east coast of Tasmania, the coverage of giant kelp Macrocystis pyrifera
in the present decade is around 9% of the coverage in the 1940’s [4
]. This decline is consistent with the intrusion of warmer, nutrient poor water from the East Australian Current, which now extends 350 km further south than in the 1940’s [5
]. Wernberg et al. [6
] reported a rapid climate-driven transition of kelp forests to seaweed turfs in the Australian temperate reef communities with kelp forests showing a 100 km poleward contraction from their pre-heatwave distribution on the Western Australia coast. This trend is alarming for the numerous endemic species that rely on kelp forests for support. Loss of kelp forests is also a major threat for Australia’s fishing and tourism industries, which generate more than 10 billion Australian dollars per annum [7
]. There is thus a pressing and immediate need for monitoring programs to document changes in kelp dominated habitats along coastlines worldwide and especially in temperate Australia.
Autonomous underwater vehicles (AUVs) are emerging as highly effective tools for monitoring changes in benthic marine environments, because (i) they can autonomously conduct non-destructive sampling in remote marine habitats; (ii) they can repeatedly survey the same spatial region to detect change over time; and (iii) they are fitted with a range of instrumentation to acquire both physical and biological data. AUVs were used to monitor the marine benthos across temperate and tropical environments in Australia [8
]; to survey invasive pest species [10
]; to document rapid loss of corals associated with warming events [9
]; to describe benthic community structure at depths greater than 1000 m [12
]; and assess environmental impacts of the Deepwater Horizon oil spill [13
]. In a large-scale study of deep waters, the distribution patterns of kelp forests were investigated to provide useful insights on the effect of environmental changes on the kelp population [14
]. The survey took an extremely long time to complete as marine biologists had to manually classify images and to identify kelp from imagery.
AUV driven monitoring can generate large quantities of imagery. For example, an AUV deployed in Western Australia collected more than 15,000 stereo image pairs each day and was deployed between 10 and 12 days each year [9
]. Manual analysis of such a large number of images per deployment (150,000 to 200,000 stereo image pairs) takes a significant amount of time and effort and is the major bottleneck in data acquisition from AUV surveys. To promptly identify changes in benthic species, especially dominant habitat formers (such as kelps and corals), it is necessary to match image-analysis time to surveying time so data can be analyzed rapidly and identification of change patterns can be accomplished. Automatic classification is critical to speed up image analysis and consequently automatic classification of benthic species has raised interest in ecologists and computer scientists (such as [15
]). Nonetheless, automated classification of AUV collected imagery is challenging because images are captured in dynamic shallow water with little to no control on lighting and significant variations in what is visible and how it is perceived.
In this paper, we tackle the challenge of automatically annotating underwater imagery for the presence of kelp to detect changes in the coverage of Australian kelp forests. The common practice is to study the distribution and density of benthic species, which involves manually annotating a smaller dataset and then extrapolating these results to make inferences about the sites under study. Automating the process of determining kelp coverage will significantly decrease image processing times and will allow for large scale analysis of datasets and for early identification of changes in kelp cover. To automate this process, it is paramount to select appropriate features. In computer vision tasks, the general trend has shifted from conventional hand-crafted features to off-the-shelf deep features [20
]. Hand-crafted features which usually encode one aspect of data (i.e., color, shape or texture) were a popular choice as image representations for benthic marine species recognition tasks in the works of [15
]. Moreover, given that hand-crafted features are designed specifically for a current task at hand, they generally do not perform well when applied on a different task. Recently, Convolutional Neural Networks (CNNs) and features extracted from pre-trained CNNs have become the preferred choice for benthic marine image classification tasks, e.g., [19
]. These off-the-shelf features are image representations learned by a deep network trained on a larger dataset such as ImageNet. Off-the-shelf CNN features are generic and have shown better performance as compared to hand-crafted features on a variety of image recognition tasks [20
]. In this paper, we propose to apply image representations extracted from deep residual networks (ResNets) to further improve the automatic annotation of benthic species. Besides better performance, one big advantage of ResNets is their faster training time and ease of optimization. Figure 1
depicts the evolution of classification pipelines for automatic benthic marine species annotation.
The main motivation for using ResNet as a base network to extract features for kelp classification is its superior performance over previous deep networks [26
]. Moreover, the feature extraction is fast due to the low computational complexity of ResNets and the reduced number of floating point operations (FLOPs). Also, the feature extracted from ResNet is 2048-dimensional, which is half of the traditional 4096-dimensional feature vector of previous networks such as VGG16 [27
]. These compact features result in reduced memory requirements for storing the features of large benthic marine datasets.
The main contributions of this paper are:
The first application of deep learning for automated kelp coverage analysis.
A supervised kelp image classification method based on features extracted from deep residual networks, termed as Deep Residual Features (DRF).
A comparison of the classification performance of the DRF with the widely used off-the-shelf CNN features for automatic annotation of kelps.
Experiments demonstrating DRF’s superior classification accuracy compared to previous methods for kelp classification.
We compare hierarchical image classification with multi-class image classification and report the accuracies and mean f1-scores for two large datasets.
An application of our proposed method to automatically analyze kelp coverage across five regions of Rottnest Island in Western Australia.
We demonstrate the performance of the proposed kelp coverage analysis technique using ground truth data provided by marine experts and show a high correlation with previously conducted manual surveys.
The paper is organized as follows. In Section 2
, we will briefly review related work. In Section 3
, we present our proposed approach and explain the features extracted from deep networks. We then report the experimental results and kelp coverage analysis. In Section 4
, we discuss the next steps required to implement our proposed method to a platform to rapidly analyze benthic images. Section 5
concludes this paper.
The use of AUVs to survey benthic marine habitats has allowed scientists to investigate remote locations such as off-shore and deep sites, which are beyond the limits of traditional SCUBA diving. Nonetheless, the efficiency of image collection does not match the availability of data for ecological analysis, as image classification is time consuming and costly given that it is performed manually by marine experts. Additionally, manual classification has other disadvantages such as observer discrepancies and biases. Automated analysis of imagery is thus essential to fully benefit from the advantages of remote surveying technologies such as AUV’s. In this study, we have addressed this problem by evaluating a machine learning automated image classification method using Deep Residual Features (DRF) for a key marine benthic species: the kelp Ecklonia radiata.
We have demonstrated that the image representations extracted from pre-trained deep residual networks can be effectively used for benthic marine image classification in general and kelps in particular. These powerful and generic features outperform traditional off-the-shelf CNN features, which have already shown superior performance over conventional hand-crafted features [19
]. The sibling and inclusive hierarchical training methods further enhance performance when compared to flat multi-class classification methods. The sibling and inclusive training methods show comparatively similar performance. However, the sibling method is superior because it has lower training time than the inclusive method. Furthermore, estimations of kelp cover by automated DRF classification closely resemble those of manual expert classifications with the added advantage of faster processing times. This work provides evidence that automatic annotations may save resources and time while providing effective estimates of benthic cover.
This method was also applied on a dataset to compare kelp coverage for multiple sites, across three depths and for a consecutive time series of four years (2010–2013) at Rottnest Island. The patterns observed showed differences in percent cover of the kelp Ecklonia radiata between sites (with higher percentage cover of kelp in shallower sites compared to deeper sites) and no considerable change of kelp cover across years. These trends were similar to those observed by manually classified data once more confirming the usefulness of automated image classifying methods and the ability to use them for ongoing monitoring of kelp beds with AUV technology.
In this study, we found no evidence of catastrophic loss of kelp over the years at any of the sites surveyed at Rottnest Island. These results are comparable to previous estimates of change in E. radiata
cover across depth in Australia, performed with manually classified images [14
]. They are in contrast with trends of significant and continuous kelp decline reported in the region after an extreme marine heatwave which resulted in widespread mortality of benthic species including corals, seagrasses, invertebrates and kelp [6
]. The loss of kelp in Western Australia resulted in a range contraction of 100 km [6
] and in crab and scallop fishery closures of benthic species associated with kelp habitat. Importantly, the kelp loss was reported in habitats shallower than 15 m, with little attention to the response of deeper habitats to the heatwave [9
]. This may be why our results contrast with studies reporting catastrophic loss of kelp, since our shallowest locations were at 15 m of depth, and most in situ studies take place even shallower (about 12 m). Additionally, all our sites were located off-shore (even the shallow ones), which may indicate that off-shore sites are less impacted by environmental pressures. This may be due to the lack of other environmental disturbances that coastal habitats are exposed to, due to their distance to shore and human populations. The interaction of several disturbances was shown to cause ecological responses such as wide spread mortality of marine benthic species [41
]. Kelps growing offshore and in deeper locations (>15 m of depth) appear to be less impacted by extreme warming in contrast to coastal shallow reefs [42
]. As a result of the catastrophic consequences that extreme climatic events may have on key habitat building species, such as kelp, deeper marine regions were identified as potential refugia for shallow marine species [43
]. This emphasizes the importance of AUV surveys to provide information on offshore and deep locations which may be influenced by different factors to their inshore counterparts [9
]. The use of automated image analysis for processing AUV images will streamline the processing of these images to efficiently identify patterns observed in deep and remote locations and compare them with patterns observed in shallow and inshore sites.
The rapid characterization of ecological changes is crucial in light of the catastrophic threats to marine biodiversity posed by the rise of extreme climatic events driven by climate change and other anthropogenic stressors. Technology has enabled the rapid collection of images even in remote locations through autonomous underwater vehicles, remotely operated vehicles, automated cameras and even satellite imagery. The subsequent annotation of such imagery is typically time consuming and consequently, the automation of marine species classification from digital images has become a priority. This study focuses on the kelp species E. radiata
, which is the dominant habitat builder of temperate reefs in Australia, though automated classification of marine species was applied to other important marine species. For example, progress in automated tropical coral identification has resulted in accurate classification the level of genera [46
]. Other successful automated classification techniques for coral reefs include the collection of multifaceted data, minimum manual classification effort (around 2% of pixels) and machine learning techniques which result in cm-scale benthic habitat maps of high taxonomic resolution and accuracy of up to 97% [47
]. Similarly, in pelagic species such as fish automated classification has advanced rapidly, with automated fish detection and identification algorithms also measuring basic fish morphological features such as total length [48
]. In contrast, automated methods for identification of marine macroalgae from benthic images still result in low agreement [46
], highlighting the need for more research into unequivocal definitions of algal groups for image classification.
Although the proposed DRF classification method allowed us to compare kelp cover in different sites and across different years providing marginal differences with the estimations from manual annotations, there were some errors associated with the proposed technique. We observed an over-prediction of kelp at high percentage cover and under-prediction at low cover. Nonetheless, the over prediction was smaller when data was divided per site and in some sites was negligible (4 and 5). Overall, the estimated kelp cover closely resembles manual classification and taking into consideration the cost effectiveness of automated DRF classification methods, the benefits of the automated classification method out-weight the drawbacks. As such, automated classification of kelp from AUV-derivated images constitute a cost-effective method for estimations of kelp abundance across space and time.
A comparison of the best overall accuracies of hierarchical classification across the two used datasets shows that both the sibling and inclusive DRF classifiers has shown better classification accuracy on Benthoz15 dataset as compared with Rottnest Island dataset. For example, the inclusive DRF classifier for Benthoz15 dataset (Table 4
) has an absolute gain of 15% over the respective classifier for the Rottnest dataset (Table 5
). This substantial difference is possibly due to the high presence of the brown algae Scytothalia dorycarpa
in the Rottnest Island data. Scytothalia dorycarpa
is very similar to kelp in appearance and usually occurs in areas of the sea floor with high cover of kelp. Therefore, marine scientists may mis-classify it as kelp in poor quality images. This misclassification is possible if the point falls on the edge of Scytothalia dorycarpa
, where the boundary between the two species is not clear. The expert misclassification of Scytothalia dorycarpa
as kelp may also explain the over-prediction of kelp by the DRF classification method at high percentage cover. The over-prediction of the automated classification is actually an overestimation of the kelp cover by the manual annotation method. The subjectivity in the classification is removed by the automated analysis, which uses several features to classify kelp. Figure 8
illustrates the similarity of appearance of these two species.
Poor quality images (low light and resolution) will also affect the manual classification of other classes of algae such as ‘turf matrix’, ‘fine branching red algae’ or other canopy forming brown algae. These and other algae classes are not as common as kelp at the sites surveyed at Rottnest Island. Thus, misclassification associated with manual annotations may also explain the over prediction of kelp at low percentage covers. At low cover of kelp, a turf and foliose matrix of red algae occurs on the rocks. In areas of low kelp cover it is easy for an expert to distinguish kelp from other classes, but perhaps due to the imbalance of data for training the classifier sometimes other classes are classified as kelp resulting in over-prediction by the DRF classification method. These issues highlight the need for larger training datasets for deep learning-based automatic annotation. Extensive and comprehensive training sets will allow for better classifier training and give the opportunity to increase the amount of biota classified automatically (e.g., other algae species, corals, sponges, invertebrates such as sea urchins, and lobsters). Future work will explore multi-class classification of benthic marine species across diverse benthic habitats so methods based on deep learning algorithms can be applied to numerous ecological problems that include other benthic marine species. Scientists who use data extracted from image classification should keep these considerations in mind when manually annotating images since these datasets are extremely valuable for deep learning-based automatic classification.