Deep and Machine Learning Image Classification of Coastal Wetlands Using Unpiloted Aircraft System Multispectral Images and Lidar Datasets

Abstract

The recent developments of new deep learning architectures create opportunities to accurately classify high-resolution unoccupied aerial system (UAS) images of natural coastal systems and mandate continuous evaluation of algorithm performance. We evaluated the performance of the U-Net and DeepLabv3 deep convolutional network architectures and two traditional machine learning techniques (support vector machine (SVM) and random forest (RF)) applied to seventeen coastal land cover types in west Florida using UAS multispectral aerial imagery and canopy height models (CHM). Twelve combinations of spectral bands and CHMs were used. Our results using the spectral bands showed that the U-Net (83.80–85.27% overall accuracy) and the DeepLabV3 (75.20–83.50% overall accuracy) deep learning techniques outperformed the SVM (60.50–71.10% overall accuracy) and the RF (57.40–71.0%) machine learning algorithms. The addition of the CHM to the spectral bands slightly increased the overall accuracy as a whole in the deep learning models, while the addition of a CHM notably improved the SVM and RF results. Similarly, using bands outside the three spectral bands, namely, near-infrared and red edge, increased the performance of the machine learning classifiers but had minimal impact on the deep learning classification results. The difference in the overall accuracies produced by using UAS-based lidar and SfM point clouds, as supplementary geometrical information, in the classification process was minimal across all classification techniques. Our results highlight the advantage of using deep learning networks to classify high-resolution UAS images in highly diverse coastal landscapes. We also found that low-cost, three-visible-band imagery produces results comparable to multispectral imagery that do not risk a significant reduction in classification accuracy when adopting deep learning models.

1. Introduction

Coastal habitats, particularly mangals and grasslands, are some of the world’s most important ecosystems due to their function in nature. Coastal wetlands are home to a variety of fish and wildlife, and their vegetation helps enhance water conditions by absorbing nutrients from runoff that might otherwise cause harmful algal blooms offshore [1,2], among other ecosystem services such as nursery grounds for shrimp, shellfish, crustaceans, and fish, as well as nesting areas for a diversity of birds [3]. A large portion of the world’s coastal wetlands have been impacted by anthropogenic and environmental disturbances [4,5,6,7,8]. Accurate, rapid, and frequent access to the physical status of coastal areas is needed for efficient coastal wetland management and restoration, which can be achieved through remote sensing data acquisition and recently introduced analysis techniques [9,10].

Related Works

The spatial and spectral resolution of satellite and airborne remote sensing data has appreciably increased in the last twenty years [11]. Higher-resolution remote sensing datasets are needed to detect small vegetation changes before those changes become irreversible [12]. Sensors mounted on an UAS can provide centimeter-level image resolution and very dense lidar point clouds (~1400 pts/m²), which can play a pivotal role in monitoring and management of small, coastal areas [13,14,15,16]. UAS high-resolution sensor systems have been successfully used for mapping marine habitats in coastal zones [17,18,19], coastline erosion assessment [20,21,22,23], identification of marine habitats [24,25,26], mapping of coastal archaeological sites [27], and invasive plant detection and monitoring [28,29,30,31].

UAS vegetation mapping is typically performed using low-altitude flight missions that have been carefully designed. Those missions collect images that are later processed using structure from motion (SfM) techniques, creating multi-band ortho-rectified reflectance images (orthomosaics) and digital surface models (DSM) of the flown areas [32]. The ortho-rectified images and DSM are, subsequently, analyzed to produce coastal vegetation maps [33,34]. Lidar sensors mounted on an UAS, on the other hand, provide very dense and accurate three-dimensional point cloud datasets that include points within the canopy and on the ground, enabling the production of both high-resolution digital terrain models (DTM) and DSM. These digital models can be used to derive canopy height models (CHM) that are commonly applied to characterize, quantify, and monitor coastal environments [35,36,37]. Moreover, UAS sensor systems usually comprise a combination of lidar sensors and high-resolution cameras or multispectral sensors with the aim of not only colorizing the point clouds but also obtaining additional spectral information. The combination of spectral, structural, and geometrical information becomes an invaluable tool to study and accurately characterize vegetation in general, and it is invaluable in describing coastal wetland vegetation in particular [38,39].

Machine learning has become one of the most broadly adopted AI techniques for image analysis purposes. This was facilitated by substantial developments in remote sensing data acquisition and advances in computational power, which allowed researchers to reach revolutionary results across various areas of knowledge [40,41,42,43]. Different image classification techniques such as a support vector machine (SVM), artificial neural network (ANN), random forest (RF), K nearest neighbor (KNN), and convolutional neural network (CNN) have been used for vegetation mapping [34,44,45]. Vegetation mapping using machine learning techniques such as RF or SVM, for example, has often been implemented on a relatively small number of broad classes that are spectrally different, aggregating features with similar characteristics into a single class, e.g., different types of forests under an aggregated upland forest class [46,47,48,49]. Previous studies have shown that machine learning algorithms can effectively handle a relatively small number of broad classes and the inclusion of a larger number of more specific classes causes a significant drop in accuracy [50,51]. The use of spectrally similar classes, e.g., at the species level, also often causes a significant drop in accuracy [48,52,53]. On the other hand, convolutional neural networks (CNN) do not require preliminary feature extraction as they automatically learn the important features present in an image without human supervision [54]. This important characteristic gives CNN a major advantage over traditional machine learning techniques and demonstrates a remarkable capacity for application to raw image data [55,56]. The CNN process provides a great advantage by lowering the complexity of training data needed and making patterns perceptible to traditional learning algorithms [57,58,59,60].

In this study, we compared the use of two CNN architectures (U-Net and DeepLabV3) and two traditional machine learning techniques (Support Vector Machine and Random Forest) to classify 17 coastal vegetation land cover types, some of them at the species level, located on the west coast of central Florida using UAS-based high-resolution multispectral images and lidar datasets. Different spectral band combinations and the use of canopy height models extracted from two different sources (photogrammetric analysis and lidar acquisition) were also examined. This study tests the assumption that deep learning algorithms will have better performance than traditional machine learning classifiers when applied to a diverse set of coastal vegetation classes, some at the species level, using high-resolution UAS multispectral images and lidar data. Additionally, this study explores the importance of using canopy height models in different vegetation mapping classifiers and the feasibility of replacing costly UAS-acquired lidar datasets with alternative, economically feasible sources of data for producing a canopy height model for land cover classification.

2. Materials and Methods

Image and lidar datasets were collected and preprocessed to create ortho-rectified image mosaics and digital CHM. Different spectral bands and CHM combinations were used to test and compare five different image classification techniques. Training and accuracy assessment datasets were prepared and used to evaluate the results. The workflow in this study spans data acquisition, preprocessing, analysis methods, and accuracy assessments, Figure 1.

2.1. Study Area

This study was conducted on an eight-acre, tidally influenced stretch in the Wolf Branch Creek Coastal Nature Preserve, located in West Florida approximately at latitude: 27°44′40.9′′N and longitude: 82°26′34.5′′W, Figure 2. The preserve comprises different types of coastal land cover types, such as upland cabbage palm hammocks, wet and dry grasslands, mangals, and pine flatwoods, where many of those habitats have already been artificially restored. The study area is also affected by the non-native invasive species Brazilian Peppertree (Schinus terebinthifolia), leucaena (Leucaena leucocephala), and cogongrass (Imperata cylindrica), being the last species listed among the top 10 most pervasive exotic invasive plants in the world [61]. The preserve periodically goes through management campaigns to control invasive plant infestations. There are mangrove species, Figure 3, and native grass species, Figure 4, in the study area. Healthy and treated cogongrass is present at the site, Figure 5. A land cover classification schema involving 17 coastal land cover types,

Table 1. Classes defined for the study.

Land Cover Type or Vegetation Specie	Class Identifier	Scientific Name	Description
Brazilian peppertree	1	(Schinus terebinthifolia)	Invasive evergreen shrub or small tree.
Cabbage palmetto	2	(Sabal palmetto)	Native species of palmetto
Water	3		Water (from any source, sea intrusion, rainwater filling muddy car tracks, brackish water ponds or tidal creeks).
Black mangrove	4	(Avicennia germinans)	Black mangrove above 0.6 m of height. (In forested mangroves, seedlings are defined as individual trees <1.37 m in height [62].
Exposed sand	5		Exposed sand, usually white or light brown.
Organic brown soil	6		Organic brown Ssil, usually dark brown or light black loam.
Cogongrass	7	(Imperata cylindrica)	Very aggressive perennial invasive grass species, light green when healthy, brownish when stressed.
Black needle rush	8	(Juncus roemerianas)	Flowering Juncus, native to North America, distributed along the Gulf Coast (since it is not a brush nor a tree, for this study it was included among the grasses).
Seashore dropseed	9	(Sporobolus virginicus)	Grass. Seashore dropseed not submerged, growing in higher and drier areas, away from water bodies.
Cordgrass	10	(Spartina spartinae)	Grass, commonly found in marshes and tidal mud flats.
Dead vegetation	11		Exterminated or naturally dead vegetation of all types.
Red mangrove	12	(Rhizophora mangle)	Considered a native, grows as a shrub or a tree up to 60 feet tall in tidal swamps.
Short mangrove	13	(Avicennia germinans)	Short plants with a characteristic longer stem compared to its branches and short height
Leucaena	14	(Leucaena leucocefala)	Fast-growing, invasive evergreen shrub or tree with a height of up to 20 m.
Submerged seashore dropseed	15	(Sporobolus virginicus)	Grass. Seashore dropseed dampened or submerged, growing close to water bodies where soils are saturated with water or where the plant is rooted but lives floating on shallow waters. The grass appears healthier, under simple visual inspection, than the ones growing in dry areas.
Shepherd’s needles	16	(Bidens alba)	Healthy, unidentified green grass and bushes of various species that cover the areas between bigger and well-defined patches of vegetation and land cover types.
Broomsedge mixed	17	(Andropogon virginicus)	Broomsedge grass mixed with numerous, less dominant, brown-colored grass and small bushes that cover the areas between bigger, well-defined patches of vegetation and other land cover types.

, was adopted. The classes include seven grass species (six natives and one invasive), Cabbage Palmetto (Sabal Palmetto), two mangrove species (Avicennia germinans and Rhizophora mangle), black mangrove (Avicennia germinans), two invasive brush tree species (Brazilian Peppertree (Schinus terebinthifolia), leucaena (Leucaena leucocephala)), water, dead vegetation, and two exposed soil classes, Table 1. The reserve site is actively managed to control and eradicate invasive plants (Brazilian Peppertree sprouts, leucaena, and cogongrass), so we also have a dead vegetation class.

2.2. Field Data Acquisition

Ground Control Point Establishment: In preparation for the UAS data acquisition missions, a set of twelve ground control points (GCP) were established in the study area, Figure 6. The points were surveyed using survey-grade Topcon HiperLite Plus Global Navigation Satellite System (GNSS) receivers with a reported accuracy for a static survey of 3 mm + 0.5 ppm horizontal and 5 mm + 0.5 ppm vertical [63]. A set of three receivers capable of processing signals from the GPS and GLONASS constellations with multipath mitigation were used to occupy each GCP for at least 2 h, collecting measurements every 5 s, Figure 6. The collected GNSS observations were processed using the National Oceanic and Atmospheric Administration’s (NOAA) Online Positioning User Service Projects (OPUS Projects) (https://www.ngs.noaa.gov/OPUS-Projects/OpusProjects.shtml), (accessed on 8 July 2022) to produce control point coordinates with an uncertainty below the centimeter level.

Multispectral Image Acquisition: A Micasense RedEdge-MX multispectral camera capable of acquiring five-band images was mounted on a DJI Inspire 2 (a multi-rotor UAS) and used to acquire the images analyzed in this study,

Table 2. UAS Mission parameters.

Altitude AGL	40 m
Flight speed	4.17 m/s
Forward overlap	80%
Cross overlap	80%
Ground sample distance	2.78 cm/pixel
Time between capture	1.28 s
Distance between tracks	7.11 m
Mission flight time	20 min (8 Acres)
Number of images (total, 5 bands)	4420

. The spectral bandwidth,

Table 3. Spectral bands in which the MicaSense RedEdge multispectral camera operates (courtesy of www.micasense.com, accessed on 8 July 2022).

Band Name	Center Wavelength (nm)	Full Width at Half Maximum (FWHM) (nm)
Blue	475	20
Green	560	20
Red	668	10
Near IR	840	40
Red Edge	717	10

, and the central wavelength for each of the five bands that were used and the Micasense-MX camera features,

Table 4. MicaSense RedEdge multispectral camera specifications (courtesy of www.micasense.com, accessed on 8 July 2022).

Camera Features
Ground sampling distance	8.2 cm/pixel at 120 m (Above Ground Level AGL)
Lens focal length (mm)	5.5
Lens horizontal field of view (HFOV)(degrees)	47.2
Imager size (mm)	4.8 × 3.6
Imagery resolution (pixels)	1280 × 960

. The UAS was equipped with a Downwelling Light Sensor (DLS) to measure the downwelling irradiance for every image spectral band at the time of acquisition. The DLS is mounted on top of the UAS, pointing toward the sky, Figure 6E. The irradiance value measured at the time each image is acquired is saved in the image metadata and is subsequently retrieved and applied to correct the images. The coordinates of each GCP are used to assure the creation of accurate photogrammetric products with the smallest possible local distortions.

Lidar Data Acquisition: lidar data was collected using a Phoenix Lidar Systems Scout-32 mounted on a DJI M600 UAS. The sensor is made of a combination of a Sony Alpha ILCE-A6000 CMOS 24-megapixel RGB camera and a Velodyne HDL-32E laser scanner and can collect survey-grade data while flying at an altitude of up to 65 m above ground level. The Velodyne HDL-32E sensor is composed of 32 laser range finders arranged in such a way as to be able to cover a 41.33° vertical field of view along the flight direction when mounted on the UAS,

Table 5. Mission and UAS-mounted lidar sensor parameters, courtesy of Velodyne lidar, Inc.

Flight Mission and Sensor Parameters
UAS flight altitude (AGL)	40 m
UAS flight speed	6 m/s
Sensor measurement range	Up to 100 m
Sensor vertical field of view	41.33°
Sensor angular resolution (vertical)	1.33°
Sensor angular resolution (horizontal/azimuth)	0.08°–0.33°
Sensor field of view (horizontal)	360°
Sensor horizontal beam divergence	2.79 milliradian
Sensor vertical beam divergence	1.395 milliradian
Number of returns recorded by the sensor per pulse	2 returns
Number of returns recorded by the sensor per second	695,000 returns/s (single return mode) 1,390,000 returns/s (double return mode)
Beam footprint at 65 m	18.1 × 9.1 cm (dimensions)

. The UAS measures up to 2 returns per pulse emitted and can record dual returns for objects spaced by at least 1 m [64]. This laser scanner generates a relatively dense cloud of points, acquiring a point cloud of 46,090,005 points covering an area of 32,375 m² (approximately 8 acres), for an average lidar point density of approximately 1400 points m⁻². Two pyramid-shaped targets [65] and two flat targets with highly contrasted patterns were positioned in the study area and located using an RTK GNSS survey to assess lidar data processing accuracy.

2.3. Preprocessing of Image and lidar Datasets

The multispectral aerial images were processed using a structure from motion (SfM) technique using the Agisoft Metashape software (version 1.7.2 build 12070, St. Petersburg, Russia) [66] to create an orthomosaic with the five spectral bands of the study site. SfM is a computer vision technique that employs series of overlapping images, taken in sequence from different viewpoints, and a highly redundant bundle adjustment to estimate the 3D structure of a scene [67]. The bundle-block adjustment process calculates the coordinates of the ground features in object space as well as the geographic position and orientation of the camera using matched features detected by the algorithm in each sequential image, which are commonly called key points [68]. The GCP coordinates and the geolocation metadata of each image are used to solve the over-constrained optimization process and ultimately transform the image pixel coordinates into a geographic or projected coordinate system.

The 3 cm per pixel resolution orthorectified image mosaic was generated and was free from perspective distortions associated with camera tilts and feature relief, Figure 7. Agisoft also produces a dense, three-dimensional point cloud that represents the features in the scene. The point cloud processing is performed by first running a dense image matching process to identify conjugate image locations within the scene and then by using the image exterior orientation parameters to generate points with real-world X, Y, and Z coordinates [69]. The point cloud was later used to produce a 6 cm DSM to be used in the analyses.

The navigation data of the lidar mission trajectory was analyzed using the Novatel Inertial Explorer software [70]. The highly accurate trajectory solution was then transferred to the Phoenix Lidar Spatial Explorer software [71] to produce a georeferenced point cloud. The georeferenced point cloud was processed using the LAStools software [72] and the function “lasnoise” to reduce noise. Noisy points were filtered out by removing all points without at least 10 other neighboring points within 1 m³, resulting in a cloud averaging 1400 points m⁻². Finally, the points were classified as ground and non-ground points using the “lasground” algorithm implemented in the LASTools software [72]. Ten cm pixel size DSM and DTM models were created using the Global Mapper software (version 23.01) [73]. Subtracting the DTM from the DSM results in a CHM that was used as an additional layer (UAS_CHM) in the classification process.

A second CHM was created by combining the following two sources of data: the DSM generated by processing the UAS-acquired multispectral images and the DTM produced from already existing airborne lidar data acquired in 2017 and published by the Southwest Florida Water Management District (SWFWMD) [74]. The airborne topographic lidar data was collected by Dewberry [74] in 2017 using a dual-channel airborne mapping system Riegl VQ-1560i sensor capturing more than 1.3 million measurements per second, flying at an altitude of 1300 m above the ground, with 60 percent overlap. A DTM of 75 cm per pixel was created using this airborne lidar dataset with a mean point density of 16-point m⁻². Subsequently, using Global Mapper, this DTM was subtracted from the 6 cm DSM created from the SfM of our multispectral images. The resulting CHM (AB-CHM) was resampled using ArcGIS Pro to reach a resolution of 3 cm per pixel.

2.4. Band Combinations

Three different experiments on data combinations were conducted in this study,

Table 6. Experiments, description.

Experiment	Description	Abbreviation
1	Only spectral band combinations.	(Exp1-OSB)
2	Same spectral band combinations as in experiment 1 plus airborne low-density lidar products and photogrammetry-based DSM.	(Exp2-SB_AB_CHM)
3	Same spectral band combination as in experiment 1 plus high-density UAS lidar products (DSM and DTM).	(Exp3-SB_ UAS_CHM)

. Experiment 1 tested a set of six spectral band combinations, while experiments 2 and 3 used the same spectral band combinations as experiment 1 plus the canopy height models extracted from UAS-lidar DSM (experiment 2) and image-based DSM (experiment 3),

Table 7. Spectral band combinations studied.

Spectral Band Combinations for Exp1-OSB
B_G_R_RE_NIR	Composite of all the bands acquired by the multispectral sensor as follows: blue, green, red, red edge, and near infrared. Micasense MX RedEdge-MX (AgEagle Sensor Systems Inc., 2022) and Sentera’s 6X (SENTERA, 2022) multispectral sensors.
B_G_R	Composite containing the visible blue, green, and red, bands traditionally captured by most low-cost consumer cameras. Emulates the use of built-in generic photographic UAV-mounted cameras.
G_R_RE G_R_NIR B_G_R_NIR B_G_R_RE	Emulates the use of relatively inexpensive cameras (compared to full multispectral sensors) on the market, designed for specialized purposes with specific band combinations, built to meet customers specifications.
Band Combinations for Exp2-SB_AB_CHM and Exp3-SB_UAS_CHM
For the two experiments that include canopy height models the 6 band combinations remain the same as above except for the addition of CHM.

. It should be noted here that the second experiment was designed to use the following two digital products from less expensive sources: a DTM from the publicly accessible lidar dataset directed by the Southwest Florida Water Management District (SWFWMD) in 2017 and a DSM from the Agisoft Metashape SfM analysis created by processing our UAS-acquired multispectral images as indicated in the previous section.

2.5. Training Dataset Preparation

The segment mean shift (SMS) algorithm [75] implemented using the ArcGIS Pro software [66] was applied to the RGB composite of the study site image. The SMS algorithm was used to identify likely objects in the image by clustering together neighboring pixels with similar spectral and spatial characteristics. These three segmentation parameters were tested, namely, spectral detail, spatial detail, and minimum size in pixels, which were simultaneously adjusted until a reasonable object size of 3 m² was agreed on and used to control the size of the polygons. A total of 24,298 objects covering the whole image were produced, and a group of 3094 objects or polygons (around 180 per land cover type) were randomly selected and manually labeled to train the classifiers. The polygons were labeled as a specific class based on their dominant species, Figure 8. Labeling was conducted in the lab and verified through site visits in collaboration with the Wolf Branch Nature Preserve manager. The same segmentation results were also used to post-process the pixel-based classification by labeling each object with the class to which the majority of its pixels are classified.

Training data was extracted for every band combination, Table 6. The 3094 training sample polygons were used to train the RF and SVM algorithms (see next section). The same training polygons were used to create training tiles (256 × 256 pixels) from the images with a 128-pixel stride and 30° rotation angles to train the deep learning networks (U-Net and DeepLabv3, see next section), Figure 9. The stride and rotation features were used to augment the training data. Training tiles were only retained for the instances where the polygons and tiles overlapped, resulting in a final set of 27,948 training tiles.

2.6. Data Training and Classification

The U-Net [76] and DeepLabv3 [77] CNN deep learning architectures, widely used for semantic segmentation in the remote sensing community [78,79,80,81], as well as the support vector machine (SVM) [82], and random forest (RF) [83], machine learning algorithms were trained and tested using the same training polygons and accuracy assessment points for all the classifiers used in this study. Training and classification were performed using the ArcGIS Pro software on a computer machine with an Intel(R) Core (TM) i9-10900KF CPU, running at 3.70 GHz with an NVIDIA Titan X 12 GB GPU and 64 GB RAM. The following briefly introduces the classification techniques used in this study.

• Machine Learning Classifiers:

The Random Forest (RF) classifier [84,85] is a decision tree algorithm that applies the most probable class label to each pixel in an image using a set of rules developed during the training process. This process relies on previously labeled training data, where the RF algorithm chooses features and observations randomly to build numerous, uncorrelated decision trees. The final output of the random forest system is formed by the output chosen by the majority of the decision trees. This algorithm is found to be adaptable, fast, simple, the least demanding in terms of computer resources, and a practical machine learning algorithm that generally produces quality outcomes with only a few parameters. After initial experimentation with different parameters, our study adopted a maximum number of trees equal to 50, a maximum tree depth equal to 30 rules (or splits), and a maximum number of samples per class equal to 1000 to be used in the training process. Applying the “Train Random Trees Classifier” from the ArcGIS Pro Image Analyst Toolbox, RF models were trained using the previously labeled training polygons as input training features. The trained model for each band combination was applied to perform pixel classification on the respective raster file using the tool “Classify Raster”, of the ArcGIS Pro, Image Analyst Toolbox.

Support Vector Machine (SVM) is another machine learning algorithm that functions by identifying the hyperplane that best segregates the close observations that belong to separate classes [86,87]. For well-defined and properly labeled classes, the quantity of observations needed to train an SVM classifier is relatively small, yielding reasonably good outcomes compared to more data-demanding algorithms such as CNN [33,88]. This algorithm was applied similarly to the random forest machine learning technique in ArcGIS Pro. The tool “Train Support Vector Machine Classifier” from the ArcGIS Pro Image Analyst Toolbox was employed using the labeled training polygons as input, and the tool “Classify Raster” from the ArcGIS Pro Image Analyst Toolbox was used to classify the rasters after the models were generated.

• Deep Learning Architectures:

The U-Net architecture was introduced in 2015 for biomedical image segmentation at the Computer Science Department of the University of Freiburg to segment cell images [76]. This architecture was designed to minimize the need for classified training data by using the labeled samples more efficiently. The U-Net architecture involves the following two paths: a contracting and an expanding path. The contracting path captures the context within the sample images by applying a sequence of convolution operations and ‘maxpool’ down-sampling to encode the input training images into feature representations. The expanding path allows for accurate localization of each pixel in the image while assigning it a class type. The expanding path uses repeated up-sampling operators that combine high-resolution elements from the contracting path with the up-sampled output, propagating context information to higher resolution layers [76,89].

To train the U-Net architecture, the algorithm was set to balance the classes and ignore the background class. Balancing the classes works by under-sampling the training data from the most numerous classes to pair them down to match the less-represented classes in the dataset. The use of class balancing in the training is intended to avoid biasing the classifier, which would accentuate the most represented class, leading to skewing the classifier decision boundary and producing overfitting [90,91]. Initially, in our study, different training rates were experimented with, and based on those trials, training rates ranging between $8\times {10}^{-6}$ and $9\times {10}^{-3}$ were used to train the deep learning models. Similarly, a different number of epochs were tested, and the results revealed that 25 epochs were an adequate amount to train the network in most cases. We used a batch size of 8 in most of our training. These parameters were applied to both deep learning architectures (U-Net and DeepLabV3) used in this study.

DeepLab Version 3: [77] DeepLab is the abbreviation for deep labeling, which is a CNN model for general, dense pixel labeling using deep neural network algorithms. One major challenge when using fully convolutional networks for image classification is that input feature maps become smaller while moving through the convolutional and pooling layers of the network, causing a loss of information contained within the images. Another drawback of the down-sampling performed by the pooling layers in convolutional networks is the lower spatial resolution of the output and blurred object boundaries. DeepLab corrects those pitfalls by removing the down-sampling operations from the last few max-pooling layers of the convolutional network, using atrous convolution [77], and instead using up-sampling in subsequent convolutional layers. The application of this technique results in feature maps processed at a higher sampling rate [92,93].

The backbone for the two deep learning architectures was fixed to ResNet34 [94] due to its powerful performance with reasonable computational power and processing time demands [95]. Although a different number of training epochs were tested, we found that training results often stopped improving after 20 epochs; thus, 25 epochs were used for most of the tested models, which took from 9 to 12 h. The pixel-based classification was then applied using the trained models for each band combination.

2.7. Post-Classification Filtering and Accuracy Assessment

A majority filter was applied to each classified output using the segmented objects initially generated by applying the segment mean shift algorithm [96] to improve the quality of the final classified image through dominant class assignment, Figure 10.

An accuracy assessment was conducted on each classified image. A total of 747 random points were created, representing all 17 classes. The points were manually labeled with their corresponding ground truth classes collected through onscreen inspection and field verification. A confusion matrix (error matrix) [97] was created using the randomly generated accuracy assessment points. For each classification result, the overall accuracy (Equation (1)), user accuracy (Equation (2)), producer accuracy (Equation (3)), and the F1-score (Equation (4)) were calculated [98,99,100].

$$Overall Accuracy=\frac{True positives+True negatives}{True positives+True negatives+False positives+False negatives}$$

(1)

$$Precision=\frac{True positives}{True positives+False positives}=User’s accuracy$$

(2)

$$Recall=\frac{True positives}{True positives+False negatives}=Producer’s accuracy$$

(3)

$$F{1}_{score}=2\ast \frac{\left(User accuracy\ast Producer accuracy\right)}{\left(User accuracy+Producer accuracy\right)}$$

(4)

3. Results

3.1. Overall Accuracy

A quantitative evaluation was conducted on the classification experiment results both from using different combinations of five spectral bands (Blue: B, Green: G, Red: R, Near InfraRed: NIR, and RedEdge: RE) in experiment 1 and when adding two canopy height bands (based on SfM solution: AB-CHM for experiment 2 and UAS lidar: UAS-CHM for experiment 3). Four different classifiers (SVM, RF, DeepLabv3, and U-net) were tested in each dataset. In general, the U-net and DeepLabv3 deep learning models require more than 24 h to train a single model. The SVM models were trained for an average of 2 h each, while the RF models only needed an average of 4 min to train. The classification time using a trained deep learning model usually took from one to two minutes, utilizing the GPU to accelerate the process. Classification using RF and SVM did not use the GPU to run the process. RF classifiers needed 3 min, while SVM required between 25 and 40 min.

The same randomly selected accuracy assessment points were used to compute the confusion matrices for all 72 (3 experiments × 6 band combinations × 4 classification techniques) image classifications conducted in this study. The highest accuracy levels achieved by the classification algorithms for the three experiments in the twelve combinations are shown in Figure 11. The user and producer accuracy for each experiment is shown in Appendix A.1, Appendix A.2 and Appendix A.3. The confusion matrices of the trials corresponding to those maps are shown in Appendix B.1, Appendix B.2 and Appendix B.3. The overall accuracy of all tested classifications is shown in Figure 12, and the F1 score is shown in Figure 13 and Figure 14.

Deep learning algorithms, especially U-Net, provided higher overall accuracies compared with the RF and SVM traditional machine learning classifiers, Figure 12, which confirms the findings of several other studies, e.g., Liu Tao et al. 2018 [33], Bhatnagar et al. 2020 [101], and Jozdani et al. For example, U-net’s overall accuracy was more than 83.40% in the first experiment for all band combinations, with a maximum of 85.27% for the B_G_R_RE band combination. This contrasts with the 62.65% overall accuracy obtained by RF applied to the same band combination. Similarly, U-net produced overall accuracies in the range of 83.00% to 85.94% in the second and third experiments, compared to SVM yielding accuracies in the range of 62.92% to 74.56% and RF yielding accuracies ranging from 67.07% to 71.89%.

Adding the near-infrared and red edge bands did not seem to have a consistent effect on the overall accuracy of the two deep learning algorithms, Figure 12, which corroborates the findings of other studies, e.g., Li et al. 2021 [102], Zhao et al. 2018 [103]. Including at least one of the near-infrared or red edge bands had a positive influence on the RF and SVM classifiers. This could be explained by the deep learning architecture’s ability to extract and model the important features in a smaller spectral subset, reducing the need for the relatively more expensive multispectral datasets.

Our research matched the findings of other studies [30,104,105], indicating that classes with contrasting spectral properties such as the cabbage palmetto class (class 2), the exposed sand class (class 5), and the black needle rush class (class 8) are in general more likely to be correctly identified by the classifiers (high producer accuracy) and to be reliably mapped (high user accuracy). On the other hand, classes such as the Brazilian pepper tree class (class 1), cordgrass class (class 10), leucaena class (class 14), and submerged seashore dropseed class (class 15), which have similar spectral signatures to other existing classes, have a much higher likelihood of being correctly classified using the deep learning methods than by traditional machine learning classifiers, Figure 13 and Figure 14. All of the classification algorithms, including deep learning, yielded relatively low user accuracies compared to their producer accuracies for the cordgrass class (class 10), leucaena class (class 14), and submerged seashore dropseed class (class 15); Figure 14, (for more detail see Appendix A), which, as mentioned above, have similar spectral signatures to other existing classes. This indicates a higher level of difficulty for those classes to be differentiated when they do not grow isolated from other vegetation species.

In general, the use of the red edge and near-infrared bands in the three experiments yielded slightly higher accuracies in mapping the Brazilian pepper class (class 1), Cabbage palmetto class (class 2), black mangrove class (class 4), organic brown soil class (6), black needle rush class (class 8), cordgrass class (class 10), red mangrove class (class 12), short mangrove class (13), and submerged seashore dropseed (class 15), which could be an argument to consider these bands when those particular land cover types are of interest and need to be mapped using the RF or SVM classifiers Figure 13 and Figure 14, (for more detail see Appendix A). We observed levels of class mixing among classes with similar spectral characteristics in the visible spectrum. For example, the RF and SVM classifiers show the broomsedge class (class 17) was frequently confused with the cogongrass class (class 7), seashore dropseed class (class 9), cordgrass class (class 10), dead vegetation class (class 11), and the Shepherd’s needles class (class 16) (Appendix B). The misclassification is so severe that the user and producer accuracies (Appendix A.1) of these classes in most cases were very low when the RF and SVM classifiers were used. This inability to isolate class 17 did not replicate itself in the deep learning classifications, where U-net was able to map class 17 with an F1 score of around 70%, Figure 14.

3.2. Effect of Canopy Height Models

When comparing the first experiment that used spectral data only with the results of the second and third experiments, we found that using canopy height as a means to incorporate geometrical information in the classification process did not consistently improve the classification accuracy for the deep learning results, Figure 12. This contrasted with the results of the SVM and RF classifiers, which benefited from the extra CHM information. Our results show that the first experiment produced an overall accuracy ranging from 57.43 to 65.33% for the SVM classifier. The overall accuracy improved to a range of 64.79% to 74.56% when the canopy height model was added in experiment 3. The Random Forest increased its accuracy from a range of 60.91% to 68.14% using the spectral bands only to a higher accuracy range of 67.7% to 74.6% when the canopy height model was included in experiment 3.

In general, the CHM obtained from the UAS lidar dataset had a better effect on the overall accuracy of the maps produced by the DeepLabv3 compared to the U-net results, Figure 12. DeepLabV3 achieved slightly higher overall accuracies in the third experiment for all the band combinations except for the G_R_RE_CH band combination. A clear example of this behavior is the band combination of B_G_R_CH, for which DeepLabV3 achieved only 75.4% overall accuracy in the second experiment, where the CHM was produced from SfM and low-resolution DTM was used, and 83.5% overall accuracy in the third experiment using the UAS-based CHM.

For classes with zero canopy height, such as the water class (class 3) and the exposed sand class (class 5), the performance of the deep learning algorithms did not seem to benefit from the addition of the canopy height model, probably due to their distinct spectral characteristics, which prompted higher accuracy outcomes even from the spectral data alone. Individual class performance reflects little to no improvement in the overall accuracy of the deep learning classifications compared to the more substantial improvement in the RF and SVM classification results when canopy heights were introduced. The introduction of the canopy heights improved the accuracy of the SVM and RF classification of the broomsedge class (class 17) noticeably, although the F1 score never reached the 50% mark. The deep learning algorithms, however, probably benefited from the structural and textural differences among the broomsedge and the other grasses, minimizing the influence of the canopy height on the classification process [106,107].

The Shepherd’s needles class (class 16) is another class where RF and SVM benefited from the use of canopy heights with minimal improvement in the accuracy of the deep learning classifiers, Figure 14. The producer accuracy of the SVM results increased from an average of nearly 52% accuracy in the first experiment to about 85% in experiment 2 utilizing (SfM/low-resolution DTM) canopy heights (Appendix A.1). The user accuracy did not improve as much, with an F1 score increasing from an average of 51% to about 56%. The RF classification showed marked improvement, with average producer accuracy growing from 38% to 83% and the user accuracy from 57% to 65%, Figure 13 and Figure 14. It should be noted that this class (class 16) was developed to represent several heavily mixed bushy vegetation species with no apparent dominance of a single species and with relatively similar spectral characteristics and canopy height values.

The introduction of canopy height information in experiments 2 and 3 had a negative effect on the classification accuracy for dead vegetation (class 11). This class represents areas where grass has been either chemically exterminated or bushes mechanically harvested (mulched), and thus, the area would have low canopy height values. When dead vegetation was classified using the spectral bands only, the results were consistent since dead vegetation areas still have consistent and distinguished spectral characteristics. Introducing the canopy heights confused the classifier since dead vegetation had inconsistent canopy height across the landscape. For example, dead but still standing grasses that had been recently chemically treated can have a different height than piles of cut-down tree branches that have not been mulched yet, Figure 14.

3.3. Invasive Plants Classification

Out of the three invasive species (Brazilian pepper, leucaena, and cogongrass) included in our study, the first two have similar spectral signatures, especially when the Brazilian pepper tree is stressed [108,109,110,111]. Healthy Brazilian pepper trees commonly have a dark green appearance, which is not similar to the healthy bright green appearance of leucaena. On the contrary, Brazilian pepper trees growing in coastal wetlands with roots constantly subjected to salt and brackish water inundation typically have leaves that have turned yellowish, making them look much more similar to the natural leucaena color [110]. For all experiments, RF, SVM, and DeepLabV3 showed a predisposition for Brazilian pepper to be confused with leucaena (high omission error), compared to the rest of the vegetation classes. This tendency is less prominent when U-net is used, with U-net more able to identify Brazilian peppers in almost all experiments better. Brazilian pepper (class 1) is often confused with other classes as well, including cabbage palmetto (class 2), black mangrove (class 4), cogongrass (class 7), and seashore dropseed class (class 9) (Appendix B).

The leucaena class (class 14) shows a high producer accuracy, in many cases, close to 100% for all three experiments (Appendix A.1, Appendix A.2 and Appendix A.3) and most of the band combinations of the deep learning classifications, except for two band combinations in trials using DeepLabV3. User accuracy, however, was frequently below 70% for the deep learning results and averaging 32% for the SVM and RF classifiers. The introduction of canopy heights, in all cases, slightly improved the user accuracy of the leucaena class in all tested models. Leucaena prefers elevated dry areas and grows next to grasses such as the seashore dropseed grass, which may explain the positive effect of introducing canopy heights.

When spatially isolated from other grasses, cogongrass was identified by all the classifiers with relatively high accuracy, Figure 13, even as several other grass species with similar spectral properties existed in the study area. Cogongrass was slightly confused with neighboring patches of other grasses when mixed with different grass types in the area. The deep learning algorithms were able to achieve the highest cogongrass classification results among all tested classifiers, Figure A2 in Appendix A. The inclusion of canopy height did not seem to benefit the deep learning classifiers, although RF and SVM showed accuracy improvement. The cogongrass class was not clearly defined by any spectral band combination.

4. Discussion

We found the U-net and DeepLabv3 deep learning networks to be superior compared to the SVM and RF traditional machine learning techniques, which matches the findings of other studies reporting similar ranges of overall accuracy [33,101]. Even though deep learning requires GPUs to train the models in a considerable amount of time, the quality of the achieved results justifies its use when there is no need for expediency, and higher accuracy mapping levels are required.

In this study, we experimented with a relatively large number of coastal land cover classes (17 classes), achieving an overall accuracy higher than 83% using deep learning networks. Previous studies have found similar or comparatively higher values of overall accuracy, although in most of the reviewed cases the number of classes tested has been smaller, either because the studies focused on identifying a single vegetation species or a reduced number of classes [112,113,114]. The outcomes of this research suggest further investigation to explore the possibility of increasing the number of classes and experimenting with different training data sizes to train the machine and deep learning models. Traditional machine learning techniques have reached results comparable to the ones achieved by those techniques in previous studies, e.g., Durgan et al. 2020 [115], highlighting the effectiveness of their use for wetland vegetation mapping using multispectral UAS-acquired datasets.

Adding the near-infrared and red edge spectral bands and the canopy heights improved the accuracy of RF and SVM classifications, while incorporating these additional datasets had minimal, inconsistent impacts on the deep learning classifications. This observation may indicate the efficiency of current deep learning networks in extracting and modeling the features in fewer bands and highlights the need for further exploration of new deep learning network architectures capable of fully utilizing additional spectral and canopy height information. The use of the red edge and near-infrared bands in the three experiments yielded slightly higher accuracies in mapping a few classes. Some classes were significantly mixed, especially when the machine learning classifiers were used.

Although deep learning networks require intensive computational power and longer training times compared to traditional machine learning algorithms, the consistent and considerable accuracy increases achieved using the deep learning models, even when only the three visible bands are used, highlight the advantages of these models. The advantage of deep learning models goes beyond accuracy improvement to sensor utilization, highlighting the potential of using only inexpensive, visible band cameras without the need for the more expensive multispectral cameras.

Based on our findings and backed by the results obtained by other studies experimenting with adding geometric data to classification models [115,116], we recommend the use of canopy height models due to their positive impact on the classification accuracy of specific classes within the deep learning experiments and their significant contribution to the increased accuracy of the RF and SVM results. We also found that both canopy height models derived from UAS lidar datasets and the one derived from SfM photogrammetric model and low-resolution DTM (from airborne lidar datasets) yielded almost similar overall accuracy results in most experiments. This latter realization highlights the fact that even though UAS-lidar remains the preferred method to assure canopy penetration, enabling the construction of high-resolution DTM and canopy height models, the use of a digital surface model extracted from the SfM solution of high-resolution UAS imagery and the often-available low-resolution DTM models extracted from low-density airborne lidar can provide a low-cost alternative to costly UAS lidar datasets.

Even though UAS remote sensing data acquisition and processing, as well as machine learning techniques, are already widely used in coastal environmental monitoring, it is still imperative to further explore and expand the boundaries of knowledge in vegetation classification and invasive species detection, especially as UAS datasets become more integrated into natural resource management operations. For example, concerns related to how to more adequately group classes together as communities to improve the accuracy of the learning models and how to improve the training dataset preparation process remain active research subjects. Additionally, our results showed confusion in some classes that could be improved with more consideration of the operational characteristics (e.g., invasive plant control and removal efforts) in the study area. Utilizing the same methods applied in this study, invasive species detection can also be explored at different phenological and physiological stages, vegetation mixes, and treatment stages [115,117,118].

5. Conclusions

In this study, we assessed the potential of two deep learning architectures (i.e., U-net and DeepLabV3) compared to other traditional machine learning algorithms (i.e., SVM and RF) for coastal wetland vegetation and land cover mapping. The study helps expand the knowledge about the effectiveness of the use of high-resolution UAS-acquired lidar and multispectral data in the generation of accurate vegetation maps. The methods experimented with not only help introduce the application of deep learning techniques in vegetation mapping but also introduce the use of alternative, publicly available lidar data to improve the efficiency of traditional machine learning algorithms in vegetation and land cover classification.

Three experiments were conducted, testing different spectral bands in combination with two canopy height models extracted from UAS lidar observations and the combination of structures from the motion digital surface model and existing digital elevation model. Our findings showed that the U-net network yielded the most consistently accurate classification results, followed in order by the DeepLabV3, RF, and SVM. While deep learning proved to have great potential in the field of vegetation mapping, as was demonstrated by the overall accuracy outcomes achieved in our study, traditional machine learning algorithms, especially RF, which produced overall accuracy that is inferior but still reasonable compared to the deep learning models in a very short time, can be a quick alternative.

Our study reached the conclusion that including the red edge and near-infrared bands in addition to the three visible bands (Green–Blue–Red) produced results that were statistically similar to the results obtained using visible bands only when the classification was carried out using the deep learning techniques. This conclusion did not apply to the RF and SVM classification results. Regarding CHM, the results obtained in this study suggest that canopy height models derived from SfM photogrammetric models and low-resolution DTM (from airborne lidar datasets) are a suitable option to add geometrical information to the machine learning classification models. The study found that even when the accuracy increased using the deep learning techniques compared to traditional machine learning algorithms, the deep learning techniques did not seem to use the full potential of the multispectral and CHM data, suggesting a need for improved deep learning algorithms.