Automatic Marine Debris Inspection

Abstract

Plastic trash can be found anywhere, around the marina, beaches, and coastal areas in recent times. This study proposes a trash dataset called HAIDA and a trash detector that uses a YOLOv4-based object detection algorithm to monitor coastal trash pollution efficiently. Model selection, model evaluation, and hyperparameter tuning were applied to obtain the best model for the lowest generalization error in the real world. Comparison of the state-of-the-art object detectors based on YOLOv3, YOLOv4, and Scaled-YOLOv4 that used hyperparameter tuning, the three-way holdout method, and k-fold cross-validation have been presented. An unmanned aerial vehicle (UAV) was also employed to detect trash in coastal areas using the proposed method. The performance on image classification was satisfactory.

1. Introduction

Close to 12.7 million tons of plastic waste enters the sea every year and poses a major threat to the ecology of the earth, according to a Greenpeace report [1]. Many countries have invested funds to deal with trash issues to solve this problem of marine debris. For instance, in 2021, the Taiwan government allocated NTD 1.6 billion in funds and implemented the “Coastal Cleaning and Maintenance Plan” [2]. Eighty-two percent of the funding is aimed at cleaning up marine debris, and 11% is for reducing marine debris. If illegal trash dumping can be monitored and the trash can be prevented from entering the sea, the area of plastic pollution can be significantly reduced. This study is aimed at building an automatic marine trash monitoring system for government agencies to reduce human labor in debris inspection. The most important part of the monitoring system is the trash detector. The efficient trash recognition method is the key issue of this detector. In this paper, we present different image processing algorithms employing different data evaluation and validation methods. For automation issues, an unmanned aerial vehicle (UAV) was applied to perform the inspection task. The UAV is suitable for marine trash monitoring systems as a scout vehicle due to its high mobility and ability to patrol the coastline efficiently.

In recent years, many pattern classification methods have been proposed. Prior to 2012, support vector machine (SVM) [3], k-nearest neighbors (KNN) [4], and back-propagation neural networks (BPNN) [5] were well-known algorithms for object classification. In 2012, AlexNet [6], a deep convolutional neural network (CNN), won the ImageNet LSVRC-2010 prize in the image classification competition. In a research study, the authors compared the performance of the KNN, SVM, BPNN, and CNN for handwriting digit recognition and reported that the CNN had the best recognition rate at 97.7% [7]. In another paper, the authors reviewed the techniques used for medical diagnoses, such as KNN, SVM, random forest, and CNN, and found that CNN had the best accuracy [8]. Since CNN has better object identification capability, many researchers have applied it to image classification. However, in this study, such object classifiers are not good enough in our UAV trash monitoring system because a trash detector that can indicate the positions of the object and the area of the trash from aerial images is required. Researchers have proposed the OverFeat using CNN to integrate classification, localization, and detection [9]. A region convolution neural network (R-CNN) that outperforms the OverFeat was proposed, but it was not for real-time object detectors [10]. In another study, the authors proposed the You Only Look Once network (YOLO), which was a new approach to object detection and possessed a fast-enough processing speed [11]. The precision and the speed of object detection algorithms have been improved with research, such as YOLOv2 [12], YOLOv3 [13], and YOLOv4 [14]. In 2021, Scaled-YOLOv4 [15] obtained the first rank in the real-time object detection benchmark on the COCO dataset. Thus, considering the precision and the image processing speed, the YOLO object detection algorithm was chosen as the trash detector in this work.

YOLO uses the $\mathrm{S}\times \mathrm{S}$ grid cells to detect objects in the input image. Each grid cell predicts the object if the center of the object is in the grid cell. YOLO has 24 convolutional layers and 2 fully connected layers. The drawback of YOLO is that because each grid cell predicts two bounding boxes with one class, it can detect only $\mathrm{S}\times \mathrm{S}$ objects. Thus, YOLOv2 uses five anchor boxes that are calculated by the k-means cluster via Intersection over Union (IOU) and not the Euclidean distance. This facilitates the detection of multiple objects in a grid cell. YOLOv2 has a 5% mean average precision (mAP) improvement with the anchor-based prediction. The backbone of YOLOv2 is Darknet-19 with 19 convolutional layers and 5 max pooling layers. In YOLOv2, every convolutional layer is followed by batch normalization [16] to accelerate the training process and obtain a 2.4% improvement in mAP, making it better than YOLO. In YOLOv3, the Darknet-19 backbone is replaced by Darknet-53. Darknet-53 achieves 93.8% accuracy in the top 5 of the ImageNet classification, which is the same as the state-of-the-art (SOTA) ResNet-152 [17] albeit at a fast processing speed. The vanishing gradient problem usually happens during the training process of deep convolutional neural networks. In ResNet, the authors used the residual block for the shortcut connection to prevent this vanishing gradient problem. A feature pyramid network (FPN) has been proposed and applied to the SOTA algorithms that showed a significant improvement [18]. In FPN, the CNN was feed-forwarded to obtain the bottom–up pathway and up-sample the top feature and later merged (added) with the bottom feature to obtain the top–down pathway and the lateral connections. For the neck, YOLOv3 predicts three scales of feature maps and uses the modified FPN, which applies concatenation to merge the multiple features instead of the adding operation. With this approach, compared with YOLOv2, the detection ability of small objects by YOLOv3 is increased.

Each positive sample is assigned by one anchor with the highest IOU in the training process. The ignored samples have a bigger IOU than the threshold (positive samples are not included). The negative samples have IOU smaller than the threshold and not the positive ones. If the sample is positive, the confidence score and the class probability are 1. If the sample is ignored, then the loss is ignored. If the sample is negative, the confidence score is 0. While YOLOv1 and YOLOv2 use the mean square error as the loss function (bounding boxes, class probability, and confidence score), YOLOv3 uses binary cross entropy (class probability and confidence score) for this purpose. In YOLOv4, the authors combined the SOTA method to obtain better performance. A cross-stage partial network (CSPNet) has been proposed [19] that uses a cross-stage feature fusion strategy to improve both the accuracy and the speed for ResNet, ResNeXt [20], and DenseNet [21]. In CSPNet, the authors divided the input feature maps of the dense block into two parts. One part was used for feature extraction, and the other part was used as the input of the transition layer. In YOLOv4, the authors proposed CSPDarknet53 and CSPResNext50, where CSPDarknet53 was more effective and had a more receptive field size than CSPResNext50. The developers of YOLOv4 noticed that CSPResNext50 was better in the classification task (top 5: 95.2%) than the CSPDarknet53 (top 5: 94.8%). However, in the object detection task, CSPDarknet53 had better performance. YOLOv4 uses a path aggregation network (PANet) [22] and spatial pyramid pooling (SPP) [23] as the neck instead of FPN [16]. In SPP, researchers used pooling layers to obtain the multiple scales of feature maps. In YOLOv4, SPP is modified to use $1\times 1$, $5\times 5$, and $13\times 13$ max pooling layers to extract the features and concatenate them to obtain the deeper depth features. PANet is an FPN-based network that performs the top–down pathway after the FPN. YOLOv4 uses concatenation as the features’ fusion operation instead of addition.

In deep learning neural networks, selecting the proper activation function is also important. A mish activation function with better image classification over Swish and ReLU has been proposed [24]. This activation function was applied to different architectures and was tested by the CIFAR10 dataset. Mish activation is smooth and nonmonotonic. The nonmonotonic property can have a negative gradient that improves the expressivity and the gradient flow. In CSPDarknet53, researchers used the mish activation to improve the performance of the classifier. Another issue that needed attention was the overfitting problem. Dropout is a method for preventing the fully connected layers from overfitting. In a CNN, the dropout method is ineffective due to the spatial correlation in convolutional layers. In DropBlock [25], the authors proposed a structured dropout method, which drops continuous regions to remove some semantic information. In the COCO dataset, RetinaNet obtained a 1.6% improvement in mAP with DropBlock. YOLOv4 uses the DropBlock method in the training process. Mixup [26], Cutout [27], and CutMix [28] are examples of recent data augmentation and regularization strategies to train a robust classifier. In Mixup, two samples are fused as one sample, which improves the accuracy of the ImageNet classifier by 1.1%. In Cutout, the researchers masked out the square regions to the sample, which is better than the Mixup method for ImageNet localization. In CutMix, the authors combined Mixup and Cutout methods to obtain a significant improvement for the ImageNet classification, ImageNet localization, and VOC 2007 detection tasks. In YOLOv4, a Mosaic data augmentation method was put forth that combines four images into one image to increase the minibatch in the training process. Mosaic was found to be better than CutMix in CSPResNext50.

In YOLOv4, it was discovered that the center of the object was usually detected at the edge of the grid cells in the real world. As it is not easy to achieve the limit of the sigmoid function, the sigmoid function can be multiplied with a factor that should be bigger than 1 to eliminate the grid sensitivity. The IOU loss and the generalized IOU (GIOU) [29] loss have been proposed to improve the accuracy of the bounding box regression. In distance-IOU (DIOU), the authors summarized the three important factors (aspect ratio, central point distance, and overlap area) in the bounding box regression [30]. They used complete IOU (CIOU) loss to improve the performance of YOLOv3 in the PASCAL VOC dataset. Additionally, the authors proposed DIOU-NMS to improve crowd detections (DIOU-NMS obtained five objects, but NMS obtained only four objects). The CIOU loss with DIOU-NMS obtained a 5.91% (baseline: IOU loss) improvement in average precision (AP). The CIOU loss was better than the GIOU loss in bounding box regression. In 2020, YOLOv4-Tiny [15] was proposed, which achieved the best performance in SOTA tiny models [14]. The backbone of YOLOv4-Tiny is CSPOSANet, which is based on CSPNet and VoVNet [31]. In VoVNet, the authors proposed a one-shot aggregation (OSA) module that concatenated all features only once in the last feature map to resolve the heavy memory cost (MAC) and the inefficiency of the computation of DenseNet [21]. Max pool layers follow three CSP blocks in CSPOSANet. The FPN structure is used in YOLOv4-Tiny-3l to fuse the feature maps and obtain more semantic information such as YOLOv3. YOLOv4-Tiny-3l uses three YOLO heads to predict objects across scales such as YOLOv4 but does not use multiple anchors to predict per object. YOLOv4-Tiny-3l uses one anchor to fit one object. In this study, YOLOv4-Tiny was used as the object detector.

To monitor trash pollution efficiently, a UAV was employed to patrol the beach [32], which used aerial images with SegNet [33], a deep neural network, for semantic segmentation to classify beach litter in pixel units. Additionally, a UAV was used in a long-term monitoring program to study the spatial and temporal accumulation of the dynamics of beached marine litter [34]. Researchers have used three supervised classification algorithms—maximum likelihood, random forest, and support vector machine—to classify marine litter [35]. The attention layer in the neuron network for improving the performance of trash detection has also been proposed [36]. Researchers have built a TACO trash dataset [37] and adopted Mask R-CNN [38] to test the performance of litter detection. An improved YOLOv2 [11] model was proposed, and an automatic garbage detection system was also built [39]. Further, a research team modified the loss function in YOLOv3 and created an automated floating trash monitoring system based on UAVs [40]. Although these methods have a good performance on trash classification, none of them are applicable for real-time debris analysis in coastal areas. These zones require a fast detection rate, object classification, and object position information. In this work, we developed a trash dataset called HAIDA and compared the state-of-the-art object detectors based on YOLOv3, YOLOv4, YOLOv4-Tiny with hyperparameter tuning, a three-way holdout method [37], and k-fold cross-validation [41]. Subsequent model evaluation and selection were performed to obtain the best model for the lowest generalization error in the real world.

2. Methods

In order to implement object detection applications in the real world, the trash recognition algorithm should be highly accurate and fast enough to process data in real time. Thus, the YOLO object detector was employed in the NVIDIA embedded systems, namely, the TX2 and Xavier NX models, for speed/accuracy tradeoff. Due to the lack of coastal trash image data, we expanded some released trash datasets, such as TACO, into the training set to conduct the data mining research outlined in Section 3. However, it was observed that the released trash datasets did not improve the accuracy of the Taiwan coastline trash detector. Hence, we developed a trash dataset called HAIDA and compared it with the state-of-the-art object detectors with hyperparameter tuning, the three-way holdout method [15], and k-fold cross-validation [16]. The model evaluation and selection were used to obtain the best model for the lowest generalization error in the real world. To better distinguish the marine debris appearance, we defined marine debris as two classes: “bottle” and “garbage.” The appearance of the “bottle” is a fixed shape, and the “garbage” is broken plastic pieces and other items with no fixed shape.

• Dataset

We created a trash dataset of 1319 aerial images called HAIDA, which includes two classes (3904 garbage objects and 2517 bottle objects) and 456 negative samples (without garbage and bottle objects) that were collected by UAVs [42]. The challenge to the coastal trash detection task is that such trash has no geometrical shape, and it is difficult for people to define and label the dataset, as shown in Figure 1. The ground sample distance (GSD) is 0.7 cm/pixel when the flight height of the UAV is 10 m.

In the HAIDA trash dataset, the images of trash in Taiwan were collected by UAVs. The bottle is the first class of the HAIDA dataset. In Figure 2a, such bottles were captured by a UAV from heights of 3 and 10 m on the campus of the National Taiwan Ocean University (NTOU). Garbage is the second class in the HAIDA trash dataset. We consider this garbage as plastic trash with no clear shape definition (unlike bottles). In Figure 2b, the bottles were captured by a UAV from heights of 3 and 5 m on the coastline of the NTOU. In Figure 2c–h, the garbage and the bottles were captured at 3 to 30 m heights in the Badouzi fishing port. In Figure 2i, the garbage and the bottles were captured by a UAV at 6 and 15 m heights along the Gongliao beach coastline. Figure 3 shows the negative samples in the HAIDA trash dataset.

B.

Object Detectors

Scaled-YOLOv4 is currently the state-of-the-art object detector based on CSPNet and YOLOv4. A comparison of average precision (AP) and latency for the state-of-the-art object detectors is shown in Figure 4.

The backbone of YOLOv4-Tiny is CSPOSANet, which is inspired by CSPNet and VoVNet [31]. In VoVNet, the authors proposed a one-shot aggregation (OSA) module that concatenated all features only once in the last feature map to resolve the heavy memory cost (MAC) and the inefficiency of the computation of DenseNet [21], as shown in Figure 5.

There are three CSP blocks followed by max pool layers in CSPOSANet, as shown in Figure 6. The FPN structure is used in YOLOv4-Tiny-3l to integrate the feature maps and obtain more semantic information like with YOLOv3. YOLOv4-Tiny-3l uses three YOLO heads to predict objects across scales like YOLOv4 but does not use multiple anchors to predict per object; YOLOv4-Tiny-3l uses one anchor to fit one object. YOLOv4-Tiny [15] achieves the best performance in SOTA tiny models, as shown in

Table 1. The performance of SOTA tiny models in the COCO dataset, C.Y. Wang 2020.

Model	Size	FPS1080ti	FPSTX2	AP
YOLOv4-tiny	416	371	42	21.7%
YOLOv4-tiny(31)	320	252	41	28.7%
ThunderS146	320	248	-	23.6%
CSPPeleeRef	320	205	41	23.5%
YOLOv3-tiny	416	368	37	16.6%

.

C.

Holdout Method and Hyperparameter Tuning

The three-way holdout method is a simple way to select the best model for a trash detector. The dataset is split into three parts: a training set for model fitting, a validation set for both hyperparameter tuning and model selection, and a testing set for model evaluation, as shown in Figure 7.

In this study, the dataset was split for the holdout method as 70/20/10 into three parts. With hyperparameter tuning, when the input size was up to $864\times 864$, the YOLOv4-Tiny-3l model achieved 71.46% AP₅₀ (Val) on the validation set and obtained an inference speed of 22 frames per second (fps) on the NVIDIA embedded system Xavier NX. This result is better than that of YOLOv4 with 1.17% AP₅₀ (Val) and (416) 12 fps. All models were trained on NVIDIA GeForce RTX-2080Ti GPU.

Table 2. Comparison of state-of-the-art object detectors on the HAIDA trash dataset.

Method	Size	FPS 2080Ti	FPS NX	BFLOPs	AP30 (Val)	AP50 (Val)	AP75 (Val)	AP50 (Test)	AP50 (Train)
YOLOv4-Tiny-3l	416	470	72	8.022	75.76%	67.42%	23.53%	71.65%	92.48%
YOLOv3-Tiny	864	231	28	23.506	-	64.46%	-	68.50%	91.22%
YOLOv4-Tiny	864	204	25	29.284	-	69.75%	-	72.48%	91.80%
YOLOv3-Tiny-3l	864	194	24	30.639	-	66.15%	-	69.85%	91.97%
YOLOv4-Tiny-3l	864	177	22	34.602	78.17%	71.46%	28.27%	73.37%	97.41%
YOLOv3	416	118	12	65.312	-	66.27%	-	69.24%	94.61%
YOLOv3-SPP	416	116	12	65.69	-	64.66%	-	66.86%	90.88%
YOLOv4	416	92	10	59.57	78.36%	70.29%	29.15%	74.20%	90.65%
YOLOv4-CSP	512	88	9	76.144	-	58.12%	-	61.76%	58.08%
YOLOv4	512	80	7	90.237	79.83%	72%	27.49%	74.41%	93.43%
YOLOv4	608	63	5	127.248	80.04%	72.20%	28.81%	72.70%	94.28%
YOLOv4x-MISH	640	41	3	220.309	74.07%	66.10%	26.49%	71.35%	96.06%

shows the performance of SOTA object detectors on the HAIDA trash dataset.

Table 3. Hyperparameter sets for the YOLOv4-Tiny-3l model tuning.

Hyperparameter.	Size	Batch	Iteration	lr	Steps	Scales	Multi-Scale	Act.	Mosaic	SAT	IOU	NMS	Pretrain	AP50 (Val)
1	864	16	10,000	0.00261	4500; 7000	0.1; 0.1	-	leaky	-	-	ciou	greedy	imageNet	69.71%
2	864	16	10,000	0.00261	8000; 9000	0.1; 0.1	-	leaky	-	-	ciou	greedy	imageNet	69.33%
3	864	16	10,000	0.00261	8000; 9000	0.1; 0.1	-	leaky	yes	-	ciou	greedy	imageNet	70.69%
4	864	16	10,000	0.00261	8000; 9000	0.1; 0.1	-	leaky	yes	-	ciou	greedy	-	68.07%
5	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	yes	leaky	yes	-	ciou	greedy	imageNet	71.15%
6	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	yes	leaky	-	-	ciou	greedy	imageNet	71.46%
7	864	16	10,000	0.00261	8000; 9000	0.1; 0.1	-	leaky	yes	-	ciou	greedy	imageNet	70.32%
8	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	-	leaky	yes	-	ciou	greedy	imageNet	69.55%
9	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	yes	leaky	-	-	ciou	diou	imageNet	71.16%
10	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	yes	leaky	-	yes	ciou	greedy	imageNet	70.73%
11	864	6	10,000	0.00261	8000; 9000	0.1; 0.1	yes	mish	-	-	ciou	greedy	imageNet	70.58%
12	864	8	10,000	0.002	8000; 9000	0.1; 0.1	yes	leaky	-	-	ciou	greedy	imageNet	71.18%
13	864	8	10,000	0.00261	8000; 9000	0.1; 0.1	yes	leaky	-	-	ciou	greedy	TACO	70.38%
14	864	8	10,000	0.004	4500; 6000	0.1; 0.1	-	leaky	-	-	ciou	greedy	imageNet	71.25%

shows the hyperparameter tuning results for the YOLOv4-Tiny-3l model.

For the holdout method, considering the speed/accuracy tradeoff to UAV applications, we chose YOLOv4-Tiny-3l (864) as the best model over YOLOv4 (416), YOLOv4 (512), and YOLOv4 (608). YOLOv4-Tiny-3l (864) was better than YOLOv4 (416) at 6.76% AP₅₀ on the HAIDA training set and could learn 97.41% of the dataset after training, as shown in Figure 8. The blue line is the loss of the training set, and the red line is the AP₅₀ of the validation set. YOLOv4 (416) overfitted the training data after 3500 iterations, and the loss converged at 9000 iterations but obtained low AP₅₀ (Val), as shown in Figure 9. To prevent overfitting of YOLOv4 (416), we used the model that obtained the best AP₅₀ (Val) at 3500 iterations.

For the hyperparameter tuning of the YOLOv4-Tiny-3l model (864), we tried 13 factors to tune the model. These included the input size, batch size, iteration, learning rate, learning rate scheduler (multistep decay), multiscale training, Mosaic data augmentation, activation functions, self-adversarial training (SAT), IoU loss, NMS, and pretrained weight. The hardware of VRAM of RTX 2080Ti GPU limited the input and batch sizes. The learning rates of hyperparameter sets 6 and 12 were 0.00261 and 0.002, and the performances were 71.46% and 71.18%, respectively, as shown in Table 3. The multiscale training improved the 0.83% AP₅₀ (Val). The steps and scales were parameters of the learning rate scheduler (multistep decay). The performances of hyperparameter set-1 and set-2 were 69.71% and 69.33%, respectively. The Leaky ReLU activation was better than the Mish by 0.88%. With Mosaic, SAT, and DIoU-NMS, these performances decreased by 0.31%, 0.73%, and 0.3%, respectively. The pretrained weights of ImageNet improved the AP₅₀ (Val) by 2.25% and were better than the TACO [42] pretrained weights. In our experiments, the best hyperparameter set of YOLOv4-Tiny-3l (864) for the HAIDA trash dataset is shown in Table 3.

D.

K-Fold Cross-Validation

The result of the holdout method is sensitive to how we split the data. If there is a large bias in the dataset, the result of the holdout method will be affected, especially for small datasets. In Table 2, the AP₅₀ (Val) and AP₅₀ (Test) in YOLOv4 (416) are very different due to the bias in the HAIDA dataset. To obtain a better result for model evaluation, k-fold cross-validation [41] was used. First, the data were split into k parts, one part for the validation set, the remaining k₁ parts for the training set, and repeated k times, as shown in Figure 10. Here, we chose k = 5 for our customized trash dataset. The YOLOv4 (416) model obtained the best average AP₅₀ (test fold) 71.77% across five folds, and we could see the bias occur especially for YOLOv4-Tiny-3l (864), as shown in

Table 4. Comparison of three models for five-fold cross-validation.

Method	Size	AP50 (Test Fold)
		5–1	5–2	5–3	5–4	5–5	Avg.
YOLOv4-Tiny-3l	864	73.61%	71.28%	68.89%	73.72%	69.76%	71.45%
YOLOv4	416	72.56%	72.48%	70.80%	72.45%	70.58%	71.77%
YOLOv4	512	72.63%	71.31%	70.18%	73.58%	70.67%	71.67%

. The average AP50’s of YOLOv4-Tiny-3l (864) and YOLOv4 (416) were 71.45% and 71.77%, respectively. Considering the fast inference speed, the YOLOv4-Tiny-3l model was chosen as the trash detector.

E.

Nested Cross-Validation

Nested cross-validation [43] is a better method for simultaneous model evaluation and hyperparameter tuning. In our previous work, we used the holdout method to tune the hyperparameters of YOLOv4-Tiny-3l and YOLOv4 and discovered a bias in the HAIDA trash dataset. Thus, we used the hyperparameters’ tuned models to conduct model evaluations. However, a problem was that the hyperparameters are tuned based on the holdout method rather than the k-fold approach. Thus, we felt that nested cross-validation should be used to conduct more accurate model evaluations.

Nested cross-validation includes the outer and inner loop; the outer loop and inner loop are divided into k₁ and k₂ folds like the k-fold cross-validation. First, the inner loop is used to tune the hyperparameters, and the best one is chosen. Subsequently, these hyperparameters are applied to the outer loop for the model evaluation. This is repeated k₁ times, and the errors are averaged, as shown in Figure 11 (k₁ = 5 and k₂ = 2).

Although the nested cross-validation is very accurate, it is time-consuming, especially for CNN. The cost of nested cross-validation is n $\times$ k₁$\times$ k₂ $\times$ m, where n is the number of models, k₁ and k₂ are the numbers of folds in the outer and inner loops, and m is the number of hyperparameter sets to compare in each inner loop. Nested cross-validation is used in linear regression, logistic regression, small neural networks, and support vector machines. In CNN, k-fold cross-validation is used for the small and medium datasets. The holdout method is commonly used for ImageNet and other large datasets.

3. Experimental Results

The YOLOv4-Tiny-3l model was implemented and utilized in the NVIDIA embedded system Xavier NX on the quadrotor. The total weight of the quadrotor was 5.05 kg with a flight duration of 21.8 min at an average speed of 2 (m/s) for patrolling the coastline. The onboard camera was at a 1080p resolution with a 30 fps video streaming ability, and the horizontal and vertical views were 65 and 70 degrees, respectively. Three areas (NTOU campus, Badouzi fishing port, and Wanghai Xiang beach) were tested, and the results are shown below.

• The NTOU Campus

First, we tested the regular shape trash (bottles) to see the outcomes in small object detection. The confidence scores of small object detections are high (98.27%, 64.65%, 91.86%), which means that the HAIDA dataset has a small bias to a real-world test on the NTOU campus, as shown in Figure 12 [44].

B.

Badouzi Fishing Port

In the Badouzi fishing port, the shape and the texture of the garbage varied due to natural erosion. This decreased the confidence scores of garbage detection. This means that the HAIDA dataset had a large bias with respect to the Badouzi fishing port. The best solution to solve this problem was to collect more data. The YOLOv4-Tiny-3l model was trained from the HAIDA dataset, and the UAV flew at a height of 10 m, as shown in Figure 13a. Figure 13b–d shows the detection of large, medium, and small objects.

C.

Wanghai Xiang Beach

There was more plastic debris on the Wanghai Xiang beach, which easily confused both human detection and the trash detector to result in false-positive (FP) detection, as shown in Figure 14. The YOLOv4-Tiny-3l model was trained from the HAIDA dataset, and the UAV flew at a height of 5 m.

D.

Data Mining

The performance of the model is dependent on the data. We had a TACO trash dataset, which was collected by anonymous providers from different places. We also downloaded some trash data from the internet (crawler trash dataset). Further, we collected some aerial images from our UAV (simple aerial trash dataset). Due to the disorganized data content of beach trash, human error was highly probable in trash labeling. This led us to explore the impact of the dataset on the target detection task from the initial aerial shot dataset. For data mining, we used the holdout method to divide the dataset into a training set and a validation set (70% and 30%, respectively). YOLOv4 was trained for data mining and tested with the HAIDA trash dataset. After training, YOLOv4 obtained 99% mean average precision (mAP) on the simple aerial trash validation set, as shown in Figure 15. We used the TACO dataset (for segmentation) and relabeled it for object detection. After training, YOLOv4 obtained 83% mAP on the TACO validation, set as shown in Figure 16. We also used the dataset from the internet (crawler trash dataset) and labeled it for object detection. After training, YOLOv4 obtained 58% mAP on the crawler trash validation, set as shown in Figure 17.

We needed the model to predict the diverse trash types in multiple scales with good precision. The TACO and crawler datasets were not suitable for our UAV trash detector. We added some new aerial trash images collected by the UAV to the HAIDA dataset. We called it the HAIDA-2 trash dataset. There were 1880 aerial images with two classes (6179 garbage objects and 2563 bottle objects) and 556 negative samples. Figure 18 shows the HAIDA-2 trash dataset that was detected by YOLOv4-Tiny-3l (864).

After all trash was detected, the images of marked trash, the location of the trash, and the area of trash were then computed via the onboard computer of the UAV. This information was subsequently transmitted to the remote site computer through the Kafka connector and Mongo database [43]. A trash heatmap could be displayed on the monitor screen of the remote site computer for usage by an environmental protection agency, as shown in Figure 19 [43].

The real-time trash detection results were sent to the ground monitoring station, as shown in Figure 19a. The red flags are the target waypoints in the UAV monitoring mission. For the trash information, the blue markers indicate where the trash is detected, and the circle area presents the severity (color based on the trash polluted area table, as shown on the bottom left of Figure 19a) of the trash pollution area. The purple circles are the flight trajectory of the UAV, and the UAV status dashboard is shown on the right-hand side of Figure 19a. The trash information is saved in the database to query the data for the heatmap analysis for potential use by an environmental protection agency, as shown in Figure 19b. The heatmap also points out the severity of the trash pollution in the Badouzi fishing port. In the future, the overall heatmap analysis can potentially cover all coastal areas of Taiwan.

4. Conclusions

The holdout method and hyperparameter tuning were used to find the best object detector for our HAIDA trash dataset. The YOLOv4-tiny-3l trash detector was better than the YOLOv4 model for the HAIDA dataset when the input size was up to 864 $\times$ 864. We also employed the trash detector in an NVIDIA embedded system, NX, on the UAV to test the trash detector in three areas. The performances were good in the NTOU campus and the Badouzi fishing port scenarios. However, in the Wanghai Xiang beach scenario, there were some false-positive (FP) detections due to confusing debris. In this study, we implement the application of a UAV for marine trash detection with our previously published UAV trash monitoring system. This can contribute to global environmental protection. In the real-world test, the confidence scores of the trash detection at the Wanghai Xiang beach and the Badouzi fishing port were low (5% to 50%). This could be due to the trash detector using its generalization to predict the eroded trash, which has varying shapes. In future work, we will increase the confidence scores of the trash detector and filter out the low confidence scores with NMS and avoid FP detection by collecting more coastal trash data for detector training. In addition, aerial image quality is another key point in efficient trash detection. As the image was disturbed by the vibration from the UAV, this point also needs to be addressed in future studies.