Targeting Plastics: Machine Learning Applied to Litter Detection in Aerial Multispectral Images

Abstract

The occurrence of litter in natural areas is nowadays one of the major environmental challenges. The uncontrolled dumping of solid waste in nature not only threatens wildlife on land and in water, but also constitutes a serious threat to human health. The detection and monitoring of areas affected by litter pollution is thus of utmost importance, as it allows for the cleaning of these areas and guides public authorities in defining mitigation measures. Among the methods used to spot littered areas, aerial surveillance stands out as a valuable alternative as it allows for the detection of relatively small such regions while covering a relatively large area in a short timeframe. In this study, remotely piloted aircraft systems equipped with multispectral cameras are deployed over littered areas with the ultimate goal of obtaining classification maps based on spectral characteristics. Our approach employs classification algorithms based on random forest approaches in order to distinguish between four classes of natural land cover types and five litter classes. The obtained results show that the detection of various litter types is feasible in the proposed scenario and the employed machine learning algorithms achieve accuracies superior to 85% for all classes in test data. The study further explores sources of errors, the effect of spatial resolution on the retrieved maps and the applicability of the designed algorithm to floating litter detection.

1. Introduction

Littered areas can be found all over the globe, directly damaging the ecological balance, threatening endangered species and affecting the health of the human population. Among all the waste generated by the world population, the plastic materials are of utmost concern in terms of both quantity and environmental threats. In [1], it has been estimated that in 2016, 242 million tons of plastic waste were generated at global level, accounting for 12% of the total municipal solid waste. In the same study it is shown that the limited capacities for collecting and recycling these residuals in many parts of the world, coupled with educational and economical challenges, call for improved policies as the waste generation is projected to outpace the population growth by more than double by 2050.

The plastic residuals pose a serious threat to natural ecosystems and human population. Although the plastics are designed to be resistant for long periods of time, it is not uncommon that they are used in single use products, such as grocery packages. The most worrisome characteristic of these materials is that they can last for hundreds or even thousands of years [2], and can break into pieces of smaller size until they spread in the form of nano-plastics which are extremely harmful to humans as they are far more toxicologically active than micro- and macro- plastics. The nano-plastics have the potential to directly interact with human cells due to their size and to produce severe reactions ranging from physical stress and damage to immune responses [3]. Some chemical compounds found in plastic, such as bisphenol A and phthalates, are part of a group of substances called endocrine-disrupting chemicals that can trigger metabolic, reproductive and degenerative disease and certain types of cancer [4]. The majority of nano-plastic particles enter the human body by ingestion and inhalation, thus they are mobile, carried by the wind, water or products, and cannot be treated as localized issues occurring at the dumping sites.

At a larger scale, micro- and macro-plastic waste is also affecting the environment once it enters natural circuits. Plastics dumped on land are often carried by river streams and end up in lakes, seas and oceans [5]. The mobility of plastic residuals is so high that micro-plastics have even been found in wild or rarely explored places on earth, e.g., polar regions [6], Mariana Trench [7] and Mount Everest [8]. Numerous reports document the direct, harmful effect of plastics on wildlife [9,10,11,12,13].

Following these alarming reports, a variety of mitigating measures have been proposed by legal regulators across the world, including the banning of single-use plastics, reinforcement of educational programs, raising citizen awareness and improved recycling procedures, among others [14,15,16,17,18,19,20,21].

To inform effective mitigation strategies, urgent monitoring action is needed to better understand sources, pathways, and the spatial and temporal distribution of plastic litter. It is also important to precisely locate plastic litter discarded in nature in a view to remove it as soon as possible, such that the course of pollution is interrupted before it reaches an uncontrollable stage—distribution in nature under the action of natural elements and breaking into small pieces, which is virtually impossible to spot and collect. Among the detection and monitoring techniques, the use of remote sensing imagery stands out as an excellent option due to the coverage of relatively large areas in a short period of time. The methods applied to achieve a high accuracy of plastic detection based on optical sensors cover a wide range of methods and data types. In [22], semantic segmentation is employed to detect beach litter in images freely taken by a human observer. Beach litter monitoring based on data acquired from unmanned aerial vehicles has also been studied in [23,24,25]. Deep learning methods have also been employed in aerial imagery, in combination with digital models of the surface, to estimate the volume of litter in such environments [26]. White and grey dunes, where vegetation with varying levels of abundance exists, are also affected by litter, which is often entangled with the green plants, as it is the case in beach-dune systems. This type of ecosystem has been also studied in recent works based on aerial imagery [27,28,29]. Aiming at quantifying the abundance of litter in aerial images, the authors in [30] employ various object-oriented machine learning methods and conclude that a Random Forest approach performs better than other tested approaches. A vast amount of research has also been devoted to monitoring plastic-covered crops (greenhouses) [31,32,33,34,35,36]. The remote sensing detection and monitoring of floating marine debris is now a very active field of research with several funding agencies promoting efforts in this research area (see [37,38,39] and the research initiatives therein). When targeting a global overview of debris pollution over water areas, satellite images are exploited in a wide range of methods, mostly targeting spectral classification approaches [40,41,42] for which novel indices have been developed to be used in conjunction with already known metrics and the reflectance values acquired at the respective pixel [43,44,45,46,47].

Despite the intense efforts to advance knowledge in the area of litter detection and classification, a series of challenges remain. First, the detection from satellite imagery is hampered by the relatively low spatial resolution of the sensors, while the signal-to-noise ratio (SNR) of operational sensors might also be of concern for this type of application, considering the limited coverage of plastic per pixel occurring in most of the cases [48]. The lack of training samples has also been signaled as a limitation, and efforts are being made for the production of such data [49,50]. The object-based detection methods retrieve possible areas of plastic contamination, however multiple technologies (from optics to microwaves), e.g., Synthetic Aperture Radar (SAR) [51], might be needed to perform reliable discrimination of accumulations containing plastic. Furthermore, many methods developed for plastic litter monitoring at a large or global scale make use of features computed from both the visible and near-infrared (VIS-NIR) and the short-wave infrared (SWIR) regions of the electromagnetic spectrum, while many airborne multispectral sensors do not have such extended spectral ranges, which makes these methods difficult to apply when regional/local studies are performed, e.g., with planes and drones. These studies are needed as the data can be acquired at higher spatial resolution, thus offering a more detailed monitoring of the pollution and also a chance to remove the local litter, e.g., from beaches and river-banks. While it is possible to employ airborne hyperspectral sensors covering the full electromagnetic spectrum, the data acquisition campaigns are more costly, and might involve additional legal procedures such as obtaining flight permits from civil and military aviation when piloted aircrafts are used, and the data processing is more resource-intensive in comparison to multispectral imagery. This is specifically the research problem tackled in this study: estimating the capabilities of off-the-shelf multispectral cameras having sensors with limited spectral range, in a two-fold goal. First, the main scope is to obtain a high performance related to plastic detection and estimation. Second, the study explores further the potential and limitations to discriminate between other types of litter.

The data used in this paper are multispectral images acquired by a MicaSense Red-Edge (https://support.micasense.com/hc/en-us, accessed on 15 November 2022) camera. We employ machine learning algorithms based on Random Forest (RF) approaches to train a fully supervised multi-class classifier for a total of nine classes, out of which four are natural materials and five correspond to litter materials. The algorithms perform per-pixel classification based on optimal features derived from a large initial pool of spectral metrics (indices). Training and validation datasets were acquired in two different data acquisition campaigns. Our experiments show that plastic litter can be found with high accuracy based on this approach (>88% in test and validation data). The study further shows that class confusions between litter types occur due to data errors and spectral similarities, highlighting shadows as a strong source of inaccuracy in the classification maps and exploring post-processing techniques to alleviate these limitations. Finally, we analyze the influence of spatial resolution on the quality of the retrieved maps and the limitations of the designed algorithm when applied over water areas.

2. Materials and Methods

2.1. Available Data

In this study, multispectral images acquired with a multispectral MicaSense RedEdge-M camera are employed. The sensor benefits from five spectral bands centered at 475 nm (blue—B), 560 nm (green—G), 668 nm (red—R), 717 nm (red-edge—RE) and 840 nm (near-infrared—NIR) and having bandwidths (FWHM) of 20 nm, 20 nm, 10 nm, 10 nm and 40 nm, respectively. The data are processed using VITO’s MAPEO water workflow, operational in Amazon Web Services (AWS). The application of the calibration parameters of the camera transforms the digital numbers into radiance. To calculate reflectance out of radiance, the irradiance should be taken into account. This can be done using a calibrated reflectance panel or an irradiance sensor. Under varying cloud conditions, the latter could better take into account the changing light conditions. For measurements above water, an additional sky glint correction is applied using iCOR4drones, based on VITO’s atmospheric correction toolbox iCOR. Pixels that are saturated are masked out of the end product.

Two data acquisition campaigns have been performed in two different days and at two different sites in the vicinity of Mol, Belgium.

The first campaign (Campaign1) took place on 1 June 2021, with the MicaSense camera mounted on a Remotely Piloted Aircraft System (RPAS), as shown in Figure 1a. Plastic accumulation areas have been simulated by placing a variety of plastics and other types of litter (cans, wood residuals) in the field of view of the camera, as illustrated in Figure 1b. The litter contains weathered plastic objects originating from the port of Antwerp, Scheldt River (Belgium) and Vietnamese coastal sites. In total, 335 individual images were acquired during the flight. In some of the images, the plastic accumulations are not present as they were outside the field of view of the camera.

The second campaign (Campaign2) took place on 8 March 2022, when the MicaSense camera was mounted on a bridge over the canal Bocholt-Herentals near the VITO facilities (Mol, Belgium). The camera is attached in a fixed way to the pedestrian railing, at a height of 7 m above ground. A plastic accumulation area has been simulated on land similarly to Campaign1. Figure 2a illustrates the setup used for data acquisition during Campaign2. Note that the plastics used during the two campaigns were different. For Campaign2, virgin plastics (placemats, plastic bottles, pure plastic samples) have also been used, next to weathered plastics previously collected from the port of Antwerp. The positions of the different plastic samples are changed during the image acquisition. A total number of 135 images are captured from this setup. In the following, this set of images will be denoted by the term “Campaign2-Land” (see Figure 2b). Once the data acquisition over land was concluded, the MicaSense camera was moved to the middle of the bridge in order to observe only the water (no land in image footprint). Floating materials were thrown on water such that they entered the field of view of the camera (Figure 2c). The field crew recollected these materials using a net maneuvered from a boat. This part of the campaign (“Campaign2-Water”) contains 143 images. The data from Campaign2-Water are only used to observe the transferability of designed algorithms from land to water areas later in the paper.

2.2. Classes of Interest and Training Data

This work targets a multi-class classification scenario, justified by the fact that a binary classification of litter and non-litter classes is not suitable to distinguishing different litter materials, nor different classes of natural materials. While the main scope remains to spot plastic litter, a total of nine classes of interest have been identified, out of which four are natural materials: grass, tree, soil and water, and five are litter classes: cement, painted surface, oxidated metal, plastic and processed wood. While the classes were defined by observing the materials present in the available images, a few observations can be made:

-

The wood class is considered litter in our approach, i.e., processed wood, however wood items can have a purely natural origin (e.g., branches, dead trees fallen into rivers);

-

The difference between plastic and painted surfaces is only related to the origin of the training pixels: plastic pixels are extracted from plastic materials, while the painted surfaces correspond to areas such as cars, boats and reflectance tarps; however, most of the plastics can embed pigments to provide colors or can be superficially painted in practice, thus confusions between these two classes are tolerated as long as the pixels are correctly identified as plastic/painted litter;

-

Confusions between vegetation pixels (tree and grass) are largely tolerated and the algorithms were not tuned to obtain the best possible discrimination between these classes;

-

The natural materials could all be gathered in one single class as the corresponding pixels do not serve in monitoring litter pollution status nor they call for interventions (cleaning) in the area; however, it was preferred to make the distinction as it offers better insights on common class confusions between litter and non-litter materials.

It is important to note that the training/testing pixels were extracted from Campaign1 and the validation pixels were mostly extracted from Campaign2-Land. However, the images from Campaign2-Land did not contain ground-truth pixels for three classes: cement, tree and oxidated metal, thus images from Campaign1 were added to the validation set, but none of them from the set of images that served as sources of training/testing pixels. This ensures the separability of training/testing and validation data.

Once the targeted classes have been defined, a fully supervised classification approach is targeted. Specifically, a total number of 22.678 training pixels were manually extracted from 32 source images from Campaign1. As the flight has been performed at variable altitudes (between 10 m and 100 m), the spatial resolution of the images varied between 0.69 and 7 cm. The four natural classes benefit from the highest number of training points, followed by the “plastic” class. Figure 3 shows a bar plot of the available number of training pixels corresponding to each class. In this figure, the natural classes are signaled by blue color, while red color is used for the litter classes. Note that the “cement” and the “oxidated metal” classes have the lowest number of training points, which is not surprising as they are less present in the acquired images than the other materials.

High heterogeneity of the extracted pixels inside most of the nine classes has been observed, as shown in Figure 4, where the spectra from each class, boxplots for all spectral bands and the median spectrum of each material are plotted. The range of the y-axis in all plots is common (covering the interval [0, 1]) in order to better observe the difference in amplitude and heterogeneity of the considered materials. The different colors of the spectral signatures correspond to the color coding that will be used in the remainder of the paper to identify distinct materials. This coding is also resumed in Figure 4j. Note, in Figure 4h, that the plastic class exhibits high heterogeneity in terms of both spectral shape and spectral amplitude, which suggests that it is a difficult class to distinguish from other classes.

2.3. Classification Algorithm

2.3.1. Overall Approach

The designed classification scheme is a machine learning algorithm based on Random Forest approaches [52]. The Random Forest classifiers belong to the class of ensemble learning methods, in which multiple decision trees run separately and the outputs of the individual trees jointly contribute to a final result (label) for each input sample. The algorithm has been developed in the Python3 programming language using the Random Forest Classifier module of the Scikit-Learn machine learning library [53]. The classifier has been tuned in a two-step approach. First, a large number of spectral metrics have been derived from the reflectance values of the training points and an intermediate classifier has been tuned for optimal performance based on the complete set of features, including the original values of the spectral bands and the derived spectral metrics. A grid search approach, in which a large number of combinations of parameters was employed to evaluate the performances of the corresponding classifier architectures, has been used for this purpose. The goal in this step was to obtain a ranking of the input metrics w.r.t. their importance in the classification performance. Second, the 30 most informative spectral features of the best performing classifier in terms of overall accuracy were kept as input metrics in order to train a final classifier.

2.3.2. Metrics Selection

In multispectral data, the classification performance when solely the band reflectance values are considered, is limited due to various factors. Even if the number of spectral bands is low, they might still exhibit a certain degree of correlation. Furthermore, the reflectance values still embed the influence of external factors such as atmospheric scattering, variations in scene illumination and topographic relief. The low number of bands further limits the performance when the number of classes in the scene is large. These limitations can be alleviated when derived parameters are used. These parameters, denoted by the terms “metrics”, “spectral indices” or simply “indices” in the following, are obtained by combinations of reflectance values from distinct spectral bands.

In our study, spectral indices available in the literature have been inventoried and all the ones that were applicable to the spectral bands of the MicaSense multispectral camera were retained. A number of 76 indices were found this way. Second, another set of spectral indices has been defined by computing band differences, band ratios and normalized band ratios applied to the five available spectral bands. This second set of indices contains 30 indices and the reflectance values for the five spectral bands of the MicaSense camera were also included, thus a total of 111 indices have been employed in the first step of algorithm development. All the indices considered in this initial database are listed in the Appendix A jointly with their formulas (applied to MicaSense spectral bands) and their corresponding reference works. Note that spectral indices specifically designed for plastic detection [45,46,47] based on SWIR reflectance values cannot be applied to the MicaSense sensor due to the limited spectral range.

For illustration purposes, Figure 5 shows the median spectra of the considered training pixels for all the classes (Figure 5a) and radar plots of the indices computed from these spectra after shifting all the values to the positive quadrant and performing peak-to-peak normalization (Figure 5b–i). Each axis of a radar plot corresponds to a specific spectral index.

The 111 indices were computed for all pixels available in the training set and the best intermediary classification algorithm has been selected after the grid search was performed. For all the tested variants of the algorithm, the training database was further split in training and testing samples on the basis of a ratio of 70%/30% split performed on the source images, such that the sets do not contain points originating from common images.

The 30 most relevant metrics for the best performing intermediate classifier (marked in bold letters in the Appendix A) are: Near-Infrared reflectance (NIR), Browning Reflectance Index (BRI), Ashburn Vegetation Index (AVI), Differenced Vegetation Index MSS (DVIMSS), Inverse Reflectance 717 (IR717), Tasseled Cap—Non Such Index MSS (NSIMSS), RedEdge reflectance (RE), normalized Blue-NIR index (nBNIR), ratio Blue-RedEdge (rBRE), Plant Senescence Reflectance Index (PSRI), normalized Blue-Green (nBG), Adjusted Transformed Soil-Adjusted Vegetation Index (ATSAVI), ratio Blue-Green (rBG), normalized ratio Green-Red (nGR), normalized green reflectance (NormG), Coloration Index (CI), ratio Green-Red (rGR), Blue-Wide Dynamic Range Vegetation Index (BWDRVI), Chlorophyll Vegetation Index (CVI), Tasseled Cap—Soil Brightness Index MSS (SBIMSS), Anthocyanin Reflectance Index (ARI), Normalized Green-Red Difference Index (NGRDI), Difference Blue-Red (dBR), Normalized Blue-Red ratio (nBR), ratio Blue-Red (rBR), Red-Edge 2 (Rededge2), Pan Normalized Difference Vegetation Index (PNDVI), Structure Intensive Pigment Index (SIPI), difference Blue-RedEdge (dBRE) and Green Leaf Index (GLI).

2.3.3. Final Configuration

The 30 selected metrics were used in the training of the final classifier. The grid search approach has been once again employed. At the end, the parameters of the final classifier are as follows (see their significance in the online documentation of the scikit-learn implementation [54]): ‘bootstrap’: True, ‘ccp_alpha’: 0.0, ‘class_weight’: None, ‘criterion’: ‘gini’, ‘max_depth’: 125, ‘max_features’: auto’, ‘max_leaf_nodes’: None, ‘max_samples’: None, ‘min_impurity_decrease’: 0.0, min_samples_leaf’: 1, ‘min_samples_split’: 3, ‘min_weight_fraction_leaf’: 0.0, ‘n_estimators’: 25, ‘n_jobs’: None, ‘oob_score’: False, ‘random_state’: None, ‘verbose’: 0, ‘warm_start’: False.

2.3.4. Validation Data

Targeting an independent validation, ground-truth maps have been manually generated based on five validation images from Campaign2-Land. Areas (polygons) containing different materials have been digitized and assigned the corresponding labels. Only areas visually homogeneous were considered. Figure 6 shows the RGB representation of the validation images and the corresponding ground-truth maps. In this figure, the color scheme from Figure 4j applies to all ground-truth maps.

Some observations should be made before further proceeding with the results analysis. First, due to a very low or null number of validation pixels in Campaign2 for cement, oxidated metal, tree and wood classes, images from Campaign1 were also used to define validation pixels, while ensuring no overlap with the training/testing images occurred. Second, despite the fact that the data from Campaign2 were acquired by the MicaSense camera mounted on a bridge, the data processing and camera settings were not different from the ones set during Campaign1, thus the images from Campaign2 have the same spectral characteristics as the ones from Campaign1 and the obtained results are valid for aerial imagery. Third, the classification algorithm is not trained to act on shadowed pixels, thus the shadowed areas will be removed from both the ground-truth maps and the inferred maps. To this end, a threshold T = 0.11 has been empirically imposed on the pixel brightness (simply computed by the sum of the reflectance values in the R, G and B bands) to separate shadowed and non-shadowed areas. More insights on the issues induced by the presence of shadows in the imagery are given further in the discussion section. The number of independent validation points per class is shown in

Table 1. Number of independent validation points, per class.

Class	Grass	Soil	Tree	Water	Cement	Painted	Oxidated	Plastic	Wood
Number of validation points	254,543	66,351	431,151	238,269	7559	6575	17,373	399,545	9543

.

3. Results

In this section, we analyze the performance achieved by the designed classifier and provide interpretations of the observed strengths and weaknesses.

3.1. Algorithm Performance

Figure 7 shows the confusion matrix (relative values) obtained for the nine classes over the test data and in

Table 2. Accuracy metrics for the designed classifier, computed in test data.

	Precision	Recall	F1-Score	Support
Grass	0.93	0.88	0.91	1074
Soil	0.91	0.95	0.93	1281
Tree	0.91	0.95	0.93	1244
Water	1.00	1.00	1.00	1109
Cement	0.88	0.88	0.88	153
Painted surface	0.90	0.88	0.89	528
Oxidated metal	0.91	0.95	0.93	215
Plastic	0.94	0.90	0.92	1363
Wood	0.86	0.87	0.86	690
Accuracy			0.93	7657

the following performance metrics are listed next to the number of corresponding supporting points:

-

Precision (P): P = TP/(TP + FP)

-

Recall (R): R = TP/(TP + FN)

-

F1-score: F1-score = (2∙P∙R)/(P + R)

-

Accuracy: Accuracy = (TP + TN)/(TP + FN + TN + FP),

where TP = True Positives, FP = False Positives, TN = True Negatives and FN = False Negatives.

From Table 2, it can be seen that all metrics are higher than 0.9 for plastic. Figure 7 shows that plastic pixels are most often confused with painted materials. The lowest values of the computed performance metrics are obtained for wood, however they are all above 0.85 which indicates a good performance. In test data, all water pixels are correctly identified. The highest confusion occurs between grass and trees, which is tolerable, as explained before. Another relatively high confusion is observed for wood and soil, which is explainable by the fact that their median spectra are very similar in terms of both shape and amplitude, as shown in Figure 4.

3.2. Validation Performance

In Figure 8, the confusion matrix computed for the validation data points (relative values) is displayed.

In Figure 8, it can be observed that most of the classes exhibit lower accuracies in validation data than in the test data, as expected. However, most of the classes still attain a relatively high accuracy, except for the grass and the wood classes, which are very low in comparison to the training performance. Another class with a relatively high drop (more than 10%) in accuracy is the painted surface class. On the one hand, the confusion matrix suggests that the high drop in accuracy for the grass class is mainly originating in the strong confusion between this class and the tree class. This is not a worrisome issue, as both classes are green natural materials and tuning the classification for this type of materials is out of the scope of our application. The same logic applies to the confusion between painted surfaces and plastic surfaces—this confusion is embedded in the class definition. On the other hand, the wood class exhibits high confusion with the soil class, but the drop in accuracy is also partly due to the confusion with the plastic class. In order to better investigate the observed confusions, Figure 9 plots the ground-truth classification polygons defined in Figure 6 next to their counterpart classification maps retrieved by the designed classifier. In the retrieved maps, the shadowed areas were masked by enforcing a threshold T = 0.11 on the pixel intensity, computed as the sum of reflectance in the B, G and R bands. The threshold has been applied to all pixels classified as litter.

Figure 9 confirms the behavior suggested by the confusion matrix. Each of the images 1 and 2 contain one plastic region that was classified as paint, which explains the observed confusion. In Image 3, the confusion between grass and tree classes is obvious. The errors are mainly observed on the grass class, for which many pixels are classified as tree. The tree class is classified with high accuracy, as illustrated by the classification maps of Image 4. The wood class suffers from confusions with more classes: in Image 1 all three wood polygons are mostly confused with soil and plastic; in Image 2, one polygon labeled as plastic is largely masked by the shadow removal approach as it is a dark material, but some pixels are also classified as wood; in Image 3, some cement pixels are classified as wood; in Image 5, wood pixels are sometimes masked by the shadow removal criterion or they are classified as soil. This shows that the wood class is difficult to correctly identify and the reasons for this behavior will be further analyzed in the discussions section.

The main class of interest, plastic, is identified with relatively high accuracy in all validation images. The confusion with painted surfaces, which is the most obvious error occurring in the retrieved maps (see Images 1 and 2) is tolerated, as explained earlier. However, the importance of distinguishing between the two materials cannot be denied as many materials can be painted without containing necessarily plastic, e.g., car bodies and roofs. It is likely that higher wavelengths are needed in order to be able to achieve this distinction with higher accuracy. The camera used in our study has the advantage of being easily accessible at a relatively low cost and easy to operate, but it does not benefit from SWIR measurement channels. From the point of view of the image pre-processing, another advantage of this camera is that it includes an irradiance sensor, which allows for the conversion of the raw data into reflectance even in the absence of a spectralon in the field. The plastic materials are shown to be identifiable from the considered data, however further distinctions, e.g., between different types of plastics, might require extended spectral range and higher spectral resolution.

4. Discussion

In this chapter we take a journey through the retrieved results and observe strengths, weaknesses, and possible improvements of the proposed approach, and we identify directions for future work.

4.1. Influence of Shadows

In shadowed areas, the classification proves to be unreliable, this issue stemming from a two-fold reason: (1) the shadowed areas suffer from distorted spectra due to the low level of radiation hitting the surface, and (2) naturally, training pixels are not selected from shadowed areas. The algorithms are thus generally not trained to deal with shadowed areas. In high resolution images such as the ones used in this work, shadows can be observed next to the tall objects, but even at a satellite level it is common to derive shadow masks from the remote sensing imagery such that caution is taken when dealing with the corresponding flagged pixels.

In this work, a simple shadow filtering method has been applied, based on the pixel intensity. As mentioned in Section 3.2, a threshold T = 0.11 has been set on the sum of reflectances in the R, G and B bands in order to separate shadowed from non-shadowed pixels. This threshold has been empirically set by finding an equilibrium between the quantity of pixels marked as shadows and the amount of useful pixels remaining in the image. Figure 10 illustrates the effect of four different threshold values on the retrieved classification map for Image 1. In this figure, it can be observed that the raw classification map is strongly affected by the occurrence of shadowed pixels. Specifically, at the transition between land and water areas a large part of the pixels is incorrectly marked as plastic, while many shadowed pixels along this transition area are classified as oxidated metal. By masking the shadowed pixels, the classification maps improve substantially. Higher thresholds lead to a more shadow-free conservative masking. However, the removal of false positives should always be in harmony with the preservation of true positives, thus the choosing of an illumination threshold should be done with caution, as dark littered pixels might also be removed. For example, the threshold applied in Figure 10f is too high as some plastic areas, correctly classified in the raw map, are masked out jointly with the true shadowed pixels. It is also likely that the shadow thresholds need tuning when different spatial scales are used, as in low resolution pixels the shadowed areas are mixed with bright areas, resulting in an intermediate brightness between the two pixel types. Efficient and accurate shadow removal approaches are needed for the post-processing of the classification maps and will be further investigated in future works.

4.2. Plastic Quantification and Representation

Once the classification map is produced, it is straightforward to estimate the area of plastic litter in the image, starting from the spatial resolution of the input imagery. Once more, the post-processing strategies play an important role in the accuracy. In

Table 3. Estimation of area covered by plastic w.r.t. the threshold employed for shadow removal.

Threshold		None (Ground-Truth Map)	None (Raw Map)	T = 0.05	T = 0.1	T = 0.15	T = 0.2
Quantification of plastics	Pixels	147,009	275,464	196,400	178,580	172,292	153,482
	Area [m2]	3.387	6.346	4.525	4.114	3.969	3.536

, the ground-truth plastic area is listed next to the estimated plastic areas for the filtered maps illustrated in Figure 10b–f, respectively. For the considered image, the spatial resolution is 0.48 cm, thus each pixel occupies an area of 2.3 × 10⁻⁵ m². Note that the true plastic area is slightly underestimated in Table 3, as very small plastic objects and transition areas between materials were not taken into account, in order to avoid contributions from mixed pixels. Furthermore, it is expected that pixels incorrectly classified as plastic are still present in the classification map after the shadow filtering. However, the table confirms without doubt that the raw classification map suffers from a strong over-estimation of plastic area, due to the shadowed pixels incorrectly classified as plastic, as previously explained. The shadow filtering significantly contributes to a better estimation of the plastic area for all considered thresholds.

The approach above provides an estimation of the total area occupied by plastic pixels in the considered image. However, for policy makers, another type of characterization might be relevant: the number of plastic objects (separate areas littered with plastic), their size and their location, among others. This type of detail requires an additional spatial post-processing approach. In Figure 11, we illustrate the results of a straightforward strategy to achieve this goal, applied to the plastic map extracted from Figure 10e: first, plastic regions are delimited to create spatial blobs by thresholding the smoothed classification map obtained after applying a spatial Gaussian kernel (Figure 11b—each color represents one blob or plastic region); second, these blobs are visualized as a pie chart w.r.t. to their sizes (measured in pixels; Figure 11c); third, the blobs are converted to circular regions having equal areas to the original blobs and represented on the map for easier visualization (Figure 11d—the colors mirror the ones in Figure 11b). In case the analyzed images are georeferenced, geographic coordinates can be assigned to the plastic regions to support the planning of cleaning activities. Depending on the scope, small blobs can be neglected, e.g., if the main targets are the large accumulations or if avoiding false positives is of utmost importance.

4.3. Sources of Classification Errors

Apart from the issues identified over shadowed areas, the quality of the retrieved classification map might be influenced by other factors, such as data quality, inter-class correlations and spatial resolution.

4.3.1. Data Quality

In order to evaluate the quality of input data, reference spectra have been acquired during Campaign2-Land using an ASD FieldSpec spectrometer (https://www.malvernpanalytical.com/en/products/product-range/asd-range/fieldspec-range/fieldspec-4-standard-res-spectroradiometer, accessed on 15 November 2022) over various targets. This spectrometer covers the spectral range 350–2500 nm with a spectral resolution of 1 nm, resulting in 2151 spectral bands across VNIR and SWIR regions. In total, twelve different materials were scanned and for each material three spectra were acquired. The reference (hyperspectral) spectrum of one material is considered as the median spectrum of the individual measurements. The reference spectra were then convolved with the spectral response functions of the MicaSense camera in order to derive ground-truth multispectral signatures of all samples. Pictures of the eleven material samples are shown in Figure 12a–l and the reference (simulated, ground-truth) MicaSense spectra are shown in Figure 12m.

In order to inspect the quality of the fitting between the reference spectra and the acquired spectra, actual spectra were extracted from polygons drawn on the corresponding regions from one image. Then, saturated pixels (e.g., the ones having reflectance larger than 1 in at least one band) and possibly shadowed pixels (based on a soft shadow threshold: T = 0.05) were excluded. Based on the remaining pixels we plot, in Figure 13, for all the considered samples, the following data: boxplots of reflectance values extracted from the actual imagery, median spectrum retrieved from the actual imagery (blue) and reference spectrum simulated from the ASD measurements by convolution with the MicaSense response functions (green).

For some materials, a large spread of the acquired reflectance spectra can be observed from the boxplots (see Materials #3, #6 and #9). Good fitting with the reference spectra is obtained for four materials (#2, #4, #8, #12). In general terms, the acquired spectra follow the shape of the reference spectra, however differences are clearly visible, especially in the RE and NIR bands. It is also interesting to observe that the materials with high amplitude (#6, #9) exhibit lower amplitude than expected in real data. This type of variability is regularly encountered in aerial multispectral imagery and it is expected that the performance of classification algorithms will evolve jointly with the advancements in sensor development and advanced pre-processing methods for the imagery itself. The use of spectral indices instead of absolute reflectance values partially alleviates the issue of erroneous spectra as long as a good fit is obtained between the reference and actual spectra in terms of spectral shape.

4.3.2. Inter-Class Correlations

The performances of classification algorithms are inherently influenced by the number of spectral bands sampling the electromagnetic spectrum. When the number of spectral bands decreases, the confusion between different classes also increases as distinctive features characterizing one specific class might not be captured. In the following, we analyze the degree of confusion between the nine considered classes in the experiments by measuring the minimum, maximum and mean Pearson correlation factor (C), in absolute values, over all pairs of spectra drawn from two different classes. In other words, for a pair of classes C1-C2, the absolute value of the correlation factor is computed over all possible pairs of spectra containing one spectrum from C1 and one spectrum from C2, and the minimum, maximum and mean values are extracted from the list of computed correlations.

Table 4. Inter-class correlations. For each pair, the minimum, maximum and mean Pearson correlations computed over all pairs of spectra drawn from the two classes. The values below the diagonal are greyed out due to the fact that they mirror the values above the diagonal.

		Grass	Soil	Tree	Water	Cement	Painted Surface	Oxidated Metal	Plastic	Wood
Grass	min (C)	-	0.605	0.520	<0.001	0.094	<0.001	<0.001	<0.001	<0.001
	max (C)	-	>0.999	1	0.953	>0.999	>0.999	0.994	>0.999	>0.999
	mean (C)	-	0.890	0.985	0.381	0.756	0.633	0.766	0.705	0.911
Soil	min (C)	0.605	-	0.460	<0.001	0.265	<0.001	0.044	<0.001	<0.001
	max (C)	>0.999	-	>0.999	0.933	>0.999	>0.999	>0.999	>0.999	>0.999
	mean (C)	0.890	-	0.862	0.430	0.908	0.618	0.877	0.637	0.939
Tree	min (C)	0.520	0.460	-	<0.001	0.074	<0.001	<0.001	<0.001	0.012
	max (C)	1	>0.999	-	0.983	0.986	>0.999	0.997	>0.999	>0.999
	mean (C)	0.985	0.862	-	0.368	0.709	0.634	0.742	0.713	0.888
Water	min (C)	<0.001	<0.001	<0.001	-	<0.001	<0.001	<0.001	<0.001	<0.001
	max (C)	0.953	0.933	0.983	-	0.937	>0.999	0.969	>0.999	>0.999
	mean (C)	0.381	0.430	0.368	-	0.381	0.388	0.327	0.432	0.449
Cement	min (C)	0.094	0.265	0.074	<0.001	-	<0.001	0.103	<0.001	<0.001
	max (C)	>0.999	>0.999	0.986	0.937	-	>0.999	>0.999	>0.999	>0.999
	mean (C)	0.756	0.908	0.709	0.381	-	0.585	0.844	0.545	0.861
Painted surface	min (C)	<0.001	<0.001	<0.001	<0.001	<0.001	-	<0.001	<0.001	<0.001
	max (C)	>0.999	>0.999	>0.999	>0.999	>0.999	-	>0.999	>0.999	>0.999
	mean (C)	0.633	0.618	0.634	0.388	0.585	-	0.571	0.571	0.618
Oxidated metal	min (C)	<0.001	0.044	<0.001	<0.001	0.103	<0.001	-	<0.001	<0.001
	max (C)	0.994	0.999	0.997	0.969	>0.999	>0.999	-	0.998	>0.999
	mean (C)	0.766	0.877	0.742	0.327	0.844	0.571	-	0.554	0.824
Plastic	min (C)	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001	<0.001	-	<0.001
	max (C)	>0.999	>0.999	>0.999	>0.999	>0.999	>0.999	0.998	-	>0.999
	mean (C)	0.705	0.637	0.713	0.432	0.545	0.571	0.554	-	0.656
Wood	min (C)	<0.001	<0.001	0.012	<0.001	<0.001	<0.001	<0.001	<0.001	-
	max (C)	>0.999	>0.999	>0.999	>0.999	>0.999	>0.999	>0.999	>0.999	-
	mean (C)	0.911	0.939	0.888	0.449	0.861	0.618	0.824	0.656	-

resumes the computed inter-class correlations.

Table 4 explains why some classes suffer from high confusion with other classes. For example, the wood class, whose abundance is often over-estimated in the classification maps, has maximum correlation factors of at least 0.999 with all the other classes. However, the mean correlation factor of the wood class with the plastic class is the third lowest among the correlation factors of the former (after the correlation factor with water and painted surface). This indicates that the wood class is more likely to be confused with other classes than with plastic, which is positive for the scope of our experiments as it does not lead to overestimation of plastic areas due to confusion with wood. The water class is the least correlated with the others, which explains the very high accuracy for this class in both testing and validation data. The metrics in Table 4 also confirm the large confusion between grass and tree spectra: these two classes are the only ones in which there exists at least one pair of spectra that are virtually identical in MicaSense multispectral data, resulting in a maximum correlation factor of 1.

4.4. Influence of Spatial Resolution

The training pixels in our experiments have been selected from relatively homogeneous areas in high-resolution images, such that they can be considered pure spectra of the materials of interest. The spatial resolution of the acquired imagery can be different if another sensor (with similar spectral characteristics) is employed or if the flight altitude of the sensor changes during the data acquisition flight. It is thus natural to question the abilities of the designed algorithm when, due to a degradation of the spatial resolution, the target materials jointly occupy mixed pixels.

In order to illustrate the behavior of the classification scheme for variable spatial resolution, the high-resolution imagery has been resampled using bilinear interpolation by binning the observed high-resolution pixels inside spatial windows of $10\times 10$, $30\times 30$ and $50\times 50$ pixels. This simulates the degradation of the spatial resolution by factors of 10, 30 and 50, respectively. The classification algorithm was then applied to the lower resolution images and the set of obtained raw classification maps for two targets of interest—one plastic litter area and one car—are inspected in Figure 14. In this figure, F denotes the degradation factor applied to the original imagery, R represents the spatial resolution of the respective image measured in centimeters, and for each degradation factor it displays the RGB representation of the area of interest, the true litter map (drawn manually over the original image) and the retrieved classification maps after applying the design classification scheme.

Figure 14 shows that the discrimination power between litter and natural materials decreases with the degradation of the spatial resolution, as expected. However, littered areas can still be spotted even if the image is degraded with a degradation factor of F = 30 in both considered cases. For lower spatial resolution, i.e., when F = 50, the car pixels are still marked as littered area, while the plastic area is entirely missed. The difference between the two cases is that the car is a continuous surface while the plastic objects are mixed with wood residuals and are more sparsely distributed over a natural background (soil and grass). When the spatial resolution degrades, more spectral mixing occurs in the area of plastic litter. The issue of mixed pixels can be tackled either by including mixed spectra in the training database during the training stage or by employing other techniques relying on pure pixels, such as sparse spectral unmixing, where the collection of pure spectra act as an external spectral library and the number of materials inside the pixel is minimized by solving a constrained sparse regression problem [55,56]. Note that all the post-processing steps (shadow removal, spatial filtering) also need tuning w.r.t. the employed thresholds when the spatial resolution changes. This results that, while a common classification framework can be followed for different spatial resolutions, the final results strongly depend on fine tuning of the employed parameters at each processing step.

4.5. A Note on Algorithm Transferability to Water Areas

One of the most important questions when referring to litter detection is the ability to spot problematic areas not only on land, but also over water. Water is a material with particular spectral characteristics in the NIR and SWIR spectral regions, i.e., the signals retrieved by a sensor over water pixels are strongly attenuated given that pure water has a very low reflectance in the NIR region and entirely absorbs the incoming radiation at wavelengths larger than 1200 nm. Moreover, litter objects in water areas can be found in different states of exposure: they can float, they can be partially/entirely submerged, and they can be dry or wet depending on their own density and weather conditions. Pixels with distorted spectral shape are also encountered in water datasets due to sun glint, introducing additional challenges despite the fact that they only contain water in nature. In [57], the authors investigated the spectral properties of various virgin and weathered plastic materials in a controlled environment in which various suspended sediment concentrations and material depths have been used. The resulting dataset (https://data.4tu.nl/articles/dataset/Hyperspectral_reflectance_of_marine_plastics_in_the_VIS_to_SWIR/12896312/2, accessed on 15 November 2022) proves a high variability of the observed spectra when these conditions vary. In this subsection, we present a limited experiment intended to investigate if a classifier designed for land areas, which is the case for the classifier described in this work, could be successfully applied to detecting litter over water areas, under these known circumstances.

During the validation campaign (Campaign2), various litter objects have been placed on the streamflow under the bridge where the camera was mounted. The camera has also been moved to cover the water area and various images were acquired. After the data acquisition, all the floating litter objects have been recollected by the VITO field crew by using a net maneuvered from a boat. The collected images were processed according to the methodology described in Section 2. Finally, the designed classifier was applied to these images.

In Figure 15, two RGB representations of images acquired over water are shown jointly with the corresponding retrieved classification maps.

In Figure 15, the vast majority of the plastic pixels are correctly classified. The wood pixels are only partially identified correctly. For these pixels, confusions with soil and plastic occur. In image 1, two floating dry leaves are mostly classified as wood which is acceptable given the similarities and taking into account that no dry leaf class has been defined, while one leaf is mostly classified as plastic. The leaf in image 2 is mostly classified as oxidated metal, but some of its pixels are also classified as plastic. The background (water) pixels are generally correctly identified, however the confusion with oxidated metal is the most frequent, resulting in a salt-and-pepper type of noise in the classification maps. The fact that the leaf in image 2 is mostly classified as oxidated metal might be due to the fact that the leaf is partially submerged. Overall, we can observe that all the litter objects are detected in these images and the plastic objects can be inferred with high accuracy, but over-detection of plastic is common and the distinction between different types of litter (wood-soil, wood-plastic etc.) is difficult. This distinction is important as biodegradable litter is less concerning than the non-biodegradable one. The large number of points classified as oxidated metal in the background pixels could be avoided by redefining the litter classes themselves. Cement, soil and oxidated metal are not floating materials, which means that these classes might be obsolete over water areas, depending on the targeted area. We can conclude that algorithms designed for land litter detection can only be used for limited purposes over water (e.g., to detect litter objects), however water and land areas should benefit from separately trained schemes which take into account the specificity of light absorption and reflection for each of them and which account for different target classes in order to maximize the probability of success. Investigations on the differences between the land/water approaches, inference of the most useful spectral bands in the two cases, and estimation of detection limits for future sensors tailored to litter detection are all targeted subjects for our future work.

5. Conclusions

In this paper, machine learning classification algorithms based on Random Forest approaches have been designed to detect litter in aerial images acquired by affordable, off-the-shelf multispectral cameras carried by remotely piloted aircraft systems. It was shown that the main targeted pollutant represented by plastic materials can be spotted with high accuracy—higher than 0.88 in both test and validation data. The distinction between plastic and other litter materials has also been achieved with high accuracy. Among the materials considered as litter, it was shown that wood residuals are more difficult to distinguish due to their very high spectral similarity to the other classes of materials, especially with soil. However, wood is a material that does not pose the same critical threats as plastic to the environment and the human health population. Better discrimination between all considered classes could be achieved by employing spectral cameras with higher spectral resolution and extended spectral range. Nevertheless, our study was designed such that a critical component of operational exploitation of the system was highly valued: the balance between detection accuracy and operational costs.

The classification algorithm has been designed in a two-step approach: first, a large pool of spectral metrics have been used in an intermediate scheme to decide on their importance; second, the most important 30 metrics have been employed in the final classification scheme. As the classifier was not trained for shadowed areas, in which spectral distortions are common due to the limited amount of radiation hitting the surface, post-processing methods have been explored to improve the quality of the retrieved classification maps. Straightforward quantification and visualization approaches intended to ease the exploitation of the maps by interested users have also been presented.

An important part of this paper has been devoted to understanding the factors that cause classification errors. Based on ground reference targets scanned with a field spectrometer, it was shown that data errors are common in multispectral imagery, thus the use of spectral indices is highly recommended as they mitigate limitations related to spectral amplitude errors and better embed useful information related to the spectral shape of the input signatures. The high inter-class correlations encountered in multispectral data, inherently induced by the relatively low number of spectral bands, are also sources of class confusions. In an experiment considering the spatial resolution of the input imagery, we have shown that the detection of areas polluted with plastic is not only dependent on the spectral-spatial responses of the scanned pixels, but also on the spatial distribution of the contained materials. In other words, mixed pixels containing litter in images with low spatial resolution call for adaptations of the training data or for exploring methods that analyze sub-pixel compositions, such as spectral unmixing. Finally, it was shown that classification schemes trained with land pixels are not directly transferable to water areas due to the particularities of water pixels, e.g., the low signal in the NIR/SWIR spectral regions. When applied to images containing floating objects, the designed classifier was able to spot non-water areas, however the classification inside these areas suffered from more confusions than over land. This implies that classification schemes should be designed separately for land and water targets such that the employed metrics are tailored to the specific case. The considered classes should also be adapted. For instance, materials that do not naturally float (soil, oxidated metal, cement) are less concerning when monitoring water areas. All the above insights will be further exploited in our future research work, jointly with adaptations of the proposed framework to account for an extended set of classes and to ensure transferability to images acquired in different geographic areas and under different environmental conditions.