Next Article in Journal
Scene Classification Method Based on Multi-Scale Convolutional Neural Network with Long Short-Term Memory and Whale Optimization Algorithm
Next Article in Special Issue
Monitoring Suspended Sediment Transport in the Lower Yellow River using Landsat Observations
Previous Article in Journal
Uncovering Plastic Litter Spectral Signatures: A Comparative Study of Hyperspectral Band Selection Algorithms
Previous Article in Special Issue
Modeling of Suspended Particulate Matter Concentration in an Extremely Turbid River Based on Multispectral Remote Sensing from an Unmanned Aerial Vehicle (UAV)
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Remote Sensing
Volume 16
Issue 1
10.3390/rs16010173
remotesensing-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Classification of River Sediment Fractions in a River Segment including Shallow Water Areas Based on Aerial Images from Unmanned Aerial Vehicles with Convolution Neural Networks

by
Mitsuteru Irie
1,*,
Shunsuke Arakaki
2,
Tomoki Suto
3 and
Takuto Umino
4
1
Faculty of Engineering, University of Miyazaki, Miyazaki 889-2192, Japan
2
Graduate School of Science and Technology, Kumamoto University, Kumamoto 860-8555, Japan
3
Japan Railway Construction, Transport and Technology Agency, Yokohama 231-8315, Japan
4
Department of Civil and Environmental Engineering, University of Miyazaki, Miyazaki 889-2192, Japan
*
Author to whom correspondence should be addressed.
Remote Sens. 2024, 16(1), 173; https://doi.org/10.3390/rs16010173
Submission received: 23 October 2023 / Revised: 28 December 2023 / Accepted: 29 December 2023 / Published: 31 December 2023
(This article belongs to the Special Issue Remotely Monitoring Water, Sediment, and Carbon Transported in Rivers and Estuaries)
Browse Figures
Review Reports Versions Notes

Abstract

:
Riverbed materials serve multiple environmental functions as a habitat for aquatic invertebrates and fish. At the same time, the particle size of the bed material reflects the tractive force of the flow regime in a flood and provides useful information for flood control. The traditional riverbed particle size surveys, such as sieving, require time and labor to investigate riverbed materials. The authors of this study have proposed a method to classify aerial images taken by unmanned aerial vehicles (UAVs) using convolutional neural networks (CNNs). Our previous study showed that terrestrial riverbed materials could be classified with high accuracy. In this study, we attempted to classify riverbed materials of terrestrial and underwater samples including that which is distributed in shallow waters where the bottom can be seen using UAVs over the river segment. It was considered that the surface flow types taken overlapping the riverbed material on images disturb the accuracy of classification. By including photographs of various surface flow conditions in the training data, the classification focusing on the patterns of riverbed materials could be achieved. The total accuracy reached 90.3%. Moreover, the proposed method was applied to the river segments to determine the distribution of the particle size. In parallel, the microtopography was surveyed using a LiDAR UAV, and the relationship between the microtopography and particle size distribution was discussed. In the steep section, coarse particles were distributed and formed riffles. Fine particles were deposited on the upstream side of those riffles, where the slope had become gentler due to the dammed part. The good concordance between the microtopographical trends and the grain size distribution supports the validity of this method.

1. Introduction

Mapping the particle size of riverbed materials is an important part of defining river habitats. Riverbed materials, as feeding grounds, spawning grounds, and shelters for aquatic invertebrates and fish, fulfill multifaceted environmental functions [1,2,3]. Their particle size reflects the tractive force during the flooding term [4,5]. In addition, in flood simulations, the roughness of riverbeds is determined based on the particle size of the material, so the evaluated particle size affects the reproducibility of the simulation [6]. Furthermore, if the vegetation that is adapted to the grain size characteristics develops, it regulates flow regimes and induces new sediment transport characteristics and river morphology [7,8]. The distribution of riverbed material particle size is essential information from the viewpoint of flood control and the environmental management of river channels.
The conventional field methods such as the grid-by-number and volumetric methods [9,10] for riverbed material investigations require a lot of labor and time for sieving or for the direct measurement of particle size. Therefore, conducting the surveys in a wide-ranging area with high density would require an extremely large amount of cost and could not be carried out. In addition, the spatial representativeness is also an issue because the sampling is within a quadrat with a limited area of less than 1 m2, while the site ranges some square kilometers.
In recent years, unmanned aerial vehicles (UAVs), which are used for a variety of purposes, are capable of comprehensively photographing a wide-ranging area. If particle sizes can be measured via the automated extraction of individual particles from a large number of captured images, it is possible to map the particle size distribution of vast river areas.
In river management, there are some applications of image recognition to identify information on river channels and other features. As an example of a method for examining riverbed materials, M. Detert et al. [11,12] developed the software BASEGRAIN which automatically identifies each particle in an image taken of a riverbed, measures the major axis and minor axis, and automatically calculates the particle size distribution. Harada et al. [13] verified the analytical accuracy of BASEGRAIN with the results obtained by actually performing the volumetric method. It showed that the particle size distributions of the two have good concordance, and they mapped the particle size of a wide range of riverbed materials. However, although BASEGRAIN is an effective method for obtaining the particle size distribution of materials contained in a single image, compared to the particle size evaluation via the image analysis in which the shooting conditions are fixed to some extent, as introduced above, the manual adjustments of the image brightness and shadows depending on the sunlight condition of the outdoors is required. In cases where the shadows that initially appear on a single stone surface are counted as multiple particles, the obtained particle size distribution may be significantly different, unless the analysis conditions are adjusted and the population brake is applied [14]. Such tuning processes increase the effort and time spent. On the other hand, in the actual river management scenario, or when providing a distribution of riverbed roughness for two-dimensional shallow water flow simulation, it is often sufficient to be able to map a representative grain size such as D50. Therefore, if the purpose is limited to mapping, there is no need to extract individual particles from the images; it is sufficient to divide the image into a large number of grids and determine the representative particle size for each grid.
A similar analysis method is the land cover classification using remote sensing data and machine learning by comparing pixel values of satellite and aerial photographs with ground truth data [15,16,17,18]. This method is effective when a wide area can be photographed under the same conditions, but the problem is that it cannot be classified properly if the pixel values change due to differences in the photographing time or weather. On the other hand, image recognition using deep learning can capture more advanced image features rather than simple pixel values, so it can be expected to achieve robust classification against differences in brightness, etc. [19,20].
Since the development of deep learning for semantic segmentation [21], which classifies regions of an entire image, it has been applied in various fields of manufacturing and resource management. The accuracy of semantic segmentation has been evaluated using the benchmark datasets, and there are also datasets for cityscapes and roadscapes. In recent years, the most accurate models have been announced every year for these datasets. Research on the application of deep learning to satellite images and aerial photography is progressing [22], and the detection of not only land cover but also features such as buildings [23,24,25], roads [26,27,28], and land cover [29,30,31], is available. Major examples of semantic segmentation as applied to rivers are the river channel detection and the river width measurement using satellite images [32,33,34]. As for the examples related to river management, there have been attempts to detect estuary sandbars [35], to monitor water levels during floods [36,37,38,39], to conduct the water binary segmentation task to aid in fluvial scene autonomous navigation [40], and to detect fine-grained river ice [41]. The number of research on semantic segmentation for river scenes is less than that for terrestrial areas, because the benchmark datasets are smaller than those for terrestrial data. Any cases mentioned above are based on the analysis of two-dimensional images, such as satellite images, UAV aerial images, and surveillance videos.
In recent years, not only two-dimensional image information but also two-dimensional point cloud data obtained from structure from motion (SfM) analysis [42,43] and terrestrial laser scanning (TLS) [44] have been analyzed to investigate riverbed morphology, roughness, and surface sedimentation.
However, the point cloud density obtained via SfM and TLS is not homogeneous depending on the characteristics of the ground surface pattern and the distance from the instrument. If the main purpose is mapping that requires high precision such as roughness or surface sedimentology, careful attention must be paid to data acquisition and selection. Furthermore, with SfM, it is possible to analyze underwater riverbed materials in rivers with high transparency and shallow water depth, but with TLS, the laser mounted on a typical model is absorbed by the water surface, so it cannot survey riverbed topography. The green laser developed for airborne laser bathymetry [45] is still an expensive and not widely available technology.
Onaka et al. used the convolution neural network (CNN) to determine the presence or absence of clam habitats [46] and to classify sediment materials in tidal flatwater areas [47]. These targets are stagnant water areas where there is no current and no diffuse reflection of waves from the surface of the water. On the other hand, in river channels, as Hedger et al. [48] pointed out, the automated extraction of information on river habitats from remote sensing imagery is difficult due to a large number of confounding factors. There are some attempts to apply artificial intelligence techniques to map surface cover types [49], hydromorphological features [50], mesohabitats [51], and salmon redds [52], etc., but there is a possibility of misclassification due to water surface waves [53]. The authors have previously attempted to automatically classify the particle size of riverbed materials in terrestrial areas using UAV aerial images and image recognition using artificial intelligence (AI) during normal water conditions [54]. The popular pre-trained CNN, GoogLeNet [55], was retrained to perform a new task using the 70 riverbed images using transfer learning. The overall accuracy of the image classification reaches 95.4%.
In this study, we applied the same method to the shallow water images taken from UAVs over the river channels when the water was highly transparent and attempted the classification. The representative particle size, which serves as a reference for training data and test data, was determined from the bed material samples taken from the quadrat underwater surrounded by fences to prevent advection during collection. Furthermore, the network trained in transfer learning was used to map the particle sizes in the river channel range. Trends of the particle size were compared with the detailed topographical changes in the longitudinal direction of the river channel measured by a LiDAR UAV, and the relationship between the particle size and flow regime was discussed.

2. Materials and Methods

2.1. Study Area

The study area, which is the same as our previous study [54], the Mimikawa River located in the Miyazaki Prefecture, Japan, has a length of 94.8 km and a watershed area of 884.1 km2. It has 7 dams, developed between the 1920s and the 1960s, and their purpose is only power generation, not flood control (Figure 1). However, sedimentation in the reservoirs breaks the natural transportation of solid matter along the channel [56,57], although sediment supplied from upstream areas greatly contributes to the conservation of the river channel and coastal ecosystems [58,59]. On the other hand, the southern part of Kyushu, Japan, where the basin of the Mimikawa River is located, was severely flooded as a result of a typhoon in 2005. These old dams, which did not have a sufficient discharge capacity, became obstacles during floods.
As a solution to these two disparate issues, the renovation of the dams and the lowering of the dam body to enhance the sluicing of sediment were planned [60,61]. The three dams constructed in succession downstream on the Mimikawa River were remodeled, and a function was added to the dams to remove the sediment at the time of flooding. Retrofitting of the Saigoh Dam, the second from the lower side of the Mimikawa River, was finished in 2017, and then the operation was started. Remodeling an operating dam requires advanced, world-leading techniques, and the sediment in Japanese rivers is actively monitored [62]. In the revised operation, the water level during floods was decreased to enhance the scrubbing and transportation of the settled sediment on the upstream side of the dam body. Due to those operations, sand bars, which have diverse grain sizes, were found especially on the downward section of the Saigoh Dam, in recent years.
In our previous study [54], the image classification method according to fraction categories using the CNN is only applied to the terrestrial area above the water surface on the day of photographing, and it does not include the area under the water surface where the color is affected by water absorption and wetting on the material surface or where the visibility may be affected by waves or reflections on the water surface. This study extends the target of image recognition to the shallow underwater area. Fortunately, the water was very transparent during our survey, and when there were no waves, we could see the bottom of the water, which was about 1 m deep.

2.2. Aerial Photography and Sieving for Validation

The UAV used for aerial photography was a Phantom 4 Pro, manufactured by DJI. Aerial photography from the UAV was conducted in parallel with the sieving for the volumetric method described below at 108 locations (76 locations in terrestrial areas + 32 locations in shallow waters). Pictures were taken from 10 m flight altitudes, the same as our previous study. The previous study has shown that the resolution of the pictures taken at that flight altitude is sufficient to distinguish between the smallest particle size we want to classify and the next particle size (Table 1). These are defined based on the classification criteria set by the Public Works Research Institute, Japan, from the viewpoint of river development [63], which is a modification of the Udden–Wentworth scale [64].
At the same 108 points, the particle size distribution of bed material was measured via sieving and a weight scale. The mesh sizes of the sieves were 75, 53, 37.5, 26.5, 19, 9.5, 4.75, and 2 mm (JIS Z 8801-1976) that have intervals considering the logarithmic axis of the graph showing the cumulative curve of particle size. The riverbed material in a 0.5 m × 0.5 m quadrat with a depth of around 10 cm was sampled. Sieving and weighing were carried out in the field. The riverbed material in the shallow water area was sampled by setting up the fences surrounding the target area to eliminate washouts by river currents. The collected materials were in a wet state, so after sieving them and measuring their wet weights on-site, some of them were sealed and brought back to the laboratory, where they were dried and the moisture content was determined. The dry weight was determined by subtracting the mass equivalent to the moisture content from the wet weight measured on-site.
Although the above sampling was performed in a 0.5 m × 0.5 m square using the volumetric method, in this study, the application area of the proposed photographing was set to a 1.0 m × 1.0 m square centered on the 0.5 m × 0.5 m square. This is because stones with a size of about 20 cm could be seen at some points, and the image range of a 50 cm square may not be large enough. Specifications of the camera are as follows: the image size was 5472 pixels × 364 pixels, the lens focal length was 8.8 mm, and the sensor size was 13.2 mm × 8.8 mm. When photographing at an altitude of 10 m using this camera, the image resolution was about 2.74 mm/pixel. Therefore, 365 × 365 pixels (equivalent to a 1.0 m × 1.0 m square) were trimmed from the original image.
Only the 108 data points of the particle size distribution were obtained via the volumetric method. This number was insufficient for training data for the calibration of the parameters in the CNN and for the test data for evaluating the accuracy of the CNN tuned in this study. Then, we increased the number of images available for the training and testing by analyzing the surrounding area with BASEGRAIN.
For the materials on the terrestrial area, the images of adjacent grids in 8 directions from the sieving quadrat were clipped, as in the previous study (Figure 2). If BASEGRAIN determined that the particle size distribution of the sieve quadrat image and the surrounding 8 images were approximately the same, the surrounding 8 images were nominated as the same class of particle size as that of the sieving quadrat. On the other hand, for the materials located under shallow water, the images around the quadrat area, as shown in Figure 3, were clipped because the class of the particle size was maintained in the flow direction, while the class changed drastically in the width or depth direction. The 108 images clipped from the same area as the quadrat measured via the volumetric method were used as training data, while the 563 images clipped from the surrounding area were considered to be in the same class as the image of the area of the quadrat and were used as test data. When determining the typical particle size value based on the result obtained via BASEGRAIN, we took into account the fact that BASEGRAIN has difficulty distinguishing fine particles, as in previous studies. When the number of particles recognized by BASEGRAIN is very small, the image is considered to be the finest particle class. Furthermore, considering the correlation between the results of the volumetric method and BASEGRAIN, the threshold value of BASEGRAIN between Class 1 (Medium) and Class 2 (small) was set to 28.2 mm, corresponding to the threshold value of 24.5 mm obtained via the volumetric method [54].

2.3. Training and Test of the CNN

The image recognition code was built with MATLAB®. The CNNs were installed in the code as a module. A CNN consists of input and output layers, as well as several hidden layers. The hidden part of a CNN is the combination of convolutional layers, pooling layers realizing the extraction of visual signs, and a fully connected classifier, which is a perceptron that processes the features obtained on the previous layers [65].
In our previous study [54], GoogLeNet (2014) [55], among the major networks pre-trained on ImageNet [66], showed the best performance in differentiating river sediment fractions in a terrestrial area in the images, so the following shows the results using GoogLeNet as the CNN. The networks can be retrained to perform a new task using transfer learning [67]. Finetuning a network with transfer learning is usually much faster and easier than training a network from scratch with randomly initialized weights. The features learned before transfer learning can be quickly transferred to the new task using a smaller number of training images.
In the previous study, we set 3 classes of particle size (Table 1) and tried image classification based on these classification criteria for the images of the materials in terrestrial areas. In this study, the materials in water are also subject to classification, so we are changing the number of classifications and searching for several classifications that can be classified with higher accuracy. For example, if the images of bed materials with the same particle size are divided into two conditions, whether they are in a terrestrial area or underwater, the six classes are defined by multiplying the three particle size classes listed above with the conditions.

2.4. Projection of the Classification Results and Microtopographic Survey

The UAV was automatically navigated to continuously photograph the two study sites, and each image was divided into small meshes, and the classification results for each mesh were projected on a map at the end of this study. However, the study site was in a narrow canyon, and the UAV flight altitude was relatively low to obtain a sufficient photographic resolution. As a result, depending on the flight periods, the signal acquisition from GPS satellites was insufficient, making it impossible to capture images along the planned automatic navigation route. In the second observation (11 November 2023), we were able to predict the defect in advance, so after completing the automatic navigations, we confirmed the missing area and manually flew the UAV over it to take supplementary photographs. There was a lack of photographs in the first observation (20 September 2023) as we had not noticed the canyon on the site.
When taken from an altitude of 10 m, each image captured a ground surface approximately the area of 9 m × 14 m. This was divided into 1 m meshes, and the surface condition was classified using the CNN. The position of each mesh was expressed in relative coordinates with the center of the image as the origin, and the relative coordinate values were transformed based on the UAV coordinates extracted and the yaw angle of the camera from the XMPMetadata. The results of the classification with the CNN were projected onto the background map, which was an orthophoto image based on the images taken from an flight altitude of 60 m, in addition to the one taken at an altitude of 10 m for the purpose of image classification, as mentioned above.
In addition, to discuss the relationship between the grain size map and the microtopography, the microtopography at the study sites was surveyed using LiDAR UAV (Matrice 300 RTK + Zenmuse L1 manufactured by DJI).

3. Results

3.1. Classification of Terrestrial and Underwater Samples

First, we attempted to determine the particle size from the photographs of the underwater samples using the network trained using only the samples from terrestrial areas in a previous study [54]. As previously reported, the network achieved a total accuracy of 95.4% for the terrestrial samples. Here, the underwater samples were used only as test data. The accuracy is shown as the confusion matrix in Table 2. Various confounding factors cause such a reduction in accuracy [48]. In this case, the decrease in accuracy is a natural result, because the classification of the underwater samples was performed using a network trained on terrestrial samples.
Next, we classified the images into six classes: three particle size ranges for terrestrial or underwater (two classes) and applied transfer learning to the network in an attempt to classify the test data into six classes (Table 3). Since the same number of training data and test data were used for more classes and the number of images in each class was reduced, it could not be evaluated as evenly as the previous results, but its discrimination accuracy is significantly lower. The underwater images could not be classified at all. In particular, the images of Class 1 and 2 underwater samples were classified as Class 3 with different particle sizes. Figure 4 shows examples of the images for each class. In particular, regarding the underwater and terrestrial images of Class 3, the water color was overlaid and has a bluish tinge, but both of them appeared to be similar with little shading and high brightness. On the other hand, in Class 1 and 2, the brightness decreased due to the wetness of the stone surface. Based on these visual trends, the misclassification progressed on Class 3 images, which showed a high degree of similarity between the underwater and terrestrial images, and the error rate also increased for Class 1 and 2 underwater images.

3.2. Uniform Class Only for Class 3 of the Particle Size

To prevent overfitting, we conducted training and testing using highly similar Class 3 terrestrial and underwater images as the same class. The results are shown in Table 4. For Class 3, by training the terrestrial and underwater samples as the same class without classifying them, we obtained the same classification accuracy as our previous research for Class 1 and 2 terrestrial samples, and at the same time, the classification accuracy for Class 1 and 2 underwater samples and Class 3 improved compared to Table 3.

4. Discussion

4.1. Reduction of the Error Factors using the Diversity of Training Data

Hedger et al. [53] proposed using the CNN to classify water surface waves into smooth, rippled, and standing waves and to combine them with water surface slope information to compartmentalize them into hydromorphological units (HMUs) for each mesohabitat type. The results suggested that the CNN is capable of recognizing and classifying water waves in images. In other words, in the classification of riverbed material particle size, which is the target of this research, the water surface waves overlaid on the image are also the target of feature extraction and are also a source of error in determining the bed material particle size. There are two possible ways to reduce the error in bed material classification due to the surface waves. The first one is to attempt learning and classification using twenty types in total (five classes of particle size, which have been shown in this study so far, x four types of water surface waves). However, as the number of classes increases, the amount of learning per class decreases in this research for which the total number of images available for training is limited. In addition, as seen in Table 3, classifying highly similar images into two classes reduces the accuracy. The other method aims to classify particle size into the five classes mentioned above based solely on particle size information and performs training by arranging images with different water surface wave conditions in the same class in the training data. It is supposed that this method orders the CNN to ignore waves on the water surface and to focus on the bottom bed condition for classification.
In this study, the number of classifications is limited to the five classes mentioned above, and we evaluate how the classification accuracy changes depending on whether or not the training data includes various types of water surface waves. If the cases where riverbed material could not be photographed due to blisters or white water were excluded from the surface flow type (SFT) shown by Hedger [53] and Newson et al. [68], (i) unbroken standing wave, (ii) rippled, (iii) broken standing wave in shallow water (riffles) (Figure 5) and smooth water surface remain. Here, the total accuracy of the test results for the network trained only with the images of “smooth” without waves on the water surface is the control for the following comparison. Then, the networks trained by sequentially adding images that contain waves (i) to (iii) were prepared. The same test data set as 3.2 was used for both verifications and compared in terms of total accuracy.
Figure 6 shows a comparison of the total accuracy for all five classes. When training by adding one type of each of the three types of water surface waves in (i)–(iii) (in Figure 6, these are the three on the left adjacent to the control), a significant improvement in accuracy was seen when (i) unbroken standing wave was added. If trained without the images containing unbroken standing waves, the wave pattern with a similar wavelength to the riverbed material was not recognized as a surface wave, and misclassifications occurred. Therefore, when (i) is included in the learning data, there is a strong tendency for the accuracy to be improved. Similarly, for those that include two of the three types of SFT as training data, whether or not (i) is included has a large influence. On the other hand, whether or not (iii) broken standing waves in shallow water (riffle) are included does not contribute much to improving the accuracy. Broken standing waves in shallow water is under a riffle state, where the fine material of Class 3 is never settled on the bed. In fact, among the pictures taken in this research, there was no image of Class 3 of fine bed material with the wave type of (iii). Therefore, whether or not the wave (iii) is learned does not contribute to improving the accuracy of fine material discrimination. In addition, when visually checking the image of the water surface of broken standing waves in shallow water created by the comparatively rough bed materials of Class 1 and 2, the particle diameter and wavelength are similar. So, whether or not the images of Class 1 and 2 with the wave type of (iii) are included does not have a large effect on the accuracy.
Finally, an accuracy of 90.3% was achieved when training included all three types of SFTs (i) to (iii). The total accuracy was higher than the results shown in Table 5, but this is because there were several more SFT images of (i) added to the training data, and Class 2 with underwater conditions could be discriminated more accurately. As mentioned above, many factors need to be taken into account that affect the brightness and pattern of the images taken under natural conditions. By taking these into account and including the images taken under various conditions in the learning data, it is possible to reduce this influence.

4.2. Mapping of the Wide-Ranging Area

It is possible to create a spatial distribution map of particle size by photographing the wide-ranging area using the automatic navigation system of a UAV and applying the proposed image classification. However, when the UAV was flown at a low altitude to obtain a sufficient photographic resolution to determine particle size in a canyon area with steep slopes on both sides, the GPS reception was insufficient, resulting in the absence of the coordination of images when the UAV was operated automatically. In this study, we manually photographed the missing points and rearranged the classification results on the map based on the information from the photographing angle.
Figure 7 shows an example of the shooting locations when the UAV flew automatic navigation at the study site with a 10% overlap rate between the shooting areas. To avoid the UAV crashing, an automated flight plan was created to avoid getting too close to cliffs with a lot of vegetation. The missing shooting area due to the loss of GPS signals is shown as the orange hatching area. Immediately after taking pictures using automatic navigation, we extracted the coordination of each image, checked the areas where no images due to the above reason were obtained, and then flew and took the pictures manually.
Each picture taken both automatically and manually was divided into small grids (1 m × 1 m), as shown in Figure 8, to classify using the network. The values of the local coordination on a picture with the center as the origin were given for the result of the classification for each grid. The results were projected on a map by converting the local coordinate values on the picture into the map coordinate values based on the GPS and yaw angle of each photo extracted from its XMPMetadata.
The classification using the CNN mentioned above aimed at validation, so that the classes were only focused on the particle size and the terrestrial or underwater condition. In the practical mapping, the surface of channels was not covered only with sediments. In our study site, there were vegetation parts and deep pools where the bed material was not visible. The classes “Grass” and “Deep Pool” were added to the original five classes which were applied in the previous chapter. In total, among the seven classes, the misclassification between terrestrial and underwater images for the same classes had no problem practically. So, the test result was evaluated with the confusion matrix in which the same classes of particle size under terrestrial and underwater conditions were combined into one class (Table 5). The 17 images in Class 3 (fine gravel and coarse sand) were misclassified as Class 2. The results reduced the accuracy, but generally, the scores were acceptable. Then, this network was applied to mapping the wide-ranging area in our study site.
To discuss the relationship between the distribution of bed material and hydraulic features, the topographic point clouds were obtained via a UAV (Matrice-300-RTK: DJI Japan, Japan, Tokyo) mounted with an LiDAR (Zenmuse-L1: DJI Japan, Japan, Tokyo). A LiDAR UAV can acquire point clouds at an altitude of 100 m, with an accuracy of 3 cm, but that accuracy is insufficient to distinguish the bed material particle on the study site. The scanning mode of the LiDAR was a repeat scanning (repeat) pattern, and the return mode was triple, to acquire the topography under the trees. The flight route was set at an altitude of about 80 m, with one survey line in the middle of the river channel. The topographic surveys were conducted only on 11 November 2023 due to the procurement of the apparatus while the sediment particle size was monitored on 4 September and 11 November 2023, before and after Typhoon No.14 on 20 September 2023.
Figure 9 and Figure 10 show the spatial distribution map of riverbed grain size and the longitudinal cross-sectional diagram created from point cloud data for Site 1 and Site 2, respectively. The spatial distribution map of particle size shows particle size, water, and plants as points in five colors, and the water edges of both sides at the time when the observation was made on a low water day are shown as a black line. The flow direction is shown by the light blue arrow below. In the vertical profile, the vertical axis represents the altitude and the horizontal axis represents the distance along the shorelines from the upper border of the measurements. Also, the numbers on the longitudinal map and the numbers on the spatial distribution map indicate the same points.
Upstream of the bend in the channel in Site 1 (around the center in Figure 9b), Class 3 of the fine sediment was widely distributed within the low water channel after the flood. When the flow channel gradually narrowed to the left bank during the decreasing phase of the flood, coarse sediment accumulated on the right bank on the inside of the bend first. As the water level was decreasing, relatively large grains were deposited in front of the bend, forming a riffle (Section 3, Section 4 and Section 5). The upstream side was dammed up and became a slow-flowing area, and the fine-grained sediment transported during the period of declining water flow was widely deposited on the flow path. Seeing the longitudinal profile, the right bank channel of the two branched low channels has a steep slope between 3 and 5, and it could be confirmed that a relatively large grain-size sediment was deposited in that section. In addition, the grain size of the sediment along the left bank channel was uniform. In the longitudinal section, the left bank side showed an almost uniform slope without pools and falls, and the channel width was also almost uniform. This slope was consistent with the absence of fine sediment deposition. The topography before the flood was not obtained. However, the rapid accumulation of coarse material at the bend and the fine sediment accumulation on its upper side were also seen in the upper half of the distribution map before the flood.
Comparing the distribution maps before and after the typhoon, the deep pool near the downstream end of the observation area shrunk, and the fine-grained sediment on the right bank side of the deep pool disappeared and was replaced by Class 2 sediment. Before the typhoon, the area near the downstream end was dammed up due to the accumulation of sediment outside the photographed area, and due to the slow flow of the deep pool, fine-grained sediment was deposited. It is presumed that the riffles outside the area were washed away by the flood, and the accumulation disappeared, resulting in the accumulation of Class 2 sediment. Looking at the longitudinal profile after the typhoon, the downstream part of the photographed area shows an almost uniform slope.
At Site 2, which is a point further from the dam, Class 2 and 3 are distributed above the low water channel on the right bank downstream in both of the distribution maps before and after the typhoon. The reason for this is different from the fine-grained sediment deposited in the slow basin that was dammed behind the small riffles at Site 1. Looking at the longitudinal profile after the typhoon, it can be seen that the slope between 4 to 7 is gentle generally, but there is an alternation of the course (Class 1) and the fine (Class 3) sediments. The channel of Site 2 is wider than Site 1 and other upstream areas, resulting in lower flow velocity and accumulation.
Comparing the distribution maps before and after the flood, it is found that, before the flood, Class 2 was deposited downstream and there was no branch on the right bank on the low water days. On the other hand, after the flood, the branching channel appeared on the right bank and its bed is an alternation of Class 1 and 3. In particular, the range of Class 1 (coarse) was extended. This is because the sluicing operations had been carried out before 2023, but only on the Saigoh dam. In 2023, the Yamasubaru dam also started sediment sluicing after completing its retrofitting. So, there is a possibility that the larger grain-sized particles were transported from the upper dam.

5. Conclusions

In this study, the particle size classification method using a CNN with UAV aerial images, which has so far targeted only terrestrial bed materials in our previous study, was carried out to classify materials underwater in shallow waters where the water bottom can be seen. When completely ignoring whether the sample was terrestrial or underwater, we could predict that the classification accuracy would drop significantly, even without the trial. The accuracy was further reduced by trying to classify similar images into different classes by doubling the number of classes. If the image similarity between the classes was high, increasing the number of classes was not a good idea. We were able to improve accuracy by using the same class, especially for fine grains that have high similarity in terrestrial and underwater conditions. On the other hand, if the particles were large enough and could be identified from the image, the colors were different between terrestrial and underwater, so classifying them into the other would have resulted in higher classification accuracy.
In addition, the waves on the water surface were thought to be a factor causing the errors, but increasing the number of classes was not considered a good idea, so we did not add classes accordingly. Since the purpose of this research is to classify riverbed images based on particle size, we were able to obtain sufficient accuracy by focusing only on the particle size and setting classes without considering the difference in wave types. In particular, it was shown that accuracy could be improved by including a variety of SFTs in the training data.
This suggests that classification based on the main target (particle size in this study), without being distracted by other environmental factors that may cause changes in the image, provides a certain level of accuracy. This is not limited to SFT. Furthermore, if the training data include images taken in as diverse conditions as possible concerning other environmental factors, the biases caused by the other factors will be reduced and higher accuracy will be obtained. In this study, photographs were taken in two relatively close areas of the river, so the environmental conditions other than waves on the water surface were relatively similar. Therefore, to make the method applicable to many other rivers generally, it is necessary to photograph under various environmental conditions with different flow rates, current speeds, parent rocks, water color, particle shape characteristics, and riverbed gradients, etc. It is preferable to prepare more training data that have been verified.
In addition, the validity of this method was supported by the fact that there was a reasonable relationship between the gradient of the microtopography observed via LiDAR and the trend of the wide-area distribution of particle size. Images of river channels taken from UAVs are the result of a combination of various environmental information. It is difficult to classify when considering multiple types of environmental conditions at the same time. However, if you simply classify by focusing on one item and provide training data without considering other environmental items, sufficient accuracy can be obtained. This is true not only for classification based on riverbed particle size but also for creating environmental classification maps based on image classification.

Author Contributions

Conceptualization, M.I.; methodology, S.A., T.S. and M.I.; software, S.A., T.S. and M.I.; validation, S.A., T.S., T.U. and M.I.; formal analysis, S.A., T.S. and M.I.; investigation, M.I.; resources, M.I.; data curation, S.A., T.S., T.U. and M.I.; writing—original draft preparation, M.I.; writing—review and editing, M.I.; visualization, S.A. and T.S.; supervision, M.I.; project administration, M.I.; funding acquisition, M.I. All authors have read and agreed to the published version of the manuscript.

Funding

This study was funded by JSPS KAKENHI Grant Number 17H03314.

Data Availability Statement

Data are contained within the article.

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. David, A.J.; Castillo, M.M. Stream Ecology: Structure and Function of Running Waters, 2nd ed.; Springer: Dordrecht, The Netherlands, 2007; pp. 1–436. [Google Scholar]
  2. Alexandre, C.M.; Ferreira, T.F.; Almeida, P.R. Fish assemblages in non-regulated and regulated rivers from permanent and temporary Iberian systems. River Res. Appl. 2013, 29, 1042–1058. [Google Scholar] [CrossRef]
  3. Almeida, D.; Merino-Aguirre, R.; Angeler, D.G. Benthic invertebrate communities in regulated Mediterranean streams and least-impacted tributaries. Limnologica 2013, 43, 34–42. [Google Scholar] [CrossRef]
  4. Eekhout, J.P.C.; Hoitink, A.J.F. Chute cutoff as a morphological response to stream reconstruction: The possible role of backwater. Water Resour. Res. 2015, 51, 3339–3352. [Google Scholar] [CrossRef]
  5. Tsubaki, R.; Baranya, S.; Muste, M.; Toda, Y. Spatio-temporal patterns of sediment particle movement on 2D and 3D bedforms. Exp. Fluids 2018, 59, 93. [Google Scholar] [CrossRef]
  6. Liu, L. Drone-based photogrammetry for riverbed characteristics extraction and flood discharge modeling in Taiwan’s mountainous rivers. Measurement 2023, 220, 113386. [Google Scholar] [CrossRef]
  7. Zhu, R.; Tsubaki, R.; Toda, Y. Effects of vegetation distribution along river transects on the morphology of a gravel bed braided river. Acta Geophys. 2023. online first. [Google Scholar] [CrossRef]
  8. Kang, T.; Kimura, I.; Shimizu, Y. Responses of bed morphology to vegetation growth and flood discharge at a sharp river bend. Water 2018, 10, 223. [Google Scholar] [CrossRef]
  9. Bunte, K.; Abt, S.R. Sampling Surface and Subsurface Particle-Size Distributions in Wadable Gravel- and Cobble-Bed Streams for Analyses in Sediment Transport, Hydraulics, and Streambed Monitoring; General Technical Report RMRS-GTR-74; U. S. Department of Agriculture: Washington, DC, USA, 2001; pp. 166–170. [Google Scholar] [CrossRef]
  10. Kellerhals, R.; Bray, D. Sampling procedure for Coarse Fluvial Sediments. J. Hydraul. Div. ASCE 1971, 97, 1165–1180. [Google Scholar] [CrossRef]
  11. Detert, M.; Weitbrecht, V. Automatic object detection to analyze the geometry of gravel grains—A free stand-alone tool. In River Flow 2012; Muños, R.M., Ed.; Taylor & Francis Group: London, UK, 2012; pp. 595–600. [Google Scholar]
  12. Detert, M.; Weitbrecht, V. User guide to gravelometric image analysis by BASEGRAIN. In Advances in River Sediment Research; Taylor & Francis Group: London, UK, 2013; pp. 1789–1796. [Google Scholar]
  13. Harada, M.; Arakawa, T.; Ooi, T.; Suzuki, H.; Sawada, K. Development of bed topography survey technique by underwater imaging progress for UAV photogrammetry. Proc. River Eng. 2016, 22, 67–72. (In Japanese) [Google Scholar]
  14. Hirao, S.; Azumi, T.; Yoshimura, M.; Nishiguchi, Y.; Kawai, S. Fundamental study on grain size distribution of river bed surface by analyzing UAV photograph. Proc. River Eng. 2018, 24, 263–266. (In Japanese) [Google Scholar] [CrossRef]
  15. Rogan, J.; Chen, D. Remote sensing technology for mapping, and monitoring land-cover and land-use change. Prog. Plan. 2004, 61, 301–325. [Google Scholar] [CrossRef]
  16. Mochizuki, S.; Murakami, T. Vegetation map using the object-oriented image classification with ensemble learning. J. Forest. Plan. 2013, 18, 127–134. [Google Scholar] [CrossRef] [PubMed]
  17. Mtibaa, S.; Irie, M. Land cover mapping in cropland dominated area using information on vegetation phenology and multi-seasonal Landsat 8 images. Euro-Mediterr. J. Environ. Integr. 2016, 1, 6. [Google Scholar] [CrossRef]
  18. Maurya, K.; Mahajan, S.; Chaube, N. Remote sensing techniques: Mapping and monitoring of mangrove ecosystem—A review. Complex Intell. Syst. 2021, 7, 2797–2818. [Google Scholar] [CrossRef]
  19. Gilcher, M.; Udelhoven, T. Field Geometry and the Spatial and Temporal Generalization of Crop Classification Algorithms—A randomized approach to compare pixel based and convolution based methods. Remote Sens. 2021, 13, 775. [Google Scholar] [CrossRef]
  20. Taravat, A.; Wagner, M.P.; Bonifacio, R.; Petit, D. Advanced Fully Convolutional Networks for Agricultural Field Boundary Detection. Remote Sens. 2021, 13, 722. [Google Scholar] [CrossRef]
  21. Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 3431–3440. [Google Scholar] [CrossRef]
  22. DeepGlobe CVPR 2018—Satellite Challenge. Available online: https://deepglobe.org (accessed on 10 October 2023).
  23. Golovanov, S.; Kurbanov, R.; Artamonov, A.; Davydow, A.; Nikolenko, S. Building Detection from Satellite Imagery Using a Composite Loss Function. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 229–232. [Google Scholar] [CrossRef]
  24. Dickenson, M.; Gueguen, L. Rotated Rectangles for Symbolized Building Footprint Extraction. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 215–2153. [Google Scholar] [CrossRef]
  25. Iglovikov, V.I.; Seferbekov, S.; Buslaev, A.V.; Shvets, A. TernausNetV2: Fully Convolutional Network for Instance Segmentation. arXiv 2018, arXiv:1806.00844. [Google Scholar]
  26. Buslaev, A.; Seferbekov, S.; Iglovikov, V.; Shvets, A. Fully Convolutional Network for Automatic Road Extraction from Satellite Imagery. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 1973–1977. [Google Scholar] [CrossRef]
  27. Costea, D.; Marcu, A.; Slusanschi, E.; Leordeanu, M. Roadmap Generation using a Multi-stage Ensemble of Deep Neural Networks with Smoothing-Based Optimization. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 220–224. [Google Scholar] [CrossRef]
  28. Doshi, J. Residual Inception Skip Network for Binary Segmentation. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 216–219. [Google Scholar] [CrossRef]
  29. Ghosh, A.; Ehrlich, M.; Shah, S.; Davis, L.; Chellappa, R. Stacked U-Nets for Ground Material Segmentation in Remote Sensing Imagery. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 257–261. [Google Scholar] [CrossRef]
  30. Samy, M.; Amer, K.; Eissa, K.; Shaker, M.; ElHelw, M. NU-Net: Deep Residual Wide Field of View Convolutional Neural Network for Semantic Segmentation. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 267–271. [Google Scholar] [CrossRef]
  31. Tian, C.; Li, C.; Shi, J. Dense Fusion Classmate Network for Land Cover Classification. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Salt Lake City, UT, USA, 18–23 June 2018; pp. 192–196. [Google Scholar] [CrossRef]
  32. Verma, U.; Chauhan, A.; Pai M, M.; Pai, R. DeepRivWidth: Deep learning based semantic segmentation approach for river identification and width measurement in SAR images of Coastal Karnataka. Comput. Geosci. 2021, 154, 104805. [Google Scholar] [CrossRef]
  33. Ling, F.; Boyd, D.; Ge, Y.; Foody, G.M.; Li, X.; Wang, L.; Zhang, Y.; Shi, L.; Shang, C.; Li, X.; et al. Measuring River Wetted Width from Remotely Sensed Imagery at the Subpixel Scale with a Deep Convolutional Neural Network. Water Resour Res. 2019, 55, 5631–5649. [Google Scholar] [CrossRef]
  34. Nama, A.H.; Abbas, A.S.; Maatooq, J.S. Field and Satellite Images-Based Investigation of Rivers Morphological Aspects. Civ. Eng. J. 2022, 8, 1339–1357. [Google Scholar] [CrossRef]
  35. Yamawaki, M.; Matsui, T.; Kawahara, M.; Yasumoto, Y.; Ueyama, K.; Kawazoe, Y.; Matsuda, K.; Hara, F. Study on enhancement of sandbar monitoring by deep learning -towards enhancement of management in Hojo-river. J. Jp. Soc. Civ. Eng. Ser. B2 2021, 77, I_511–I_516. (In Japanese) [Google Scholar] [CrossRef]
  36. Akiyama, T.S.; Marcato Junior, J.; Gonçalves, W.N.; Bressan, P.O.; Eltner, A.; Binder, F.; Singer, T. Deep learning applied to water segmentation. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. 2020, XLIII-B2-2020, 1189–1193. [Google Scholar] [CrossRef]
  37. Muhadi, N.A.; Abdullah, A.F.; Bejo, S.K.; Mahadi, M.R.; Mijic, A. Deep Learning Semantic Segmentation for Water Level Estimation Using Surveillance Camera. Appl. Sci. 2020, 11, 9691. [Google Scholar] [CrossRef]
  38. Lopez-Fuentes, L.; Rossi, C.; Skinnemoen, H. River segmentation for flood monitoring. In Proceedings of the 2017 IEEE International Conference on Big Data (Big Data), Boston, MA, USA, 11–14 December 2017; pp. 3746–3749. [Google Scholar] [CrossRef]
  39. Inoue, H.; Katayama, T.; Song, T.; Shimamoto, T. Semantic Segmentation of River Video for Efficient River Surveillance System. In Proceedings of the 2023 International Technical Conference on Circuits/Systems, Computers, and Communications (ITC-CSCC), Jeju, Republic of Korea, 25–28 June 2023; pp. 1–5. [Google Scholar] [CrossRef]
  40. Lambert, R.; Li, J.; Chavez-Galaviz, J.; Mahmoudian, N. A Survey on the Deployability of Semantic Segmentation Networks for Fluvial Navigation. In Proceedings of the 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), Waikoloa, HI, USA, 3–7 January 2023; pp. 255–264. [Google Scholar] [CrossRef]
  41. Zhang, X.; Zhou, Y.; Jin, J.; Wang, Y.; Fan, M.; Wang, N.; Zhang, Y. ICENETv2: A Fine-Grained River Ice Semantic Segmentation Network Based on UAV Images. Remote Sens. 2020, 13, 633. [Google Scholar] [CrossRef]
  42. Ren, B.; Pan, Y.; Lin, X.; Yang, K. Statistical Roughness Properties of the Bed Surface in Braided Rivers. Water 2022, 15, 2612. [Google Scholar] [CrossRef]
  43. Piton, G.; Recking, A.; Le Coz, J.; Bellot, H.; Hauet, A.; Jodeau, M. Reconstructing depth-averaged open-channel flows using image velocimetry and photogrammetry. Water Resour. Res. 2018, 54, 4164–4179. [Google Scholar] [CrossRef]
  44. Brasington, J.; Vericat, D.; Rychkov, I. Modeling river bed morphology, roughness, and surface sedimentology using high resolution terrestrial laser scanning. Water Resour. Res. 2012, 48, W11519. [Google Scholar] [CrossRef]
  45. Guenther, G.C.; Brooks, M.W.; Larocque, P.E. New capabilities of the ‘SHOALS’ airborne lidar bathymeter. Remote Sens. Environ. 2000, 73, 247–255. [Google Scholar] [CrossRef]
  46. Onaka, N.; Akamatsu, Y.; Mabu, S.; Inui, R.; Hanaoka, T. Development of a habitat prediction method for ruditapes philippinarum based on image analysis using deep learning. J. Jp. Soc. Civ. Eng. Ser. B1 2020, 76, I_1279–I_1284. [Google Scholar] [CrossRef]
  47. Onaka, N.; Akamatsu, Y.; Koyama, A.; Inui, R.; Saito, M.; Mabu, S. Development of a sediment particle size prediction method for tidal flat based on image analysis using deep learning. J. Jp. Soc. Civ. Eng. Ser. B1 2022, 78, I_1117–I_1122. [Google Scholar] [CrossRef]
  48. Hedger, R.D.; Sundt-Hansen, L.; Foldvik, A. Evaluating the Suitability of Aerial Photo Surveys for Assessing Atlantic Salmon Habitat in Norway. NINA Report 2105. 2022. Available online: https://brage.nina.no/nina-xmlui/bitstream/handle/11250/2975990/ninarapport2105.pdf?sequence=5&isAllowed=y (accessed on 10 October 2023).
  49. Carbonneau, P.E.; Dugdale, S.J.; Breckon, T.P.; Dietrich, J.T.; Fonstad, M.A.; Miyamoto, H.; Woodget, A.S. Adopting deep learning methods for airborne RGB fluvial scene classification. Remote Sens. Environ. 2020, 251, 112107. [Google Scholar] [CrossRef]
  50. Casado, M.R.; Gonzalez, R.B.; Kriechbaumer, T.; Veal, A. Automated identification of river hydromorphological features using UAV high resolution aerial imagery. Sensors 2015, 15, 27969–27989. [Google Scholar] [CrossRef]
  51. Milan, D.J.; Heritage, G.L.; Large, A.R.G.; Entwistle, N.S. Mapping hydraulic biotopes using terrestrial laser scan data of water surface properties. Earth Surf. Process. Landf. 2010, 35, 918–931. [Google Scholar] [CrossRef]
  52. Harrison, L.R.; Legleiter, C.J.; Overstreet, B.T.; Bell, T.W.; Hannon, J. Assessing the potential for spectrally based remote sensing of salmon spawning locations. River Res. Appl. 2020, 36, 1618–1632. [Google Scholar] [CrossRef]
  53. Hedger, R.D.; Gosselin, P. Automated fluvial hydromorphology mapping from airborne remote sensing. River Res. Appl. 2023, 39, 1889–1901. [Google Scholar] [CrossRef]
  54. Takechi, H.; Aragaki, S.; Irie, M. Differentiation of River Sediments Fractions in UAV Aerial Images by Convolution Neural Network. Remote Sens. 2020, 13, 3188. [Google Scholar] [CrossRef]
  55. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vanhoucke, V.; Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 1–9. [Google Scholar] [CrossRef]
  56. Nukazawa, K.; Shirasaka, K.; Kajiwara, S.; Saito, T.; Irie, M.; Suzuki, Y. Gradients of flow regulation shape community structures of stream fishes and insects within a catchment subject to typhoon events. Sci. Total Environ. 2020, 748, 141398. [Google Scholar] [CrossRef]
  57. Nakano, D.; Nakane, Y.; Kajiwara, S.; Sakada, K.; Nishimura, K.; Fukaike, M.; Honjo, T. Macrozoobenthos distribution after flood events offshore the Mimi River estuary, Japan. Plankton Benthos Res. 2022, 17, 277–289. [Google Scholar] [CrossRef]
  58. Ito, K.; Matsunaga, M.; Itakiyo, T.; Oishi, H.; Nukazawa, K.; Irie, M.; Suzuki, Y. Tracing sediment transport history using mineralogical fingerprinting in a river basin with dams utilizing sediment sluicing. Intl. J. Sediment Res. 2022, 38, 469–480. [Google Scholar] [CrossRef]
  59. Nukazawa, K.; Kajiwara, S.; Saito, T.; Suzuki, Y. Preliminary assessment of the impacts of sediment sluicing events on stream insects in the Mimi River. Jp Ecol. Eng. 2020, 145, 105726. [Google Scholar] [CrossRef]
  60. Sumi, T.; Yoshimura, T.; Asazaki, K.; Kaku, M.; Kashiwai, J.; Sato, T. Retrofitting and change in operation of cascade dams to facilitate sediment sluicing in the Mimikawa river basin. In Proceedings of the 25th Congress of International Commission on Large Dams, Stavanger, Norway, 14–20 June 2015; Volume Q99-R45, pp. 597–616. [Google Scholar]
  61. Yoshimura, T.; Shinya, H. Environmental impact assessment plan due to sediment sluicing at dams along Mimikawa river system. J. Disaster Res. 2018, 13, 709–719. [Google Scholar] [CrossRef]
  62. Landwehr, T.; Kantoush, S.A.; Pahl-Wostl, C.; Sumi, T.; Irie, M. The effect of optimism bias and governmental action on siltation management within Japanese reservoirs surveyed via artificial neural network. Big Earth Data 2020, 4, 68–89. [Google Scholar] [CrossRef]
  63. Technical Standards for River Erosion Control—Research Version-4. Available online: https://www.mlit.go.jp/river/shishin_guideline/gijutsu/gijutsukijunn/chousa/ (accessed on 15 December 2023).
  64. Wentworth, C.K. A scale of grade and class terms for clastic sediments. J. Geol. 1922, 30, 377–392. Available online: https://www.jstor.org/stable/30063207 (accessed on 15 December 2023). [CrossRef]
  65. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS’12), Lake Tahoe, NV, USA, 3–6 December 2012; pp. 1097–1105. [Google Scholar]
  66. ImageNet Website. Available online: http://image-net.org/ (accessed on 28 September 2023).
  67. Zamir, A.R.; Sax, A.; Shen, W.; Guibas, L.; Malik, J.; Savarese, S. Taskonomy: Disentangling Task Transfer Learning. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3712–3722. [Google Scholar] [CrossRef]
  68. Newson, M.D.; Newson, C.L. Geomorphology, ecology and river channel habitat: Mesoscale approaches to basin-scale challenges. Prog. Phys. Geogr. 2000, 24, 195–217. [Google Scholar] [CrossRef]
Remotesensing 16 00173 g001
Figure 1. Location of the study site on the Mimikawa River in Miyazaki, Japan.
Figure 1. Location of the study site on the Mimikawa River in Miyazaki, Japan.
Remotesensing 16 00173 g001
Remotesensing 16 00173 g002
Figure 2. Image preprocessing and how to increase the image data for the image for the terrestrial area. (Red: volumetric method and BASEGRAIN; yellow: BASEGRAIN only).
Figure 2. Image preprocessing and how to increase the image data for the image for the terrestrial area. (Red: volumetric method and BASEGRAIN; yellow: BASEGRAIN only).
Remotesensing 16 00173 g002
Remotesensing 16 00173 g003
Figure 3. Image preprocessing and how to increase the image data for the image in shallow water. (Red: volumetric method and BASEGRAIN; yellow: BASEGRAIN only).
Figure 3. Image preprocessing and how to increase the image data for the image in shallow water. (Red: volumetric method and BASEGRAIN; yellow: BASEGRAIN only).
Remotesensing 16 00173 g003
Remotesensing 16 00173 g004
Figure 4. The trimmed image samples (1 m × 1 m) of the three classes of particle size and terrestrial/underwater conditions.
Figure 4. The trimmed image samples (1 m × 1 m) of the three classes of particle size and terrestrial/underwater conditions.
Remotesensing 16 00173 g004
Remotesensing 16 00173 g005
Figure 5. Trimmed image examples (1 m × 1 m) overlaying surface flow type (SFT) on the bed materials.
Figure 5. Trimmed image examples (1 m × 1 m) overlaying surface flow type (SFT) on the bed materials.
Remotesensing 16 00173 g005
Remotesensing 16 00173 g006
Figure 6. Comparison of the total accuracies between the learning cases with the different wave type.
Figure 6. Comparison of the total accuracies between the learning cases with the different wave type.
Remotesensing 16 00173 g006
Remotesensing 16 00173 g007
Figure 7. Shooting points via an automatic flight plan.
Figure 7. Shooting points via an automatic flight plan.
Remotesensing 16 00173 g007
Remotesensing 16 00173 g008
Figure 8. Process for mapping the results of image classification using CNN: the images in each grid are classified and the results are reprojected onto a map based on the yaw angle and GPS coordinates attached to the original image.
Figure 8. Process for mapping the results of image classification using CNN: the images in each grid are classified and the results are reprojected onto a map based on the yaw angle and GPS coordinates attached to the original image.
Remotesensing 16 00173 g008
Remotesensing 16 00173 g009
Figure 9. Distribution map of grain size (a) before the sluicing, (b) after the sluicing, and (c) longitudinal cross-section after the sluicing of Site 1.
Figure 9. Distribution map of grain size (a) before the sluicing, (b) after the sluicing, and (c) longitudinal cross-section after the sluicing of Site 1.
Remotesensing 16 00173 g009
Remotesensing 16 00173 g010
Figure 10. Distribution map of grain size (a) before the sluicing, (b) after the sluicing, and (c) longitudinal cross-section after the sluicing of Site 2.
Figure 10. Distribution map of grain size (a) before the sluicing, (b) after the sluicing, and (c) longitudinal cross-section after the sluicing of Site 2.
Remotesensing 16 00173 g010
Table 1. Classification criteria in this study [63].
Table 1. Classification criteria in this study [63].
ClassificationParticle Size (mm)Classification Name
in This Study
Medium gravel64–24.5Class 1
Small gravel24.5–2Class 2
Fine gravel and coarse sand2>Class 3
Table 2. Confusion matrix of the results classifying all the images with the network trained only with terrestrial sample images (batch size 10, epoch 21).
Table 2. Confusion matrix of the results classifying all the images with the network trained only with terrestrial sample images (batch size 10, epoch 21).
Class 1Class 2Class 3RecallF-Score
Class 1678089.3%86.4%
Class 213581865.2%70.3%
Class 30102268.8%61.1%
Precision83.8%76.3%55.0%
Micro Prec.75.0%
Macro Prec.71.7%
Micro Recall75.0%
Macro Recall74.4%
Overall Acc.75.0%
Average Acc.71.7%
Table 3. Confusion matrix with the six classes: three particle size classes and terrestrial/underwater conditions (batch size 10, epoch 21).
Table 3. Confusion matrix with the six classes: three particle size classes and terrestrial/underwater conditions (batch size 10, epoch 21).
TerrestrialUnderwaterRecallF-Score
Class 1Class 2Class 3Class 1Class 2Class 3
Class 15600000100.0%62.9%
Class 216122101285.9%92.4%
Class 35005880651921.3%35.0%
Class 1000000--
Class 2000000--
Class 3000061976.0%58.5%
Precision45.9%100.0%98.3%0.0%0.0%47.5%
Micro Prec.51.5%
Macro Prec.48.6%
Micro Recall51.5%
Macro Recall69.1%
Overall Acc.51.5%
Average Acc.48.6%
Table 4. Confusion matrix with the five classes: Class 1 and 2 of particle size x terrestrial/underwater conditions and Class 3 (batch size 10, epoch 21).
Table 4. Confusion matrix with the five classes: Class 1 and 2 of particle size x terrestrial/underwater conditions and Class 3 (batch size 10, epoch 21).
TerrestrialUnderwaterBothRecallF-Score
Class 1Class 2Class 1Class 2Class 3
Class 1113601094.2%93.3%
Class 2511502392.0%93.1%
Class 14178181169.6%81.2%
Class 2001481278.7%68.1%
Class 3001119488.7%83.1%
Precision92.6%94.3%97.5%60.0%78.3%
Micro Prec.85.5%
Macro Prec.84.5%
Micro Recall85.5%
Macro Recall84.5%
Overall Acc.85.5%
Average Acc.84.6%
Table 5. Confusion matrix classifying with the three classes of the particle, deep pool and grass (batch size 26, epoch 21).
Table 5. Confusion matrix classifying with the three classes of the particle, deep pool and grass (batch size 26, epoch 21).
Class 1Class 2Class 3Deep PoolGrassRecallF-Score
Class 1134100099.3%93.7%
Class 21614411088.9%88.3%
Class 3117562073.7%81.2%
Deep Pool02573091.3%93.6%
Grass000080100%100%
Precision88.7%87.8%90.3%96.1%100%
Overall Acc.91.3%
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Irie, M.; Arakaki, S.; Suto, T.; Umino, T. Classification of River Sediment Fractions in a River Segment including Shallow Water Areas Based on Aerial Images from Unmanned Aerial Vehicles with Convolution Neural Networks. Remote Sens. 2024, 16, 173. https://doi.org/10.3390/rs16010173

AMA Style

Irie M, Arakaki S, Suto T, Umino T. Classification of River Sediment Fractions in a River Segment including Shallow Water Areas Based on Aerial Images from Unmanned Aerial Vehicles with Convolution Neural Networks. Remote Sensing. 2024; 16(1):173. https://doi.org/10.3390/rs16010173

Chicago/Turabian Style

Irie, Mitsuteru, Shunsuke Arakaki, Tomoki Suto, and Takuto Umino. 2024. "Classification of River Sediment Fractions in a River Segment including Shallow Water Areas Based on Aerial Images from Unmanned Aerial Vehicles with Convolution Neural Networks" Remote Sensing 16, no. 1: 173. https://doi.org/10.3390/rs16010173

APA Style

Irie, M., Arakaki, S., Suto, T., & Umino, T. (2024). Classification of River Sediment Fractions in a River Segment including Shallow Water Areas Based on Aerial Images from Unmanned Aerial Vehicles with Convolution Neural Networks. Remote Sensing, 16(1), 173. https://doi.org/10.3390/rs16010173

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop