Image Processing of UAV Imagery for River Feature Recognition of Kerian River, Malaysia

Abstract

The impact of floods is the most severe among the natural calamities occurring in Malaysia. The knock of floods is consistent and annually forces thousands of Malaysians to relocate. The lack of information from the Ministry of Environment and Water, Malaysia is the foremost obstacle in upgrading the flood mapping. With the expeditious evolution of computer techniques, processing of satellite and unmanned aerial vehicle (UAV) images for river hydromorphological feature detection and flood management have gathered pace in the last two decades. Different image processing algorithms—structure from motion (SfM), multi-view stereo (MVS), gradient vector flow (GVF) snake algorithm, etc.—and artificial neural networks are implemented for the monitoring and classification of river features. This paper presents the application of the k-means algorithm along with image thresholding to quantify variation in river surface flow areas and vegetation growth along Kerian River, Malaysia. The river characteristic recognition directly or indirectly assists in studying river behavior and flood monitoring. Dice similarity coefficient and Jaccard index are numerated between thresholded images that are clustered using the k-means algorithm and manually segmented images. Based on quantitative evaluation, a dice similarity coefficient and Jaccard index of up to 97.86% and 94.36% were yielded for flow area and vegetation calculation. Thus, the present technique is functional in evaluating river characteristics with reduced errors. With minimum errors, the present technique can be utilized for quantifying agricultural areas and urban areas around the river basin.

1. Introduction

Floods have been the most damaging disasters in Malaysia for a long time. The rise in sea level due to climate change, population growth, and abrupt urbanization have given rise to frequent flood disasters in many countries around the globe. Those living near the seashore and rivers are the most affected due to floods, which often force them to rehabilitate and disturbs their life. One of the factors that leads to degradation of the environment and represents a principal cause of floods in Malaysia is the fact that human activities, including the rapid development of densely populated flood plains, destruction of trees, and encroachment on flood-prone areas for development are viewed only as a positive development, turning a blind eye to their negative aspects [1]. Floods constitute 49% of the total disasters faced by Malaysia post-independence [2]. The consequences of floods in Malaysia include loss of human and animal life, destruction of infrastructures, and crumbling health conditions. In Malaysia, the National Security Council is responsible for flood management by implementing new techniques for flood control, forecasting, warning, and eviction, thus handling pre-disaster, disaster, and post-disaster activities [3]. The Malaysian National Security Council still requires additional overview of floods by researchers and government agencies to assist in decision making [4]. Researchers are finally coming up with community-based flood mitigation policies for inhabitants of towns located in the vicinity of different rivers that are prone to floods [5].

The backwash of floods disturbing the lives of Malaysians has led to more researchers and government agencies escalating the aim of developing enhanced and upgraded flood tracking systems. Quickly predicting the temporal level rise of river water and spatial mapping of flood consequences forms the core of flood monitoring systems. Early detection of flood after-effects and timely measures being taken by government agencies and disaster relief teams in advance can save many lives and protect resources from extreme destruction. All these factors are directly dependent on the availability of suitable data for developing a well-grounded flood monitoring system.

Various researchers have applied different techniques to analyze remotely sensed data for flood mapping that are captured by satellite platforms or UAV high-resolution imagery. Annis et al. [6] compared the digital elevation model (DEM) produced by data obtained using UAVs and light detection and ranging (LiDAR) satellites. Their results, which were determined via quantitative analyses, proved that DEMs acquired using UAVs have very high resolution along with a low surveying cost and require less time compared to DEMs procured by field visits or LiDAR satellites. Tamminga et al. [7] used small drones to investigate 3D dynamic changes due to floods on the Elbow River, Canada. Due to floods, a sediment flux mobility of a complex nature was confirmed in the study. UAVs and satellite images are also useful for examining riparian vegetation along the river basin [8]. Another advantage of UAVs is that they fill the gap between space-borne and field data by quickly providing high spatial resolution data [9]. This edge of UAVs over data acquired through other means helps in measuring river water levels with a better overall accuracy using a canny method [10] or DEM generation [11]. For better validation and calibration of flood models, georeferenced information regarding river shorelines, river geometry, and vegetation along the river basin attained using UAVs provides high temporal resolution, which is limited in the case of satellite images [12]. Post-flood surveys of high-water marks and river cross-sections using small, unmanned aircrafts have proven to be more accurate and effective compared to ground surveys [13]. Ephemeral rivers flowing in ungauged regions are essential for ecosystem balance and runoff. The techniques for measuring the peak discharge of these transient rivers are limited. UAV technology merged with the incipient motion of stones is useful for calculating the peak discharges of such short-lived rivers [14]. Flash floods occurring in ephemeral streams have a direct relationship with human activities, and the magnitude of destruction caused by flash floods is better calibrated by combining field survey data with UAV observations [15]. From the existing literature, it is evident that UAV imagery has a lead over satellite images and field survey data due to its ability to provide high temporal resolution information in less time with a low overall cost. Apart from this, high-resolution satellite images are costly and can sometimes be limited due to weather conditions, whereas field surveys take a long time. Therefore, the present work includes river images captured using UAVs for river feature recognition. Satellite data are also used in the present work for tertiary information, such as weather forecasting, crucial site locating, and river mapping. An amalgamation of UAV technology and satellite images is employed to collect reliable results.

Implementation of image processing algorithms for flood mapping has improved the reliability of results. Although there is no common solution for accurate flood monitoring, image processing has been proven to be a most reliable and widely used technique for analyzing digital images of river ecosystems [16]. Supervised image analysis consisting of classical image processing techniques such as multispectral classification along with pixel aggregation and shadow treatment is useful for analyzing flooded riverbeds [17]. Digital image processing is also functional as regards historical archives of river imagery to capture features of interest [18] as well as for LiDAR images [19]. Image processing of webcam images utilizing Raspberry-Pi and optical sensors is handy for flood detection [20]. Image processing algorithms are convenient for flood mapping as they can analyze terrestrial features and satellite images [21,22]. Real-time flood monitoring through the investigation of river hydromorphological features has become convenient due to video processing [23] and image processing [24] of data captured using UAVs. Cyber surveillance of an object in a flooded region for automatic monitoring and alert feedback are other applications of image processing [25]. The applications of image processing are not limited to early flood detection and flood monitoring, however. The segmentation and classification of UAV imagery are convenient for detecting the hydromorphological effects of floods in riverbeds and floodplains [26]. Plotting flood hydrographs has become possible using object-based image analysis (OBIA) with the help of image processing technology [27]. Jafari et al. [28] used live cameras and image processing techniques for real-time stream water level monitoring in this study. Apart from image processing techniques, convolutional neural networks (CNNs) [29], deep learning (DL) [30], artificial neural networks (ANN) [31], etc., for flood mapping using UAVs are other useful techniques that can be integrated, but image processing techniques have an edge over them due to the availability of high-resolution time saving algorithms [32]. Thus, from the existing literature, a range of applications of image processing for flood detection, flood monitoring, and analyzing post-flood effects is observed.

The hydromorphological feature data of the Kerian River are very limited. Therefore, this research presents the application of image processing for feature extraction in the Kerian River. The k-means algorithm for clustering along with thresholding for image segmentation are used in the present study to evaluate temporal variation in flow width and vegetation along river banks during a period of 3 months.

2. Materials and Methods

2.1. Study Area

We carried out the study in the Kerian River (Figure 1) that flows through the northern states of Peninsular Malaysia. The Kerian River is a meandering river of approximately 90 km length, originating in the Bintang Range in the northern state of Perak. This river serves as a border between the state of Kedah and Perak for approximately 73 km until it enters the Malaysian state of Penang for its remaining 17 km, where it finally discharges into the Strait of Melaka. The study region lies downstream of the Kerian River at the location 5$°$7′31″ N, 100$°$29′50″ E in Bandar Baharu, Kedah, where the width of the river measures 34.51 m. The width of the mouth of the Kerian River is approximately 166.71 m.

2.2. Flight Mission

The UAV used in the study is a DJI Mavic Pro fitted with a 1/2.3″ complementary metal-oxide-semiconductor (CMOS) 12MP 4K camera. We tabulate the characteristics of the camera in

Table 1. Characteristics of the DJI Mavic Pro camera.

Characteristics	DJI Mavic Pro Camera
Sensor Type	1/2.3″ CMOS Sensor
Million Effective Pixels	12.34
Image Size	4000 $\times$ 3000
Lens	35 mm
ISO range	100–1600

. We carry out the flight operations twice within three months. The flight mission comprised reconnaissance of the mapped site and pre-flight fieldwork. The H83 mm $\times$ W83 mm $\times$ L198 mm copter has an upright take-off weight of 734 g. Its fully charged lithium polymer (LiPo) 3830 mAh batteries provide a maximum flight time of 27 min. The DJI Mavic Pro can withstand wind speed up to 7 m/s, which was never exceeded during the flight mission. The flight altitude while capturing the images was 70 m, which the UAV climbed with an ascent speed of 4 m/s. The resulting ground sample distance was (GSD) 2.4 cm. To observe variation in flow width and vegetation growth, we execute a flight mission to capture UAV images on 31 July 2019 at 09:04 am and 6 November 2019 at 09:20 am at the location 5$°$7′40″ N, 100$°$29′48″ E in Bandar Baharu, Kedah. We name the sites S1 and S2 on 31 July 2019 and 6 November 2019, respectively. The wind speed recorded was 1.67 m/s with the partly cloudy conditions at S1 and 1.11 m/s with broken clouds at S2. The temperature recorded on these two days was 29 $°$ and 31 $°$, respectively. The level of tides was 1.69 m at S1 and 1.73 m at S2.

2.3. Image Processing of UAV Images

2.3.1. Implementation of Bilateral Filter

The first step in image processing was the enhancement of all UAV images, using a bilateral filter. Bilateral filter, a type of time-domain non-linear filter, reduces the noise with edge preservation by restoring the intensity of every pixel with weighted intensity values of surrounding pixels [33]. A bilateral filter uses the Gaussian distribution for calculating the weighted intensity values with an additional component, which is a function of pixel intensity difference [34]. This additional component ensures it used only weighted intensity values in computing blurred intensity values. The bilateral filter is defined as:

$$I_{filtered}=\frac{1}{W_P}{\displaystyle \sum }_{x_i\in \mathsf{\Omega }}I\left(x_i\right)f_r\left(\Vert I\left(x_i\right)-I\left(x\right)\Vert \right)g_s(\Vert x_i-x\Vert )$$

(1)

$W_P$ is the normalization term defined by:

$$W_P={\displaystyle \sum }_{x_i\in \mathsf{\Omega }}{f}_r(\Vert I\left(x_i\right)-I\left(x\right)\Vert )g_s(\Vert x_i-x\Vert )$$

(2)

In Equations (1) and (2), $I_{filtered}$ is the filtered image, $I$ is the input image, $x$ are the coordinates of the current pixel, $\mathsf{\Omega }$ is the window centered in $x$, $x_i ϵ \mathsf{\Omega }$ is another pixel, $f_r$ is the range kernel and $g_s$ is the spatial kernel. Allocation of the weight $W_P$ uses $f_r$ and $g_s$ [34].

If a pixel located at $\left(i,j\right)$ is to be denoised in image using surrounding pixels and we consider one of its neighbors at $\left(k,l\right)$, then the weight assigned to a pixel at $\left(k,l\right)$ to denoise a pixel at $\left(i,j\right)$ is given by:

$$w\left(i,j,k,l\right)=\exp (-\frac{{\left(i-k\right)}^2+{\left(j-l\right)}^2}{2{\sigma }_d^2}-\frac{\Vert I\left(i,j\right)-I{\left(k,l\right)\Vert }^2}{2{\sigma }_r^2})$$

(3)

In Equation (3), ${\sigma }_d$ and ${\sigma }_r$ are smoothing parameters, whereas $I\left(i,j\right)$ and $I\left(k,l\right)$ are pixel intensities at $\left(i,j\right)$ and $\left(k,l\right)$, respectively [34]. We calculate the denoised intensity of pixel, $I_D$ at $\left(i,j\right)$ after evaluating the weights and normalizing them using the equation:

$$I_D\left(i,j\right)=\frac{{{\displaystyle \sum }}_{k,l}I\left(k,l\right)w\left(i,j,k,l\right)}{{{\displaystyle \sum }}_{k,l}W\left(i,j,k,l\right)}$$

(4)

Bilateral blur can fragment the image into different scales [35]. Along with edge preservation, bilateral filter finds application in denoising, tone mapping, tone management, data fusion, retinex, and texture and illumination separation [36]. The present work required denoising, which the bilateral blur serves, and hence we select it for this study. We did not detect the staircase effect and gradient reversal in the filtered image in this work, which are the limitations of the bilateral blur.

2.3.2. Clustering of Pixels Using k-Means Clustering Method

We then clustered, filtered, and enhanced images using the k-means clustering algorithm. The k-means clustering method is a clustering algorithm. In image analysis, it can cluster pixels of an image in unsupervised classification. Implementation of the k-means clustering in Python is uncomplicated, and therefore widely used by novice programmers and data scientists [37]. We implement the k-means for image pixel clustering in Python using its multiprocessing feature. The multiprocessing feature enables the parallel implementation of the k-means algorithm, reducing the execution time. The principal component of k-means clustering involves two steps, expectation and maximization. We assign each pixel to its nearest centroid in expectation, while the maximization step involves computing the mean of all pixels for each cluster and sets the new centroid. The flowchart in Figure 2 depicts the steps involved in implementing the k-means clustering algorithm.

2.3.3. Thresholding of Clustered Images

We use HSV thresholding for the analysis of clustered images post-k-means clustering. In vision and image processing, separating image intensity from color information is very important for many applications that are possible only in HSV color space. For this reason, we convert the clustered images from RGB to HSV color space.

Image segmentation is significant to gain clear perceptibility of the region of interest (ROI) [38]. We compute limits of the segmented pairs of hue (H_lower, H_upper), saturation (S_lower, S_upper), and value (V_lower, V_upper) to convert HSV color space to binary form using the equation:

$$\begin{gathered} C\left(x,y\right)=\{1,H_{lower}<H_{input}\left(x,y\right)<H_{upper},\} \\ \{S_{lower}<S_{input}\left(x,y\right)<S_{upper},\} \\ \{V_{lower}\le V_{input}\left(x,y\right)\le V_{upper},\} \\ \left\{0,Otherwise\right\}. \end{gathered}$$

(5)

In Equation (5), $C\left(x,y\right)$ is the thresholded part. Additionally, Equation (5) shows that if the HSV values for the pixels of the input image lie between the range of lower and upper bound values, then its associated output pixel belongs to class object 1, otherwise it is null (0). The segmented image is further analyzed to compute the area of white pixels. We calibrate the percentage of white pixels to calculate the ROI in desired units.

2.4. Evaluation of Image Processing Algorithm Effectiveness

The human evaluator or quantitative evaluation methods can calculate the efficiency and effectiveness of image processing algorithms [17]. Some of the metrics approached and extensively used by researchers for evaluating the accuracy of image processing algorithms include dice similarity coefficient (DSC), Jaccard index, precision, and recall. In the present work, we use DSC and Jaccard index to assess the effectiveness of image thresholding with the implementation of the k-means clustering. We used DSC and Jaccard index for evaluating variation in flow area and vegetation growth. We calculate the efficiency of clustered segmented images to manually segmented images.

The DSC works on binary numbers, and its value ranges from 0% to 100% for no overlap to complete overlap between two data sets of image thresholding. The DSC is defined as:

$$DSC=\frac{2\left|A{\displaystyle \cap }B\right|}{\left|A\right|+\left|B\right|}$$

(6)

where $\left|A\right|$ and $\left|B\right|$ are the number of elements of set A and set B, respectively.

The Jaccard index (J) is the ratio of the intersection of two data sets to the union of the two sets. It also ranges from 0 to 100%, describing the same overlap conditions as DSC. The Jaccard index (J) is given by:

$$J\left(A,B\right)=\frac{\left|A{\displaystyle \cap }B\right|}{\left|A{\displaystyle \cup }B\right|}$$

(7)

3. Results

We show the oblique UAV images captured in the present study and visualize the study reach for the present study along with the in-channel structure in Figure 3. Apart from this, we also capture the floodplains, water areas, low vegetation, shrub vegetation, and tree vegetation. We also captured the urban areas around the floodplain on the terrace. The present work only focuses on water area and vegetation growth in order to enumerate their variation. We cropped the study region as per the ROI, maintaining the pixel ratio. Further, in order to remove the noise, we apply a bilateral filter. On application of the bilateral filter, we obscured considerable noise and smoothened the site images. The edge preservation benefit of the bilateral filter paves the way for better clustering. Sunlight and water reflectance that are denoised to an extent by the application of the filter influences the various forms of vegetation along the river basin and flow area.

The next step post-implementation of the bilateral filter is the clustering of pixels using the k-means clustering algorithm. The necessity of clustering is that it segregates the largely distributed values of color space into desired numbers of color variation equivalent to the number of clusters (k). Figure 4 shows the color inspection in RGB color space and HSV color space. The color inspector feature of ImageJ software was used for the aforementioned purpose. Prior to clustering of pixels, it was difficult to extract the RGB or HSV limits, which lead to a decrease in the effectiveness of image segmentation. Extracting the RGB or HSV limits from the color inspector models for the water area and vegetation cover region is extremely tedious work. The researcher needs to analyze every pixel of the image for extracting the limits.

Figure 5 shows the images filtered and clustered using a bilateral filter and the k-means clustering algorithm. As per the flowchart in Figure 2, for all the site images, we selected the value of k randomly from k = 2 onwards until the clusters appeared unchanged. We increased the number of clusters by 2 and visualized them by human vision. Further, we increased the number of clusters in order to classify the image into an optimum number of color clusters. Finally, we clustered Figure 5a,b with k = 16 as the images converged at this number of clusters. Further increase in k produced unchanged cluster centroids as observed by human evaluation. Human visualization incorporated with interpreting elbow graphs produces better results in terms of accuracy in predicting an optimum number of clusters [39]. We identified the optimum number of clusters using an elbow graph in the current work.

Post clustering of images, the color inspection of HSV components becomes a simple and effective task. We converted the color space of all the images from RGB to HSV. HSV color space proves useful for robustness, removing shadows, lighting changes, etc., because of the separation of color components from the intensity. Thus, increasing the effectiveness of the image segmentation algorithm. Figure 6 shows the color inspection model in HSV color space for clustered image (k = 16). Compared to Figure 5, the HSV color inspector post-application of the k-means clustering algorithm (Figure 6) produces a better visual for extraction of HSV limits, which saves the energy and time of the human evaluator. We tabulated the thresholding HSV limits of water area and vegetation for sites S1 and S2 in

Table 2. HSV limits of flow region and vegetation for Figure 6a,b.

Site	Feature	Minimum Limits			Maximum Limits
		H ($\mathbf{°}$)	S (%)	V (%)	H ($\mathbf{°}$)	S (%)	V (%)
S1	Flow Area	34	24	100	38	31	94
	Vegetation	64	10	40	83	62	89
S2	Flow Area	25	26	95	28	49	97
	Vegetation	35	6	21	78	52	85

and used these values for image thresholding (Figure 7). Then, we counted the white pixels depicting ROI in thresholded images using a pixel counting algorithm in Python. Finally, we performed the RGB to HSV color space conversion and HSV limit evaluation using ImageJ (Version 1.53c). ImageJ allows the human evaluator to move the pointer over the image, which reflects the HSV value on the color inspector. An image with an optimum number of clusters classifies the features of the image into a clear perspective. In the present work, we clustered the water area and vegetation spread into six color components each. We got the maximum and minimum values of hue, saturation, and value component of the flow area and vegetation from Figure 6 (Table 2).

4. Discussion

4.1. Variation in Water Flow Area and Vegetation Cover

We calibrated the number of white pixels as a percentage of the total number of pixels and in terms of actual area (

Table 3. Image segmentation results.

Site	Feature	White Pixels% (ROI)	Area (m2)
S1	Flow Area	39.441	4437.1
	Vegetation	19.513	2195.2
S2	Flow Area	39.531	4447.2
	Vegetation	25.314	2847.8

). The flow area varied slightly from 39.441% (4437.1 m²) to 39.531% (4447.2 m²). This slight variation is because of the tide difference, while all the other conditions were similar. An increase of 8.45% of the flow area will lead to overflow conditions. The vegetation growth over the period changed from 19.513% (2195.2 m²) to 25.314% (2847.8 m²). Malaysia having a tropical climate with year-round rainfall is one of the main reasons for this good amount of vegetation growth [40]. Vegetation significantly influenced the linkages between terrestrial ecosystems and atmospheric processes. It is workable to measure the advantages of ecosystem restoration by identifying the mechanisms through which vegetation dynamics and sensitivities to water resource availability respond at various temporal and geographical scales.

4.2. Quantitative Evaluation

The effectiveness of thresholding post-implementation of the k-means clustering is clear from the DSC values ranging from 92.51% to 97.86% and the Jaccard index ranging from 91.39% to 94.36%. Compared to thresholding of non-clustered images where the DSC is 77.99%, the Jaccard index is 63.92% for normal flow conditions [17], the present method scores much better effectiveness for normal flow conditions. The metric values prove better overlap between the manually segmented images and the algorithm segmented images. We expect we can score a much better DSC and Jaccard index for overflow conditions since in overflow conditions the ROI becomes smoother. We encountered the commission and omission errors in the image classification technique in the present work to a small extent. During image classification, at some point, no actual change is classified as a change in the model, which leads to commission errors. While in other instances, we left some actual changes unsegmented in image thresholding, leading to omission errors [41]. We observed the combined effect of these errors in the quantitative evaluation results (

Table 4. Quantitative evaluation results.

Site	Feature	DSC	Jaccard Index
S1	Flow Area	97.86	94.36
	Vegetation	94.91	92.20
S2	Flow Area	96.22	93.58
	Vegetation	92.51	91.39

). For the suggested research, the quantitative method can aid in an analysis where qualitative data is mainly in terms of analytic yield. The technique expands qualitative analyses horizons by providing researchers with a systematic way of guiding subgroup analysis within a qualitative data set.

4.3. Advantages and Limitations of UAV in River Feature Recognition

The advancement in UAV has paved the way for vertical dimension formation density and position referencing [42]. UAVs are also reliable for fluvial geomorphic study and reduce the error in observation arising in field study [43]. UAVs ensure less time consumption in data acquisition. They also have an upper hand in capturing real-time series data that we use in capturing flood events and landslides. Also, UAV technology compiles different data models produced in a single flight. Apart from being less expensive, UAV mapping provides equivalent spatial information and better operational features than airborne LIDAR [44]. However, the optical data acquisition of UAVs cannot pierce through vegetation cover and terrain features [45]. Another limitation of UAV technology is that the region with extensively low vegetation is unclassified in removal from the surface [46], but point cloud acquisition from the ground under the vegetation is possible using UAV LIDAR [47]. Another disadvantage of using small UAV platforms is that it limits the payload, while implementing drone-based imaging to capture temporal images of the same cross-sections is a tedious job.

4.4. UAV Imagery in Landscape of Kerian River and other Malaysian Rivers

The tropical climate of Malaysia and the year-round rainfall makes it difficult to analyze the river characteristics using low-cost satellite images because of heavy cloud covers [48]. UAV technology has a lead over satellite imagery because of its low operating cost. The Kerian River, for most of its upstream length, flows in an isolated region. The Kerian and other river basins of Malaysian rivers are the habitats of crocodiles, which makes it threatening for field visits. UAV imagery proves to be safe considering the climate and threats. The present study focuses only on flow area and vegetation growth using UAVs. We will get more features of the Kerian River using UAVs as inputs to the Ministry of Environment and Water, Malaysia in the future for river management. The sediment concentration by collecting sediments using samplers [49] and its validation using image processing are some gaps in this research topic. Apparently, for the Kerian River, there is limited river surveillance camera and flood does not occur frequently in this region. Therefore, direct use of a convolutional neural network (CNN) may not be a viable option for the river state classification. However, for future consideration, if enough surveillance cameras are installed at various cross-sections of the Kerian River, then we can develop a CNN model, as there will be a sufficient dataset to train the network. We may plan further work to use spectral cameras in UAVs to find the sediment concentration and composition using reflectance.

5. Conclusions

This work presented the application of the k-means clustering algorithm on UAV captured images with image thresholding for pressing out the flow area and vegetation growth. We calculated the binary white pixels to numerate the water area and vegetation cover. The results of image processing were proficient in withdrawing the river features. The dice similarity coefficient and Jaccard index for flow area and vegetation segmentation techniques fetched a value of over 90%, better than the existing techniques. Apart from this, the UAV technology is a promising tool to identify the river characteristics with high temporal information, low cost, and less time. A better understanding of river complexity may help the river management authorities in the proper planning of activities in the river basin. Our results will open a new corridor for the local officials in improving river hydro morphological conditions for sustainable river management, deploying various assessment tools.