Convolutional Neural Network and Optical Flow for the Assessment of Wave and Tide Parameters from Video Analysis (LEUCOTEA): An Innovative Tool for Coastal Monitoring

Abstract

Coastal monitoring is a topic continuously developing, which has been applied using different approaches to assess the meteo-marine features, for example, to contribute to the development of improved management strategies. Among these different approaches, coastal video monitoring coupled with recent machine learning and computer vision techniques has spread widely to assess the meteo-marine features. Video monitoring allows to obtain large spatially and temporally datasets well-distributed along the coasts. The video records can compile a series of continuous frames where tide phases, wave parameters, and storm features are clearly observable. In this work, we present LEUCOTEA, an innovative system composed of a combined approach between Geophysical surveys, Convolutional Neural Network (CNN), and Optical Flow techniques to assess tide and storm parameters by a video record. Tide phases and storm surge were obtained through CNN classification techniques, while Optical Flow techniques were used to assess the wave flow and wave height impacting the coasts. Neural network predictions were compared with tide gauge records. Furthermore, water levels and wave heights were validated through spatial reference points obtained from pre-event topographic surveys in the proximity of surveillance cameras. This approach improved the calibration between network results and field data. Results were evaluated through a Root Mean Square Error analysis and analyses of the correlation coefficient between results and field data. LEUCOTEA system has been developed in the Mediterranean Sea through the use of video records acquired by surveillance cameras located in the proximity of south-eastern Sicily (Italy) and subsequently applied on the Atlantic coasts of Portugal to test the use of action cameras with the CNN and show the difference in terms of wave settings when compared with the Mediterranean coasts. The application of CNN and Optical Flow techniques could represent an improvement in the application of monitoring techniques in coastal environments, permitting to automatically collect a continuous record of data that are usually not densely distributed or available.

1. Introduction

Video monitoring is becoming widely used on coastal systems, considering that a huge amount of data can be collected over a long-time range [1,2,3]. Ongoing widespread dissemination of digital systems for video monitoring allowed the development of innovative approaches to automatically obtain and process data along coastal areas. Previous studies have considered the time series of optical images to extract hydrodynamic properties of nearshore waves and currents, swash and runup [4], breaking wave, measures of breaking intensity [5,6], wave period, and wavelength (by monitoring a single pixel in the image over time), wave celerity [7], wave dissipation [8], attenuation [9], and alongshore currents [10]. These studies followed a supervised approach to establish or detect the physical wave parameters. The widespread of machine learning allows to automatically analyze a large amount of data [11,12,13]. The machine learning techniques can represent innovative approaches for coastal video monitoring, considering the computational gain that can be obtained from analyzing the long time series of video records.

On the other hand, conventional machine learning methods require manual image feature selection and need image data transformation so that they are more amenable to specific algorithms [14,15]. With the wide spread of deep learning methods, Earth-system processes began to be studied through new architectures and artificial intelligence methods to automatically obtain the meteo-marine data [16]. Deep learning architectures represent a set of learning methods attempting to model data with complex architectures combining different non-linear transformations in computer vision. The combination of different types of neural networks constitutes the bases of deep learning architecture [17,18]. Before the development of deep learning for computer vision, machine learning was based on extracting variables of interest, called features, throughout image processing training processes. The Convolutional Neural Network (CNN) introduced by LeCun [19] has revolutionized image processing and removed the manual extraction of features. CNNs act directly on matrices, or even on tensors for images with three RGB color channels, with Infrared channels, or hyperspectral images. CNNs are now widely used for image classification, image segmentation, object recognition, and face recognition [20,21,22]. Many improvements of CNN were obtained in the video monitoring field, like object recognition, action recognition, and classification of identified actions into categories like anomalous or normal [23,24]. Usually, CNN obtains the low-frequency data of sea motion, such as tide and storm surge. Alternatively, optical methods are used to obtain high-frequency data, to identify contrasting surface features to act as targets that are assumed to move from frame to frame. Foam is the predominant target traced within the surf zone through Optical Flow [3,25]. Moreover, optical images will also sense light reflections from sloped water surfaces and turbidity that alters background water colors [26]. Furthermore, alongshore currents have been estimated by extracting alongshore pixel time stacks from video [27,28].

In this work, we present a new monitoring system, called LEUCOTEA (convolutionaL nEUral networks for assessing COasTal hazard by video analyses), developed to assess tide, wave, and storm parameters from a video record. In this system, CNN was implemented into a MATLAB environment to obtain the tide phases and storm surge from specific frames of the video records. Subsequently, Optical Flow techniques were used to extract the wave flow and wave height from the wave motion recorded in the video. In the last decade, the Ionian coast of Sicily and the Atlantic coast of Portugal have been studied. Portugal’s coast has been influenced by the impact of hurricanes (Ophelia, 2017, Leslie 2018, Alpha 2020), while Sicily’s coast has been influenced by the impact of medicanes (Zorbas 2018; Trudy 2019; Ianos 2020; Apollo 2021).

Most importantly, both hurricanes and medicanes occurring in the two selected areas are related to a Tropical Transition able to transform an extratropical system into a tropical system [29,30]. Several authors predict that, soon, climate changes could modify hurricanes and medicanes, increasing their intensity and eventually also frequency along the coast of the Ionian Sea and the Atlantic coast of Portugal [31,32]. The combined method of deep CNN and Optical Flow, proposed in this work, has been developed to automatically obtain tide phases, surge, and physical wave parameters, useful for monitoring and studying hurricanes and medicanes, and which could also represent an innovative approach for early warning systems.

2. Geographic Framework

The Mediterranean area studied in this work is located along the coasts of the Maddalena Peninsula (south-eastern Sicily) (Figure 1). In contrast, the Atlantic areas are located along the central coast of Portugal, between Costa Nova in the north and Coxos in the south (Figure 2).

The coastal area of south-eastern Sicily is extended for about 300 km and is undergoing a heavy coastal retreat (shoreline erosion rate of about 5 m/year [33]). This area has been exposed to severe storms during the last decades (wave height of about 6 m and storm surge greater than 1 m [34,35]). The studied coastal area is characterized by an alternation of small rocky promontories and low-lying beach systems (pocket beaches) often bordering the coastal lagoon. Effects of several tsunamis have been reconstructed from the analyses of boulder accumulations [36,37,38], high-energy deposits [39,40,41], and lithostratigraphic analysis inside lagoonal areas [42,43]. The analyzed area also experienced the effects of several storms over the last decades, mainly translated by boulder and cobble dislocation along the coastal fringe [36,37,39]. As described for other Mediterranean coastal areas [44,45], this suggests that storms can produce effects comparable to other previous extreme marine events (i.e., tsunamis).

Scicchitano et al. [37,46] analyzed data recorded by the wave buoy of Catania (RON—Rete Ondametrica Nazionale; www.idromare.com accessed on 23 March 2018) and identified the strongest storm that has occurred in south-eastern Sicily since 1990 was characterized by significant wave height (H0) of about 6.2 m and peak period Tp = 11.3 s. Several medicanes impacted the coasts of south-eastern Sicily, and in the last decades, these events produced stronger effects when compared to common seasonal storms [35]. Between 2014 and 2021, five medicanes developed in the Ionian Sea. Three of these, called Qendresa (2014), Zorbas (2018), and Apollo (2021), strongly impacted the coast of south-eastern Sicily, in particular in Massolivieri, Maddalena Peninsula, and Varco 11 (Figure 2), with duration ranging from 4 to 7 days. As the Ionian Sea is not an area eligible for the formation of Tropical Cyclones, these phenomena occur due to a Tropical Transition (TT) process that can change an extratropical system into a tropical system or induce a hybrid cyclone [47,48]. However, several authors predict that, soon, climate changes could modify medicanes, decreasing the frequency of their occurrence but increasing the strength of their impacts [49]. However, in the Ionian basin, since 2018, a medicane has developed each year.

The Portuguese coastline extends for approximately 990 km. It is divided into different geomorphological units represented by sandy beaches, dunes, sand, and rocky cliffs, combined with river mouths, estuaries, lagoonal systems, and anthropogenic areas (Figure 2a,b). We selected four sites between Costa Nova in the north and Coxos in the south (Costa Nova, Cova Gala, Nazaré, and Coxos).

Costa Nova is located South of the Ria de Aveiro mouth (Figure 2b) and is part of the southern dune barrier of the Ria de Aveiro complex [50,51]. The dune system of Costa Nova is becoming more exposed to wave action due to the increasing erosion of this coastal stretch. On the other hand, some intervention strategies have been performed in Costa Nova, building several groynes, rock armor revetments, and shoreface nourishment [52]. On the other hand, and slightly to the south, Cova Gala is located south of the Figueira da Foz port (Figure 2c). The port was built inside the Mondego River estuary and has caused a great impact on the sediment dynamics along this coastal fringe, namely on its southern sector, which has led to an increase in erosion episodes over recent times [53]. A series of cross-shore groynes were constructed in the 1990s to mitigate the increasing effects of erosion in this area. South of the last breakwater, approximately 250 m of exposed geotextile sandbags protect the sand dunes. Similar to the previous locations, dominant waves are from the NW-W and frequently exceed significant wave height values of 6 m [53,54]. Nazaré’s Norte beach is located on the central west coast of Portugal (Figure 2d); more precisely, in the southward limit of the littoral cell that extends from the Douro river mouth and the Nazaré Canyon head [55]. This coastal stretch is exposed to the North Atlantic wave regime characterized by a predominant swell from the NW quadrant superimposed with a generally less energetic local wind sea with a wider directional spread [55].

Coxos is located on the Atlantic central west-facing coast of Portugal (Figure 2e). This is an irregular coastline section with an N-S general trend, where 20–50 m-high cliffs alternate with pocket beaches and structurally controlled rocky platforms [56]. There are only a few shore platforms at the toe of the cliffs, and they mostly correspond to plunging cliffs. These features allow for moderately high waves approaching the shoreline to preserve most of their energy until breaking or surging at the cliff face [56].

From a hydrodynamic point of view, the central Atlantic coast of Portugal is a high-energy system characterized by a swell-dominated stretch of coast that is fully exposed to waves generated far out in the Atlantic. The yearly mean significant wave height and peak period offshore the north-western coast is 2–2.5 m and 9–11 s, and the modal height and period range between 1.5–3 m and 9–14 s; this wave regime corresponds with WNW to NNW swell generated in high latitudes in the North Atlantic [57]. Costa et al. [58], based on the several wave buoy data along the Portuguese coast, were able to characterize the mean wave conditions for Figueira da Foz. They concluded that the annual average period was 6.6 s, and the most common peak values observed were 11.4 s.

The Atlantic studied area has been affected by several extreme marine events in historical times. Research undertaken in Portugal on extreme events and their recurrence is essentially based on historical data, roughly covering the last two millennia, and on geological data extending further in time. Besides major storms and hurricanes, tsunami events have impinged the Portuguese coast. Baptista and Miranda [59] and Andrade et al. [60] list historical and geological imprints of past tsunamis that impacted the Portuguese coast and concluded on a list of 17 reliable events since 60 before the Common Era. Hurricanes have increased in frequency and have caused widespread impact along the Portuguese coast and in some of the study areas. For example, Storm Christina in Coxos and Hurricane Leslie in Cova Gala caused extensive inland inundation, sediment erosion and transport, and huge economic damages [61,62].

As with south-eastern Sicily, the central Atlantic coast of Portugal is not an area where Tropical Cyclones form. Tropical Transition is the natural process responsible for the transformation of an extra tropical system into a tropical system. In addition, a recent study [61,63], based on a climate change scenario showing 1 °C less global warming than the SRES A1B scenario [64,65], suggests an eastward extension of the development region of tropical storms. In the current climate, the main genesis region for hurricanes is confined to the western tropical Atlantic, where sea surface temperatures are above the threshold (27 °C) required for tropical cyclones to develop. Future tropical storms that reach western European coasts (and cause hurricane-force storms) predominantly originate from the eastern part of the tropical Atlantic. This is because climate warming in the eastern tropical Atlantic causes sea surface temperatures to rise well above the 27 °C threshold.

3. Materials and Methods

LEUCOTEA system is composed of four different phases. The first step is dedicated to the video dataset definition, the second is related to the acquisition of field data, and the other two are the video analysis with CNN and Optical Flow, respectively.

3.1. Video Dataset Definition

The tide and wave parameters were assessed by applying deep learning and computer vision techniques to videos recorded in the areas reported in

Table 1. Datasets of Mediterranean and Atlantic video records used to assess tide phases, surge, and physical wave parameters.

Sites	Coastal Type	Acquisition	Tide	Storm Surge	Wave Height—Wave Length—Wave Flow
Varco 11—Sicily, Italy	Convex sloping rocky coast	Fixed remote-controlled camera	15–22 September 2021	20 March 2021 14 April 2021	15–22 September 2021
Santa Lucia—Sicily, Italy	Convex sloping rocky coast	Fixed remote-controlled camera	4–28 August 2021 15–22 September 2021	28 September 2018 20 March 2021 14 April 2021	28 September 2018 15–22 September 2021
Massolivieri—Sicily, Italy	Convex sloping rocky coast	Fixed remote-controlled camera	4–28 August 2021 15–22 September 2021	20 March 2021 14 April 2021	15–22 September 2021
Coxos—Lisboa, Portugal	High rocky coast	Action camera	N/A	N/A	25 January 2022
Nazaré—Leiria, Portugal	Sandy coast	Action camera	N/A	N/A	25 January 2022
Costa Nova—Aveiro, Portugal	Sandy coast	Action camera	N/A	N/A	26 January 2022
Cova Gala—Coimbra, Portugal	Sandy coast	Action camera	N/A	N/A	27 January 2022

. The data was acquired with two different systems: fixed remote-controlled cameras and action HD cameras. Continuous 24 h video recorded by the surveillance camera system on the Marine Protected Area of Plemmirio, along the coasts of Maddalena Peninsula in south-eastern Sicily, were used to develop the Convolutional Neural Network (CNN). This was later used to obtain tide values and storm surge, and the Optical Flow was applied to assess the wave flow, period, and length and to derive wave height. In 2003, the technicians of the Marine Protected Area of Plemmirio installed 10 surveillance camera stations intending to detect illegal fishing operations and monitor sea conditions. The stations are mounted on 9 m high poles and equipped with an HDTV camera AXIS P5635-E Mk II with 30× optical zoom framing the coastal area (lens 4.3–129 mm; F1.6–4.7; horizontal field of view: 65.6–2.0; vertical field of view: 39.0–1.2). In 2018, during the impact of medicane Zorbas, the video from the camera Santa Lucia (Figure 1a) was used to detect several boulders displacement occurring during the storm induced by the Mediterranean hurricane. Since 2021, continuous video from three distinct cameras (Massolivieri, Santa Lucia, and Varco 11) has been collected.

Along the Atlantic coast of Portugal, we tested the use of Action HD Cameras instead of a fixed remote-controlled cameras system. In this case, we used three GoPro Hero Black10, installed in distinct positions along the studied coastal areas, to define the wave setting from different points of view, perpendicular to the shoreline. Each GoPro Hero Black10 recorded about 3 h of video, with a resolution of 5.3 k and frame rate of 60 fps, in each one of the four selected sites (Figure 1b). The acquisition with action cameras was performed to test the difference in the Optical Flow results compared to the fixed remote-controlled cameras.

3.2. Field Data

Topographic data were acquired through a phase-shifting Laser Scanner Faro Focus X130, obtaining the point clouds useful to calibrate the geometrical elements on the webcam perspective. The point clouds were georeferenced with GPS-RTK (Leica Geosystem) acquisitions and corrected for the geodetic elevation. Furthermore, GPS—RTK acquisitions were used to obtain Ground Control Points (GCPs) needed for calibrating deep learning and Optical Flow outputs. The georeferencing of point clouds with GCPs showed a root mean square (RMS) equal to 0.014 m.

To validate the deep learning results with field data on the Italian sites, a tide gauge sensor was installed in the Varco 11 coastal area. The tide data were obtained using a pressure gauge Valeport, installed on a wood brick on the Varco 11 (Figure 3), acquiring data from 15 September 2021 to 20 September 2021, with a sampling rate of 10 min. Offsets of the pressure sensor were assessed through the 3D cloud points reconstructed with Laser Scanner; measured tide gauge data were compared with the Catania tide gauge records to highlight the semidiurnal phases in the area of Siracusa.

3.3. CNN for the Tide Assessment

The CNN developed in this work is based on the assumption that it is possible to assign a tide value for each video frame recorded by webcam. From the video records, images were automatically sampled each 10 min and resized with a 224 × 224 pixel size. The architecture of the classification network requires the tide classes to represent the tide height during the acquisition of the video frames. Tide classes were chosen using the tide records of Varco 11 and Catania and the vertical spatial reference measured through TLS data. For these reasons, the tide classes were divided with a sample pass of 0.05 m, ranging from 0 to 1.2 m, the highest limit measured by water level during the extreme marine events recorded in video. For each class, a set of 20 images was extracted every 15 min from video frames for a global training set composed of 540 images for each surveyed area of south-eastern Sicily.

The architecture of the classification network was developed in a MATLAB environment (version 2021b), following the architecture of GoogleNET [66] (Figure 4a,

Table 2. Structure of GoogleNET modified for LEUCOTEA to assess the tide classes from video records.

Layer	Patch Size/Stride	Function
Input images of time frames sampled each 10 min
Conv2D	7 × 7/2	ReLU
Max pool	3 × 3/2	Normalization
Conv2D	3 × 3/1	ReLU + Normalization
Max pool	3 × 3/2
Inception module
Inception module
Max pool	3 × 3/2
Inception module
Inception module
Inception module
Inception module
Max pool	3 × 3/2
Inception module
Avg pool	7 × 7/1
Fully connected
Softmax
Classification—output of tide class

). The network development was performed with the following hardware device: CPU Intel Core i5 10th generation, GPU Intel^® UHD Graphics, 12 GB of RAM running on Windows 10. The first layer corresponds to the input image of the sea-state recorded by webcam. The images are analyzed with a continuous series of layers of three types: convolutional layers, pooling layers, and fully connected layers. The convolutional layers for 2-Dimensional signals (such as images) are properly called 2D-convolutional layers, in which a convolution kernel is applied to an image. At each position, convolution between the kernel and the part of the image that is processed. Then, the kernel moves by a number n of pixels, n is called the stride. When the stride is referred to few numbers of pixels (from 5 to 10), redundant information is obtained. A zero padding is added, a margin of size l containing zero values around the image to control the output size. The convolution operations were combined with an activation function called Rectified Linear Unit (ReLU), generally the ReLU activation layer [67]. ReLU is a piecewise linear function that returns 0 if it receives any negative input, but for any positive value x, it returns that value. Thus, it provides an output with a range from 0 to infinity.

Furthermore, a cross-channel normalization operation was used for different channels to normalize each activation. CNN also has pooling layers, which allow for reducing the dimension, also referred to as subsampling, by taking the mean or the maximum on patches of the image (mean-pooling or max-pooling). Like the convolutional layers, pooling layers act on small patches of the image, producing a stride. In the deep learning network, pooling layers were used, which returned the maximum patches (max-pooling 2D layer) [68]. The depth concatenation layers use inputs with the same height and width and concatenate them along the third dimension (the channel dimension). The CNN ends with a fully connected layer, where each element of the previous layer is connected to each element of the softmax layer. The softmax layer is a function that estimates the relative probabilities and determines the final probable value. The architecture of the deep neural network is based on the Inception modules [69] (Figura 4b), which allow the network to choose between multiple convolutional filter sizes in each block. An Inception network stacks these modules on top of each other, with occasional max-pooling layers with stride 2 to halve the resolution of the grid. The CNN was trained with 70% of training images and 30% of validation images. The maximum number of epochs for training is 20, and a mini-batch with 42 observations was used at each iteration. The total number of iterations equals 80, with 4 iterations for each epoch. The final layer provides the probability of the tide classes annexed to a given image. Considering the micro-tidal regime of the Mediterranean coast, the entire range of tide phases has been classified. Thus, the highest probability was used as a classification criterion for a given time frame. On the other hand, extreme values of temporary sea-level rise are given by storm surge, which could exceed 1 m, as showed by historical tide records and direct field surveys [35].

3.4. Tracking Waves via Optical Flow

The wave features were assessed through a dense Optical Flow module in a MATLAB environment, with the same approach followed by Wu et al. [26]. The Optical Flow assessment is based on the object motion estimation observed between consecutive sequence frames caused by the relative movement between the object and camera. Considering the pixel I(x, y, t) within a frame of an image, the pixel has moved by a certain distance (dx, dy) in the next frame, taken in the time increment dt (Figure 5).

In the work presented here, webcams are fixed, so the motion of objects is only relative to the objects themselves. The original image resolutions of the captured videos were 1080 × 1920 pixels. To obtain the wave motion parameters, the Farneback method was used [70,71]. With the GPU acceleration, the processing was performed for 25 frames per second (fps). First, the video frames were converted from RGB to greyscale to obtain the pixel intensity of the images. In the MATLAB environment, the Farneback algorithm generates an image pyramid of the pixel intensity, where each level has a lower resolution than the previous level. Then, the sequence of videoframes is analyzed as the pixel intensity changes during the time step. From pixel intensity images, the flow vectors were assessed by computing spatio-temporal image brightness derivatives using the Farneback technique. This method approximates the windows of image frames by quadratic polynomial expansion transform and approximates the neighborhood of both frames at consecutive times using a polynomial function. The Optical Flow outputs provide the flow vectors in x-y directions, the magnitude of flow vectors, and the vectors’ orientations. The square root of the magnitude of flow vectors represents the wave flow on the water surface. The difference in the pixel intensity was observed on the water surface, allowing us to detect the different wave fronts through videos. Outputs of Optical Flow showed the wave flow with dimensional units of pixel/frames. To convert the pixel/frame units in meters/second, the TLS data were used to obtain the dimensional grid on the water surface (Figure 6), and the frame rate was converted in seconds. This way, the distance between the wave fronts can be measured in metric units. The distance between the wave front allowed for assessing the wavelength, and the time measured between two consecutive crests (one wavelength) to a given point provided the wave period. Knowing the wave flow, wavelength, and wave period, it is possible to obtain the wave height by applying different relationships. In this work, the wave heights were obtained through the linear theory of Airy on shallow water [72,73,74], and breaking-wave height was obtained through empirical formulas [75]. The airy theory is based on the functions to their first order (hence linear) only, which provide the wave height in shallow water with the following relationship:

$$H=\frac{4\pi }{gT}{\cos }^{-1}\left(\sigma t-kx\right)$$

(1)

where $g$ is the gravity acceleration, $T$ is the wave period, $\sigma$ is the wave frequence, $k$ is the wave number, $x$ is the linear length on the orthogonal direction of the wave front, and $t$ is the temporal range between two video frames.

To compare the linear theory results with empirical studies, formulas for assessing breaking-wave height were considered. The relationships of the breaking-wave height are based on the bottom slope and wave characteristics (Table 3). Rattanapitikon and Shibayama [6,76] have analyzed different formulas for the breaking-wave height and showed that the best match between formula results and observational data was obtained by the following relationship:

$$H_b=0.17L_0\left[1-exp\left(\frac{\pi d_b}{L_0}\left(16.21m^2-7.07m-1.55\right)\right)\right]$$

(2)

where:

• $H_b$ is the wave break height (m);
• $L_0$ is the wavelength in offshore (m);
• $d_b$ is the water depth with breaker conditions (m);
• m is the bottom slope (radiant).

For each area of the Mediterranean and Atlantic coast, average bottom slopes m were obtained from available bathymetric surveys [54,77,78,79,80]. Bathymetric data were obtained for the winter seasons (during 2015–2016 [77,78,79,80]) to assess a range value of coastal slope m similar to the coastal slope and wave climate recorded during the video monitoring. Wavelength in offshore $L_0$ was assessed through wave propagation modeled for the offshore areas of Sicily [35] and Portugal [81]. The water depth at the breaker zone was derived from literature [82,83] (Table 3).

Table 3. Parameters used for the assessment of breaking-wave height at the offshore conditions. Average bottom slope m was obtained through available bathymetric surveys. The water depth at the breaker zone was derived from literature [78,82,83,84,85]; wavelength in deep water was assessed from the SWAN model (reported in Scicchitano et al. [35]) and wave model of Spanish Harbor Authority (SIMAR 1,041,058 point—9.75° W and 39.50° N; SIMAR 1,042,065 point—9.5° W and 41.25° N [81]).

Coastal Area	Average Bottom Slope m (Radiant)	Wavelength in Deep Water L0 (m)	Water-Depth db at the Breaker Zone (m)	References
Massolivieri	0.03–0.05	30	6	[79,82]
S.Lucia	0.05–0.06	30	6	[46,79,82]
Varco 11	0.10–0.12	30	6	[46,79,82]
Cova Gala	0.02–0.05	45	5.5	[54,83]
Costa Nova	0.02–0.05	45	5.5	[3,62,83]
Nazaré	0.19–0.05	70	12	[78,80,85]
Coxos	0.2–0.23	70	15	[78,80,85]

Table 3. Parameters used for the assessment of breaking-wave height at the offshore conditions. Average bottom slope m was obtained through available bathymetric surveys. The water depth at the breaker zone was derived from literature [78,82,83,84,85]; wavelength in deep water was assessed from the SWAN model (reported in Scicchitano et al. [35]) and wave model of Spanish Harbor Authority (SIMAR 1,041,058 point—9.75° W and 39.50° N; SIMAR 1,042,065 point—9.5° W and 41.25° N [81]).

Coastal Area	Average Bottom Slope m (Radiant)	Wavelength in Deep Water L0 (m)	Water-Depth db at the Breaker Zone (m)	References
Massolivieri	0.03–0.05	30	6	[79,82]
S.Lucia	0.05–0.06	30	6	[46,79,82]
Varco 11	0.10–0.12	30	6	[46,79,82]
Cova Gala	0.02–0.05	45	5.5	[54,83]
Costa Nova	0.02–0.05	45	5.5	[3,62,83]
Nazaré	0.19–0.05	70	12	[78,80,85]
Coxos	0.2–0.23	70	15	[78,80,85]

4. Results

4.1. Tide Assessment through CNN Classification Technique

The CNN outputs provide the probability for the tide classes annexed to a given video frame of the webcams. In Figure 7, a snapshot of Varco 11 is shown with the probability of tide classes associated with that image. To represent the tide phases in a continuous temporal range, different snapshots were automatically extracted from the video records, and tide classes were associated to each image of three rocky coasts (Varco 11, Massolivieri, Santa Lucia). The accuracy reached by the CNN is 84.5%, with a Categorical Cross Entropy Loss function that provided a value of 0.75 at the end of iterations (Figure 8a). The CNN predictions were compared with the actual values of the tide signal recorded in Varco 11 and Catania through a Root Mean Square Error (RMSE) analysis. An RMSE analysis was performed to assess the error between the CNN prediction and the actual value of the tide signal, showing an RMSE equal to 0.12 m. A three-order polynomial interpolation was applied to the tide values obtained from the CNN to reproduce the tide phases. The results were compared with the tide data acquired in Varco 11 (Figure 8 and Figure 9). The metrics of the CNN output were assessed through the confusion matrix between predicted classes (tide classes obtained from CNN) and true classes (tide values obtained from spatial reference) (Figure 8b). Since the class distribution is unbalanced data, the F1-Score was used, which provides an excellent compromise between precision and recall [86]. F1-Score was assessed from the confusion matrix, obtaining a value of 0.81, representing the percentage of the model reliability. Since tide classes are mutually exclusive and cover all possible tide phases, the highest probability of predicted class was selected as reliable tide values. However, extreme values can occur during storms and medicanes, where predicted classes are associated with higher values than common tide phases, reaching values greater than 1 m.

To highlight the relation between the CNN predictions and different tide phases, such as neap tide and spring tide, the CNN was applied for a longer record of 1 month, acquired in August 2021 from Santa Lucia and Massolivieri cameras. The RMSE assessed for the two areas showed 0.018 m for Santa Lucia and 0.027 m for Massolivieri.

4.2. Wave Flow and Wave Height

Through Optical Flow, it was possible to obtain the wave flow for each video frame recorded by surveillance and action cameras. The Optical Flow module provides the wave flow vectors of sea–surface (Figure 10a). Once the values of wave flow vectors are known for a given area, and through the application of the Airy linear theory, it was possible to obtain the tensors of wave heights for the area (Figure 10b,c). An RMSE analysis was performed considering the wave height assessed through the linear theory with reference points assessed throughout the TLS data for all studied areas, showing an error of 0.1 m. The application of Optical Flow on the continuous video records allowed us to obtain the wave heights impacting the coasts for the entire duration of the video records. Furthermore, it was possible to assess the wave break heights that impacted the coasts. In Figure 11, it is reported the automatic extraction of wave break heights on the Cova Gala coast, showing the greatest wave heights recorded in the video. The Optical Flow allowed assessing for all surveyed areas (Supplementary Material—Optical Flow Wave Height), showing the results for the different records of surveillance cameras and action HD cameras.

4.3. Wave Insights during Storm Events

CNN and Optical Flow were applied to assess the storm features recorded by the webcam located in the Greek quarry of Santa Lucia (Figure 1b). In particular, four extreme marine events were analyzed: medicane Zorbas of 2018, two storms in 2021, and medicane Apollo 2021. Medicane Zorbas was the strongest recorded and determined a significant surge of about 1.0 ± 0.2 m, measured by spatial reference of TLS data and GPS-RTK acquisitions by Scicchitano et al. [35] and Nandasena et al. [38]. Furthermore, during medicane Zorbas, the movements of several boulders were identified inside the Greek quarry of Santa Lucia [46]. The classification techniques provided the height of the storm surges (Figure 12), showing maximum storm surge values for Zorbas and Apollo, respectively, equal to 1.0 ± 0.2 m and 0.9± 0.2 m, and lower values for the two other analyzed storms (Figure 13a,

Table 4. Average values of marine parameters obtained through CNN and Optical Flow (relationship (1)) on the video records of medicanes and storm events.

	Storm Surge (m)	Wave Height (m)	Wavelength (m)	Wave Flow (m/s)
medicane Zorbas (September 2018)	1.1 ± 0.2	3.3 ± 0.4	25 ± 0.2	5.1 ± 0.5
Storm (March 2021)	0.15 ± 0.05	1.1 ± 0.4	13 ± 0.5	4.2 ± 0.5
Storm (April 2021)	0.4 ± 0.1	0.9 ± 0.4	13 ± 0.5	4.0 ± 0.5
medicane Apollo (October 2021)	0.9 ± 0.2	2.5 ± 0.4	22 ± 0.2	5.1 ± 0.5

). Furthermore, Optical Flow allowed us to assess wave height impacting during the storm events (Figure 13b). The wave heights impacting during medicane Zorbas showed an average value of 3.3 ± 0.4 m, compared to 1 m in wave height recorded during the storms of 2021.

5. Discussions

During the last few decades, the studies about the video monitoring of the coasts followed a supervised approach to obtain results from video records [4,46,87,88]. Considering the large amount of data contained in a video, procedures able to process data without a supervised approach are fundamental to improving the video monitoring of coastal environments. This approach was previously performed to classify the type of breaking wave [7,89,90], detect the physical units of the landscape [89], for the beach attendance prediction [91], and to monitor the marine and beach litter [21]. In this work, we developed a combined method of deep CNN with Optical Flow to assess, from video recorded with different systems, important hydrodynamic parameters such as tide phases, storm surge level, wave flow, and wave height impacting the Ionian coast of south-eastern Sicily and the Atlantic coast of Portugal. Video has been acquired using two different systems: fixed remote-controlled cameras and action HD cameras. Continuous 24 h video recorded by the surveillance camera system of the Marine Protected Area of Plemmirio, along the coasts of Maddalena Peninsula in south-eastern Sicily, were used to develop the Convolutional Neural Network (CNN), used to obtain tide phases and storm surge levels, and of the Optical Flow, applied to assess the wave flow, period and wavelength and to derive wave height. Nets composed by action HD cameras were tested along the Atlantic coast of Portugal to evaluate their efficiency in the assessment of the wave flow, period, and length through the use of the Optical Flow developed in south-eastern Sicily. Although the fixed surveillance camera allowed for assessing the low-frequency components of the sea-state [92], such as the tide phases and storm surge, action cameras represent a more versatile system for coastal monitoring and allow acquisitions from multiple views. The different views calibrated with spatial reference allow for assessing the value of surge and wave setup [1,3,93,94].

Several studies have attempted to assess the tide values based on historical tide records to predict tide phases using machine learning with regression functions [95,96]. However, these studies are based on neural networks which predict tide values using previously recorded data and astronomical factors [97,98]. In our study, we developed a CNN able to assess tide phases from video recorded along the coasts, using a specific classification of tide classes. The tide classes were chosen through the tidal signal recorded with a tide sensor and through the spatial reference measured by tide data. The tide classes were only obtained during diurnal hours when the tide phases are observable. The missing tide values during the night were assessed through a three-order polynomial interpolation of diurnal tide classes.

This intense classification let our CNN accurately assess the tide phases for a long-time record (about 1 month), using videos recorded continuously for 24 h by three different cameras located along the Maddalena Peninsula (south-eastern Sicily). The long-time record was compared with the tide signal of the Catania tide gauge (ISPRA), showing the difference in amplitude between neap tide and spring tide (Figure 14).

Although this method, requiring continuous video monitoring, was not applicable with the action HD cameras along the Atlantic coast of Portugal, it provided important results along the Ionian coasts of south-eastern Sicily. Considering that this area is characterized by a micro-tidal regime and by a coast extended for about 300 km with only three tide stations, located in Messina, Catania, and Porto Palo di Capo Passero, the use of light-weighted video records for tide assessment could allow for the possibility of increasing the monitoring points and thus increase spatial resolution. On the other hand, widespread video monitoring stations could improve the assessment of storm surge parameters, in which the tide gauges are unevenly distributed along the coasts and mainly located in the harbors, meaning that video monitoring could assess the storm surge directly impacting the coasts [4,99]. This is particularly important to predict the effect of coastal flooding and submersion in susceptible areas, such as wetlands [100], deltas [101], low-lying coasts [33], and highly populated coastal cities [33,102].

Similarly, these methods can also be applied on the mesotidal and macrotidal coasts of the Atlantic to monitor its higher tidal phase [92]. However, these methods need continuous video monitoring that did not apply to the action cameras. Thus, the application of CNN for tide assessment was made through surveillance cameras that continuously record the state of the sea.

Optical Flow techniques have been used to determine wave setting in the proximity of the coast. In particular, wave flow has been determined by detecting the pixel intensity of the wave front recorded in the videos [26,27]. The conversion of the pixel width into spatial scales, using topographic data detected with a Laser scanner, allowed for evaluating the wave flow vectors. Later, it also allowed applying the linear Airy wave theory to assess the wave height impact on the coast. A comparison between spatial reference and wave height obtained by the model showed a coefficient of determination equal to 0.97 (Figure 15). The Optical Flow module allowed us to assess continuously and automatically the wave heights for the entire duration of video records. The assessed wave heights showed the high frequency of wave oscillations (see Figures S2 and S3 in Supplementary Material) and the heights reached by breaking waves (see Figures S4 and S7 in Supplementary Material). The outputs of Optical Flow are integrated with linear Airy theory and breaking-wave relationship [6,76] that provide the time series plot of wave height impacting, useful to discriminate the storm wave parameters.

This combined approach between CNN and Optical Flow was particularly useful for detecting oceanographic parameters in the proximity of the shoreline. Moreover, it is useful to assess the wave’s physical parameters and its behavior during extreme marine events.

The coast of south-eastern Sicily and the Atlantic side of Portugal are greatly influenced by storm events, medicanes [46,103,104], and hurricanes [32,62,63]. Both areas are characterized by the development of Tropical Like Cyclone for the Tropical transition of an extratropical system. This characteristic makes the two areas highly vulnerable to hurricane impact, especially under the ongoing climate change scenarios. Regarding the end of the 21st century, several studies suggest a modification in the frequency and energy of the medicanes and hurricanes that will occur along the coast of south-eastern Sicily and the Atlantic coast of Portugal [31,32,105]. With this scenario in mind, our deep learning methods seem to be able to measure hydrodynamical parameters from video records. They could represent a very important tool for monitoring and studying these events, and potentially for developing early warning systems.

The impacts of medicane Zorbas during 27–29 September 2018 and storm events of 2021 were recorded by webcams of MPA of Plemmirio, and data obtained through CNN and Optical Flow techniques highlighted two main features of these events: the storm surge and wave features from the nearshore to the coastline. The CNN predictions for the storm surge of medicane Zorbas match the field data collected by Scicchitano et al. [35], where the storm surge showed a value of 1.1 ± 0.2 m. Furthermore, according to Scicchitano et al. [35], measured storm surge values of medicane presented comparable to those measured by the tide gauge. On the other hand, the Optical Flow showed accurate average wave height values during medicane Zorbas, greater than storm events. However, the Sicilian coasts experienced the impact of common seasonal storms with wave heights comparable to medicane Zorbas, and even higher than medicanes [36,82,104]. The coastal flooding during medicane was characterized by a greater inland inundation than common seasonal storms. This difference in inland inundation was attributed to high storm surge during medicane occurrence [103,104,106].

Further Developments

Although LEUCOTEA allowed to assess some parameters of the energetic balance on the coasts, further studies are needed to integrate the deep learning techniques with other architectures to assess the mass balance [16,91,107]. The assessment of mass balance requires the implementation of different machine learning models, which could involve classification and segmentation techniques for the quantitative evaluation of coastal landforms and sediment typologies [108,109]. For the energetic balance, a full framework of all components must be realized through a joint assessment of current, tide, wave, and wind parameters. This latter topic can be addressed through experimental studies with tanks integrating pressure sensors and sea-surface temperatures with machine learning models [75]. These approaches can be easily applied to the long-time video records, for example, in the surveillance system of south-eastern Sicily. On the other hand, the LEUCOTEA system will also be improved for the Atlantic coast to increase the tide class and storm surge values for mesotidal and macrotidal coasts. Currently, applying machine learning models for climate changes and sea-level rise is of great interest [20,92,110]. Several machine learning models have already been implemented to predict sea-level rise through regression functions [111,112] or track Hurricane forecasting [113,114]. Integrating the different machine learning models can improve the prediction of sea-level rise scenarios, allowing the application of new strategies for coastal management.

6. Conclusions

LEUCOTEA system has been developed to derive meteo-marine insights from video records using CNN with Optical Flow techniques on the Mediterranean and Atlantic coasts, which might be helpful for local stakeholders. The classification algorithm of the CNN allowed us to obtain:

• the tide phases from video frames.
• the storm surge values during the occurrence of an extreme marine event.

The application of Optical Flow module was applied to obtain the high-frequency component of the marine parameters, allowing us to calculate:

• the wave flow vectors from pixel intensity values converted in m/s units through TLS data.
• the wave height, wave period, and wavelength obtained for each position of wave flow vectors.

For coastal monitoring, the application of deep learning and object detection represents a low-cost tool to increase the amount of data that usually are not densely distributed along the coasts. On the other hand, video records highlight the high-energy events, such as storms or medicanes, as happened in south-eastern Sicily for medicane Zorbas (September 2018). The application of CNN and Optical Flow for these events allowed us to evaluate their hydrodynamic features, like storm surge and wave height impact, which are difficult to assess in the field during the occurrence of the events. Prediction models concerning the increase in the intensity of future extreme marine events need a large amount of data, and applying deep learning to coastal monitoring can be useful to increase the available dataset for coastal monitoring.