Early Surge Warning Using a Machine Learning System with Real-Time Surveillance Camera Images

Abstract

While extreme oceanic phenomena can often be accurately predicted, sudden abnormal waves along the shore (surge) are often difficult to foresee; therefore, an immediate sensing system was developed to monitor sudden and extreme events to take necessary actions to prevent further risks and damage. Real-time images from coastal surveillance video and meteorological data were used to construct a warning model for incoming waves using long short-term memory (LSTM) machine learning. This model can predict the wave magnitude that will strike the destination area seconds later and issue an alarm before the surge arrives. The warning model was trained and tested using 110 h of historical data to predict the wave magnitude in the destination area 6 s ahead of its arrival. If the forecasting wave magnitude exceeds the threshold value, a warning will be issued, indicating that a surge will strike in 6 s, alerting personnel to take the necessary actions. This configuration had an accuracy of 60% and 88% recall. The proposed prediction model could issue a surge alarm 5 s ahead with an accuracy of 90% and recall of 80%. For surge caused by a typhoon, this approach could offer 10 s of early waring with recall of 76% and an accuracy of 74%.

1. Introduction

Real-time sensing systems are essential in monitoring sudden or extreme events and taking the necessary actions to prevent further risks and damage. Ocean movement is complex, involving numerous factors such as the amplitude of longitudinal and vertical waves, sea temperature, wind, transportation phenomena, and coastal erosion. Rapid scientific and technological developments have allowed most extreme events to be accurately predicted; however, transient surges such as freak or rogue waves are often difficult to predict. These surges can also occur on the shore, affecting anglers along the northeast coast of Taiwan [1]. There are currently two common wave observation techniques, contact and non-contact. Traditional observation technology is mainly contact, such as a buoy system equipped with a three-axis accelerometer [2], which is currently the most widely used instrument for wave observation. In addition, the high-precision global satellite navigation system (Global Navigation Satellite System, GNSS) is also a promising wave-measuring tool [3] with other tactile observation techniques including bottom spindle pressure and sonometers. Common non-contact devices include optical (optics) systems and radar wave (radar) systems [4]. Satellite SAR imagery could detect the advancing direction and velocity of group waves, but spaceborne SAR imaging is not a conducive product to fulfilling the need of early wave warnings. And the optical sensing system is limited by illumination and not suitable for monitoring at night. Recently, X-band radar signals were applied to infer the effective wave height, peak period, and average direction [5,6]. This technique has spatial information, so it has its advantages in long-term statistics of waves in the same area, but this method cannot be applied to nearshore surge observation. Coastal optical surveillance video-camera systems are a low-cost alternative and their easy instalment characteristic means that they are commonly used to monitor rip currents [7] and shoreline changes [8]. Optical cameras can also be used to monitor low energy microtidal beach processes of beach erosion and accretion [9], or convolutional neural networks can automatically create a storm impact database using time stack images provided by coastal video monitoring stations [10]. Coastal video monitoring is a shore-based remote sensing technology that uses optical cameras mounted at heights to capture high-resolution images of nearshore areas [11]. Due to the distinct characteristics of waves on the sea surface, it has proven to be effective for measuring breaking wave height [12] and also offers an alternative to identifying the wave transformation domain boundaries based on spatial variation in wave types [13]. However, it is dedicated to identifying the wave breaking point and image processing is based on a threshold image pixel magnitude [12], so it is limited to specific wave flow phenomena that have a clear breakpoint. Consequently, a solution using the ‘brightest’ of surveillance camera images, which is formed by the highest pixel magnitude within the sampling interval, was proposed to chart the wave magnitude with 20% error [14].

Machine learning is popular for image recognition and sequence-to-sequence translation, with the former making use of convolutional neural networks (CNNs) and the latter using recurrent neural networks (RNNs), especially the long-term and short-term memory model (LSTM) [15]. Jörges et al. integrated bathymetric data with an LSTM to reconstruct and predict wave height [16]. Meng et al. proposed a new LSTM for long-term accurate predictions of sea wave trains considering typical rule waves, irregular waves and real sea wave trains [17]. A deep learning method based on LSTM was used to predict focused waves [18], applying GRU and LSTM for wave forecasting and hindcasting and achieving good agreement [19]. Primitive instantaneous waveforms were generated by computational fluid dynamics (CFDs) and a surveillance camera, via AI, accurately predicted the individual observation point’s focused waves, as well as rogue ocean behavior, achieving an accuracy surpassing 90% and maintaining a loss value below 1 [20]. No study has presented a system that could raise the alarm about an incoming surge a few seconds prior its strike with acceptable accuracy.

This paper presents a multi-shot prediction method to improve the accuracy of the long-term estimation of irregular waves, offering 6 s early warnings with 60% accuracy. The training strategy involved multi-point data fusion [21] and the wave data were obtained from different locations. An artificial neural network (ANN), LSTM and time convolutional network (TCN) were used to test the offshore data to verify the feasibility of the training strategy.

2. Study Site and Video Data Source

The chosen study site is the Tiangong Temple, Dali village, Yilan county, Taiwan (24°58′06.2″ N, 121°55′40.4″ E). Real-time surveillance video images were obtained from a fixed camera set up by the Central Meteorological Administration (CMA). The fixed camera was aimed at the shoreline of northeastern Taiwan at location A in the location index panel of Figure 1, and the surveillance footage was transmitted in real-time on a live YouTube webcast (https://www.youtube.com/watch?v=68qSYgEjl4k accessed on 1 October 2024). The video files were stored by screen recording on the computer with a screen sampling rate of 12 Hz and a total of 110 h of daylight recorded between the 3rd and 19th of June 2024, comprising different conditions such as tide heights, wave heights, and wind speeds, as well as sunny, cloudy, rainy days, wet lenses, fog, early morning, and dusk. Marine data from the Meteorological Department at Honeymoon Bay buoy (121.9291 E, 24.9488 N) are included for wave character prediction. Eight hours of video data taken at Duoliang railway station in eastern Taiwan, shown as location B in Figure 1, during typhoon Krathon in 2024 were utilized for a model accuracy independent test.

3. Methods

3.1. Define Wave Detection and Surge Threshold with Averaged Pixel DN

The pixels in the captured surveillance screen images were analyzed to predict surges; the clipped image in the red square of Figure 1 is automatically transformed into 8 bits greyscale. Normalized averaged digital numbers (DNs) of 62 × 25 pixels from the clipped image are shown as vertical values in Figure 2. The averaged pixel DN value of destination zone without any spray is set as the case of wave strength 0, as shown in the bottom of right panel of Figure 2. The horizontal axis of Figure 2 is time, and the vertical axis is the normalized, observed DN minus the base DN value of no spray, average greyscale pixel DN value in the destination area, which is proportional to the wave magnitude, with an undulation representing an incoming wave. In this example, the threshold value of differenced DN is 90 and the time of DN value over 90 is a surge strike; there are three surges detected within 140 s in this demonstrative example.

3.2. Converting Sliced-Image Pixel DN to Temporal and Spatial Variation in Waves

To calculate the time of waves attacking the destination area, as shown in Figure 3, only the images between the two yellow lines, the wave advancing direction, were processed, that is 972 × 30 pixels of the image. Several wave magnitude factors were calculated according to the pixel DN changes in time and space to predict the incoming waves’ magnitudes. First, considering the image taken at a certain point t, the wave advancing direction is set as X and the direction parallel to the coastline is set as Y. After converting the image to greyscale, the averaged pixel DN value of each Y cross section, 30 pixels, was calculated along each pixel increment, 972 pixels, in the X direction (top panel of Figure 3). This change in averaged DN along the wave’s advancing direction reflects the profile of wave amplitude. Therefore, the degree of pixel DN changes reflects the condition of the wave; the pixel DN values in the X direction are convoluted to quantify the undulating features and enhance the reflected values from the rising portion of the wave, as shown in Figure 4.

The convoluted features of chosen pixels DN corresponding to time t and space x is plotted as a variation diagram in Figure 5. The right margin of the horizontal axis is the position on the shore and to the left is the direction of open sea. The lower left portion of the chart denotes the offshore wave feature in the past and the upper right is the future nearshore wave. The brightness in the dash box reflects the magnitude of the wave, the closer to the shore, the higher the wave. The reciprocal of the dashed line slope is the speed of the wave moving in space (pixel/s) and the distance between the two waves is the reciprocal (1/pixel) of the wave number k. There is a total of 20 s recorded and illustrated in the figure; the lower left portion (70%, thin dash box) is the known input data and the upper right 30%, presented as a thick dashed box, is the corresponding output simulated wave. While the surveillance camera covers enough range, the wave characters that are far away from the destination zone are brought into the proposed LSTM model to predict the wave features while approaching the shore.

The future dynamic parameters of wave magnitude were predicted based on the spatiotemporal variation in the observed wave characteristics. Since this prediction model uses current information to predict the wave attack magnitude in the next few seconds, it is necessary to cover some known information before training the model. As shown in Figure 5, information from the observed 14 s record is the foundation for forecasting the wave character 6 s in the future. First, a spatial–temporal region is demarcated on the characteristic spatial–temporal change map according to the past wave velocity (pixel/s). The endpoint of this spatial–temporal region is the position on the shore at which the surge will strike 6 s later and the slope is the reciprocal of the wave velocity. The wave features are represented as a pixel DN value, ${Pix}^{index}$, conversion ${ Conv}^{index}$, and absolute value ${abs\_Conv}^{index}$.

Furthermore, considering the spatial–temporal region of this calibration, the wave characteristics at different time points are extracted in chronological order from the past to the present to obtain the wave characteristics: ${Pix}_{ti}^{index}, {Conv}_{ti}^{index}, {abs\_Conv}_{ti}^{index}$, t_i = −14~0. In this case, the starting time point is 14 s ahead of the current time, and the sampling time interval is 1/12 s. These time series reflect the changes in wave development characteristics in the past 14 s. Then, the summarized wave characteristics will reflect the total magnitude of the wave at each sampled point during the development process (factors 4 to 6):

$${Pix}_{ts}^{sum}=\sum _{ti=ts-14}^{ts}{Pix}_{ti}^{index}$$

$${Conv}_{ts}^{sum}=\sum _{ti=ts-14}^{ts}{Conv}_{ti}^{index}$$

$${abs\_Conv}_{ts}^{sum}=\sum _{ti=ts-14}^{ts}{abs\_Conv}_{ti}^{index}$$

These changes reflected the fluctuations of wave magnitudes from the past 14 s, expressed by the roughness of the sea’s surface. This value reflects the overall magnitude but does not show the wave dynamic development process. These pre-event time series wave magnitude data, transformed from the image pixel DN values associated with marine data, are the parameters that are inputted into the proposed LSTM model to establish the dynamic evolution pattern; then, this pattern could be extended to predict wave characters in the future.

3.3. Marine Parameter Data for Predicting Wave Magnitude

Marine data measured by the Meteorological Department at nearby weather stations were used to improve the accuracy of the predicted wave magnitude. This observation data were updated hourly from the Honeymoon Bay buoy (121.9291 E, 24.9488 N) of the Meteorological Department, 2.4 km away from the testing site; these included wave height, wave direction, wave period, wind force, maximum wind speed and current velocity data (https://www.cwa.gov.tw/V8/C/M/OBS_Marine_30day.html?MID=OAC005, accessed on 1 October 2024). The tidal height information was obtained from the records of the Wushi tide level station (121.8396 E, 24.8686 N) that is 14.2 km away (https://www.cwa.gov.tw/V8/C/M/OBS_Marine_30day.html?MID=C4U02, accessed on 1 October 2024). Four pieces of included data are shown in Figure 6.

3.4. Data Grouping for Model Training and Testing

The dynamic parameters of wave magnitude calculation via an on shore surveillance camera and the synchronous marine records from the Meteorological Bureau website were inputted into the proposed LSTM model for training and testing. In this study, hourly footage of detected surges was applied in the training, which was 27 h of data, sufficiently covering variations in tide heights, wave heights, wind speeds, sunny, cloudy, and rainy weather, wet lenses, fog, and early morning and dusk.

A total of 70% of the video data was randomly chosen for pattern training and validation and the remaining 30% was set for testing with the proposed LSTM. At first, vague video footage—whenever the pixel DN values in the image over the destination zone were all the same—was ruled out. There were more than thousands of events in which the detected DN value exceeded the threshold of 90 in this database; thousands of no surge events were also included. A total of 11.6 million frames of sample images served as the raw data for training the proposed LSTM model; thus, the diversity of raw information was fulfilled. Since the purpose of this study is to seize the moment when the wave magnitude exceeds the threshold, the magnitude of the approaching wave was analyzed and predicted, as shown in Figure 7. Thus, the information of the descending segment was ignored to avoid unnecessary interference and speed up the processing. The event information in the model training and validation period was resampled to make the data ratio of the ‘surge’ and ‘non-surge’ events equal to 1:1, thus reducing the data imbalance problem in the prediction model.

3.5. The LSTM Model

The training and validation data groups were inputted into the LSTM model with concatenation and two dense networks (Tensor Flow 2.18) to predict future wave magnitude, as shown in Figure 8. Each of the 972-pixel time series transformed from the image were resampled into 275 number sequences for LSTM model computing and the length of the kernel was set as 45. The model integrates the dynamic parameters of wave intensity (time series data) with static meteorological data to predict wave magnitude. The dynamic data are processed through an LSTM layer with 50 units to capture temporal patterns, while the static data are processed separately through a dense layer with 50 neurons and ReLU activation to extract meaningful static relationships. The outputs of these two layers are concatenated and further refined through two fully connected layers: one with 50 neurons and a sigmoid activation function, and another with a single neuron and a linear activation function for final prediction. The model is compiled with the Adam optimizer and has Mean Squared Error (MSE) as the loss function and Mean Absolute Error (MAE) as the evaluation metric, model result of error is shown in Figure 9. Training is conducted over 200 epochs with a batch size of 32, using a 10% validation split. This architecture effectively combines temporal and static features, enabling the model to learn complex dependencies between wave dynamics and meteorological conditions for accurate predictions. The model error and loss become convergent after 40 epochs of computing. In this study, the captured 14 s image records were sampled as 12 Hz information, providing raw data to support 168-epoch model computing, i.e., four times the amount of required data for the model to converge.

4. Results and Validation

4.1. Relationship Between Surge Occurence and Marine Data

The correlation between the marine data and the probability of a surge striking the destination area was examined, and the occurrence of a surge was positively correlated to the tide height, wave height and wind strength. The red asterisks in Figure 10 are the data points that have a surge striking the destination in the hour and the blue asterisks indicate no surge, showing that surges are more likely to occur in the right-upper portion of Figure 10, which is an environment of high tide height, high wave height and strong winds.

This marine data is not updated in real time, and only hourly updated observation records are available for download. As such, there are two possible adjustment options:

Option 1.

Use the latest directly recorded marine parameters.

Option 2.

Automatically capture the marine parameters of the previous 100 h, predict the current marine parameters through the LSTM, and then input the forecasted marine parameters into the surge prediction model.

4.2. Surge Prediction Model Test Results

The first 30% of the hourly video was used as the validation group and the following image was used as the data input unit. The surge magnitude in the destination area at time zero was estimated by the pixel DN value of the chosen image section taken 14 to 6 s before surge strike. There are six various time frame-predicted examples vs. observations, and these are illustrated in Figure 11. Only those events with predicted magnitudes larger than the threshold, the average pixel DN value of 90 in this case, were examined. The confusion matrix for evaluating the exactness of this wave magnitude prediction is shown in

Table 1. Wave prediction confusion matrix in terms of image moments.

Recall = 79% Accuracy = 98%		Prediction
		Surge Event	No Surge Event
Observation	Surge event	1793	481
	No surge event	4861	284,339

, where the recall is 79% and the accuracy is 98%. The definitions of recall and accuracy are listed in

Table 2. Confusion matrix correctness parameter definition for recall and accuracy.

Recall = TP/(TP + FN) Accuracy = (TP + TN)/(TP + TN + FP + FN)		Prediction
		True	False
Observation	True	TP	FN
	False	FP	TN

.

4.3. Wave Prediction Model Testing Results: Case of 6 s Early Warning

The major function of this early warning system is to offer a surge attack alarm ahead of time. First, it must be determined whether there is an incoming wave heading towards the destination zone. The moment for the average pixel DN value over a destination area greater than 15 is defined as the classifying criteria of a wave (Figure 12). When the observed pixel DN value over the destination zone was greater than 90, this was defined as the moment that a ‘surge’ occurred. With this model, when a surge event is predicted to strike the destination zone in 6 s, then a surge warning will be issued. The criterion for determining prediction success is the difference between the surge strike time and the prediction. The accuracy of the 6 s early warning is shown in

Table 3. Surge prediction confusion matrix for 6 s early warning time case.

Recall = 60% Accuracy = 88%		Prediction
		Surge Event	No Surge Event
Observation	Surge event	116	77
	No surge event	55	924

, with a recall of 60% and an accuracy of 88%.

4.4. Effect of Reducing Early Warning Time by 1 s

Figure 13 shows the forecasting lag time of five surge events if the alarm is sent 6 s ahead of the strike; the surge predictions are very close to the observed time, with 1 s of error in striking time. Then, reducing the surge warning time from 6 to 5 s (Figure 14) to predict the same five consecutive surges means that the image from 15 to 5 s prior to the strike time is utilized to predict the wave magnitude in the future 5 s, as shown by the blue line in Figure 14. Therefore, this loss of 1 s of reaction time improved the model prediction accuracy (

Table 4. Surge prediction confusion matrix for 5 s early warning time case.

Recall = 80% Accuracy = 90%		Prediction
		Surge Event	No Surge Event
Observation	Surge event	155	38
	No surge event	79	900

), with the recall increasing from 60% to 80%. The warning time vs. accuracy rate in Figure 15 shows that a 5 s warning time is the most appropriate configuration for a balanced warning time and accuracy rate.

4.5. Validation Model Accuracy with Waves Caused by Typhoons

The severe surges associated with typhoon Krathon, which struck Taiwan (30 September 2024 10:00~17:00), were used for validating model accuracy at other locations and in other environments. Eight hours of video data taken at Duoliang railway station in eastern Taiwan was utilized (https://www.youtube.com/watch?v=UCG1aXVO8H8, accessed on 20 October 2024, Transportation and Tourism Development Department, Taitung County Government), with the location of the destination zone and detected wave pattern shown in Figure 16. The first 30% of the video was used as the test group and the remaining 70% used for model training and validation. The model training and validation data were randomly resampled to achieve data balance; further data randomization was performed in order to choose 90% for training and the other 10% for validation. A surge warning will be issued once a predicted wave magnitude over the destination zone exceeds the threshold, that is DN value of 120, which is 10 s later with the proposed LSTM model. The confusion matrix for evaluating the correctness of the prediction in

Table 5. Surge prediction confusion matrix for incoming wave caused by typhoon.

Recall = 76% Accuracy = 74%		Prediction
		Surge Event	No Surge Event
Observation	Surge event	32	10
	No surge event	40	107

shows a recall of 76% and an accuracy of 74%, so the model prediction result is plausible for a typhoon surge with video taken at location other than model training site.

5. Conclusions

A surge early warning system was developed using real-time coastal video footage, (updated hourly), meteorological data and LSTM machine learning. The LSTM model could preserve a longer memory during network evolution without high-end computing hardware, which means an entry-level PC could offer prediction within the designed time with this model. The system was trained and tested using 110 h of historical data to predict the wave magnitude in the destination area 6 s ahead of the strike; it issued a warning if the predicted wave magnitude was greater than the threshold. The prediction model had a 60% accuracy and 88% recall. Reducing the warning time to 5 s ahead of a strike, we achieved an accuracy of 90% and 80% recall. In the case of surge during typhoons, a surge could be predicted 10 s ahead of strike with an accuracy of 74% and a recall of 76%. This early warning model can be applied to other coasts and provides a reference for the establishment of early warning models and follow-up studies of disaster mechanisms. This reliable and low-cost system performs well in predicting a surge 6 s ahead of time. Eventually, 6 s did not offer a very long-time span for evacuating and taking hazard mitigation action. However, for those who fish on the shore, swim on the beach or drivers who drive on highways adjacent to the ocean, a siren 6 s ahead of the surge could notify them to seek a security facility or reduce their speed to diminish their vulnerability to hazards. There are several ways of improving the accuracy or extent of the warming time in this method; applying pairs of cameras at various incision angles with high-definition and -resolution cameras should improve the data quality and thus elevate the prediction accuracy. The use of near infrared images or radar could detect surges at night.

Since machine learning requires a vast number of events to train and validate, continuous monitoring in an area is required to fully understand the process of wave occurrence and formation characteristics. Adding more videos at various station would also help to enrich the database and thus improve the model’s performance. Both on-site and real-time metrological data are valuable resources for improving the accuracy of the model’s outcome.