Prediction Model for Maritime 5G Signal Strength Based on ConvLSTM-PSO-XGBoost Algorithm

Abstract

The accurate prediction of signal strength plays an important role in estimating radio signal quality, thus forming the essential foundation for the planning, optimization, and reliable operation of modern wireless network systems. This paper proposes a new hybrid model for predicting maritime 5G signal strength, combing Convolutional Long Short-Term Memory (ConvLSTM) with Particle Swarm Optimization-extreme Gradient Boosting (PSO-XGBoost). The model was developed and validated using a dataset comprising 22 columns, 2994 rows, and 21 features, collected via a research vessel in Zhoushan Port, China. Four evaluation metrics, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), and the coefficient of determination (R²) were employed to assess model performance and interpretability. Comparative experiments against various popular models demonstrated the hybrid model’s superior performance in predicting maritime 5G signals. Its accuracy surpassed both standalone ConvLSTM and XGBoost models, while achieving lower MAE and RMSE values compared to various popular models. This study provides a method for predicting coverage conditions based on navigation and environmental data, without relying on radio key performance indicators. Furthermore, it supplies high-quality signal data to advance the modeling of marine communication channels.

1. Introduction

1.1. Background

Over 70% of the Earth’s surface is covered by oceans and seas [1]. With the continuous increase in maritime activities, the demand for reliable maritime communication technologies is also rising. Effective maritime communications are critical for numerous applications [2], including environmental monitoring, offshore oil extraction, wind power generation, maritime surveillance, fisheries, and tourism. Simultaneously, maritime wireless channel modeling forms the core foundation for designing and optimizing marine communication systems. By accurately describing the propagation characteristics of electromagnetic waves in marine environments, it enhances communication reliability, efficiency, and coverage.

Maritime wireless channel modeling presents a challenging research topic, aiming to characterize the channel properties between vessels [3], satellites [4], drones [5], unmanned surface vehicles [6], and other maritime infrastructure. To achieve efficient wireless communication systems, establishing accurate maritime wireless communication channel models is crucial [7]. Currently, maritime wireless channel modeling relies on collected data; yet, acquiring reliable maritime communication data for channel modeling remains challenging. Additionally, maritime 5G signals face complex maritime environments and vessel motion/interference, rendering many of the collected data unusable [8]. For instance, air temperature and humidity affect sea surface reflectivity [9,10] and spatial refractive index [11], thereby affecting signal strength. Concurrently, sea waves influence the dynamic multipath propagation of signals while inducing Doppler shifts and alterations in antenna directivity, thereby causing fluctuations in maritime wireless communications [12]. This complex and dynamic environment poses significant challenges for modeling maritime wireless channels. Therefore, this paper addresses the current difficulty in acquiring data for maritime modeling by providing usable data for maritime wireless channel modeling through predicting 5G signals at sea.

1.2. Related Works

Driven by rapid advancements in computer science and big data, artificial intelligence (AI) and neural networks have evolved significantly. AI technologies offer novel approaches for modeling marine wireless channels. AI applications span machine learning [13], computer vision [14] and natural language processing [15]. These advancements have enabled applications such as autonomous driving [16], image recognition [17] and facial recognition [18]. Some scholars have employed machine learning to predict maritime 5G signal strength in complex marine environments. However, model usability hinges on prediction accuracy. Thus, developing a predictive model capable of accurately forecasting maritime 5G signal strength is essential.

As artificial intelligence systems have gained widespread adoption, common methods for predicting maritime 5G signal strength previously included machine learning-based and deep learning-based spatial feature models. Ahmed [19] elaborated on the application prospects of AI in Industry Internet 4.0 and pointed out that the application prospects of AI at sea are broad. Sánchez et al. [20] investigated Wi-Fi coverage modeling in Checa, Ecuador, using deterministic and empirical propagation models with Raspberry Pi and GPS measurements, demonstrating that a logarithmic regression-adjusted model achieved significantly improved accuracy (path loss exponent of 8.96) for predicting signal strength in community Wi-Fi network planning. Palacios Játiva et al. [21] analyzed the performance of four machine learning algorithms (Coalition Game Theory, Naive Bayesian Classifier, Support Vector Machine, and Decision Trees) for spectrum detection and decision in Cognitive Mobile Radio Networks using Network Simulator 3, demonstrating that SVM outperformed other algorithms across all metrics including probability of detection, probability of false alarm, classification quality, and simulation time. Machine learning-based models can be implemented using Python’s Sklearn package [22] or Tensorflow [23] and PyTorch [24]. Currently, approaches to predicting maritime 5G signal strength can be divided into three categories: traditional machine learning techniques, deep learning machine techniques, and hybrid model techniques.

1.2.1. Traditional Machine Learning Research

Through traditional machine learning techniques serving as a critical means to extract patterns from data for prediction and decision making through algorithms and statistical methods, we have witnessed the sequential emergence of various algorithms throughout their extensive development history. Each algorithm, driven by its inherent characteristics and evolving demands of the era, has collectively advanced the field. In the early stages, linear regression emerged, focusing on modeling linear relationships by constructing linear equations between input and output variables to achieve data fitting and prediction. As data complexity increased, decision tree algorithms were introduced. Decision trees employ a tree-like structure to decompose the decision-making process into a series of feature-based judgment nodes. Starting from the root node, the process branches downward according to different feature conditions, ultimately arriving at leaf nodes to yield decision outcomes. This intuitive tree-based representation renders the decision process transparent, facilitating comprehension and interpretability. Subsequently, to address more complex classification tasks, particularly in high-dimensional data spaces, the Support Vector Regression (SVR) algorithm was developed. SVR is a supervised learning model used for regression problems. It attempts to find an optimal hyperplane such that the distances (errors) from all training data points to this plane are as small as possible, while controlling the complexity of the model to avoid overfitting. Subsequently, XGBoost was developed primarily to address the suboptimal performance of models such as SVR, linear regression, and decision trees in handling certain complex data structures [25]. In signal prediction, these algorithms can effectively extract features and make predictions. For instance, C.Peréz-Wences et al. [26] employed SVR algorithm for power prediction, while Chaudhary, P et al. [27] proposed an enhanced linear regression framework for the common Bernoulli–Gaussian noise problem in wireless communication. Through the gradient descent optimization algorithm, more accurate channel estimation was achieved in non-Gaussian noise environments, providing an efficient solution for 5G/6G communication systems. G. Jangir et al. [28] employed the regression method of the machine learning model decision tree for prediction. However, when dealing with large datasets, traditional machine learning algorithms may suffer from insufficient computational efficiency, limiting real-time prediction capabilities [29]. Furthermore, traditional methods heavily rely on features; if features are not effectively selected and processed, model performance often suffers significantly.

1.2.2. Research on Deep Learning Technologies

Deep learning technology, a branch of machine learning, fundamentally operates through multi-layer neural networks to perform feature extraction and pattern recognition [30]. Historically, deep learning has evolved from early single-layer neural networks to encompass diverse models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and generative adversarial networks (GANs). CNNs excel at processing image data, automatically extracting local features, while RNNs demonstrate significant advantages in handling time-series data, capturing dynamic characteristics of data over time.

Deep learning demonstrates formidable modeling capabilities in signal strength prediction, automatically learning complex data features and proving particularly well suited for processing image and time-series data. For instance, researchers can extract valuable information from a signal’s spatial characteristics using convolutional neural networks, while recurrent neural networks can model temporal variations in signals to achieve more precise predictions. CNNs are employed for prediction, leveraging their powerful feature extraction capabilities to achieve significant improvements in accuracy. By constructing multi-layer CNNs, signal strength across different sea areas can be forecasted, with the model maintaining high predictive precision even in complex environments. Additionally, RNN models excel at processing time-series signals, effectively capturing dynamic characteristics of signal variations—making them particularly well suited for scenarios with frequent changes in marine environments. For instance, Oroza et al. [31]. employed four machine learning models to predict Received Signal Strength Indicator (RSSI). Peng et al. [32]. utilized a deep learning model to predict Received Signal Strength Probability (RSRP), evaluating results using MAE.

Deep learning technology is revolutionizing protocol design in the fields of maritime Internet of Things (IoT) communication and machine learning, with the DL-HEED protocol serving as a prime example. The core innovation of the DL-HEED protocol lies in its utilization of GNNs to capture complex relationships among nodes, integrating multi-dimensional features such as residual energy, node degree, spatial location, and signal strength to enable context-aware dynamic decision making.

In practical maritime communication applications, deep learning-driven protocols also demonstrate immense potential. For instance, Juwaied et al. [33] proposed DL-HEED, a novel deep learning-based clustering protocol that integrates Graph Neural Networks (GNNs) into the HEED algorithm for energy-efficient cluster head selection in heterogeneous wireless sensor networks, achieving up to 60% improvement in network lifetime and energy efficiency compared to traditional HEED through intelligent, context-aware decision making using node features including residual energy, degree, spatial position, and signal strength. Rigas et al. [34] proposed an end-to-end deep learning framework for fault detection in marine machinery, leveraging Graph Attention Networks (GATs) to extract spatio-temporal information from multivariate time-series sensor data, providing explainable predictions through a feature-wise scoring mechanism and addressing maritime-specific challenges including scarce labeled data and proprietary datasets through a scalable edge-to-cloud architecture using MQTT protocol for data collection and a custom evaluation metric balancing prediction accuracy and fault identification timeliness. In marine IoT semantic communication, Han et al. [35] proposed a Turbo-based deep semantic auto encoder that combines Transformers with Turbo coding to achieve efficient text transmission under low signal-to-noise ratio conditions. Hussain et al. [36] proposed the Cooperative-Relay Neighboring-Based Energy Efficient Routing (CR-NBEER) protocol for Marine Underwater Sensor Networks, implementing advanced relay optimization and cooperative routing schemes that achieved superior performance compared to existing protocols, including Co-UWSN, CEER, and NBEER, with simulation results demonstrating reduced end-to-end delay (19.3 ms average), lower energy consumption (7.644 j average), a higher packet delivery ratio (94.831% average), and reduced path loss (79.9 dB average). These studies demonstrate that deep learning technologies effectively address challenges such as energy constraints, high dynamic variability, and stringent reliability requirements in maritime/IoT communication environments through intelligent routing selection, data aggregation, and anomaly detection mechanisms. These advancements provide technical support for the construction of efficient, reliable, and intelligent marine information systems.

Although deep learning technologies demonstrate numerous advantages in marine 5G signal strength prediction—such as high accuracy and adaptive capabilities—they still face considerable challenges. Compared to traditional machine learning methods, deep learning models typically require substantial data and powerful computational resources, with particularly significant computational resource consumption during the training phase [37]. Furthermore, deep learning models exhibit relatively low interpretability, which may pose difficulties in practical applications, especially when explanations and analyses of prediction results are required.

1.2.3. Research on Hybrid Model Techniques

With the exponential growth of training datasets, individual deep learning models struggle to handle such complex and massive data. Concurrently, hybrid models adapt better to increasingly complex data by integrating multiple deep learning models. Hybrid models offer the following advantages: First, by combining the strengths of multiple individual models, hybrid models achieve breakthrough improvements in predictive performance [38]. Second, hybrid models enhance adaptability to complex data and distribution shifts through diverse architectural designs and decision strategies. Third, while improving performance, hybrid models substantially reduce computational costs [39]. Fourth, ensemble models demonstrate exceptional task adaptability and flexibility. Therefore, many studies began to use hybrid models to predict signal strength. Abimannan et al. [40] found that hybrid learning methods have become powerful approaches to overcome the limitations of single learning models in terms of generalization, robustness, and performance. Meti [41] used an ensemble model to predict throughput in the deployment of a Wireless Local Area Network (WLAN).

Despite the increasingly widespread application of fusion models in maritime communication, research on the prediction of maritime 5G signal strength remains relatively scarce. Accurate prediction of wireless signal strength is of paramount importance for maritime wireless channel modeling, 5G network planning, mobility management, and resource scheduling. Meanwhile, most previous studies have relied on statistical methods or traditional machine learning models, failing to fully account for the complex and dynamic environmental factors in real-world scenarios, which results in limitations in prediction accuracy and applicability.

As a pioneering effort to address these challenges, this paper makes the following contributions: Firstly, the proposed ConvLSTM- PSO-XGBoost fusion model stands out conceptually for its systematic three-stage decomposition design. Initially, ConvLSTM is employed to extract spatiotemporal correlation features from raw signal sequences. This choice outperforms pure time-series models (e.g., TCN) and pure attention-based models (e.g., Transformer), offering lower computational overhead and better data efficiency in short-term high-frequency prediction tasks compared to Transformer. Secondly, a PSO-optimized XGBoost module is introduced for residual correction. The global search capability of PSO effectively avoids local optima traps in the high-dimensional hyperparameter space of XGBoost, significantly enhancing model generalization performance compared to default parameters or grid search strategies. Moreover, XGBoost’s regularization mechanism and tree ensemble structure demonstrate unique noise resistance advantages in handling tabular environmental features and fitting residuals unexplained by ConvLSTM. Finally, an entropy-weighted fusion is adopted to dynamically adjust the output weights of the two sub-models. This design surpasses traditional fixed-weight fusion and computationally expensive attention-based fusion mechanisms by quantifying the uncertainty of each model’s predictions in real time to allocate confidence levels, thereby improving prediction accuracy and robustness.

Overall, ConvLSTM is responsible for spatiotemporal pattern modeling, PSO-XGBoost handles nonlinear residual refinement, and entropy-based fusion manages adaptive weight scheduling. Together, they form a comprehensive solution that achieves improved accuracy, enhanced robustness, and preserved interpretability. Compared to alternative architectures such as Transformer and TCN, this solution demonstrates clearer engineering value and applicability boundaries in real-world deployment scenarios characterized by limited data and complex environments.

2. Maritime 5G Signal Strength Prediction Model

2.1. Particle Swarm Optimization Algorithm

PSO algorithm is a swarm-based stochastic optimization technique [42]. It simulates the foraging behavior of flocks, where particles move through the solution space to find optimal solutions [43]. Several critical parameters require configuration and selection in the PSO algorithm, such as the inertia weight and acceleration coefficient. These parameters determine how particle velocity and position are updated, significantly impacting algorithm performance and convergence. The particle velocity update formula and position update formula are as follows:

$$V_{id}^{k+1}=\omega V_{id}^k+C_1{r}_1(P_{id}-X_{id}^k)+C_2{r}_2(P_{gd}-X_{id}^k)$$

(1)

$$X_{id}^{k+1}=X_{id}^k+V_{id}^{k+1}$$

(2)

where $V_{id}^{k+1}$ denotes the particle’s velocity after the kth iteration, $X_{id}^{k+1}$ primarily represents the particle’s position after the kth iteration, k denotes the iteration count, ω represents the inertia weight, C₁ and C₂ represent the learning factors, r₁ and r₂ denote acceleration factors, and P_id and P_gd, respectively, denote the local optimum of the particle swarm. Figure 1 elaborates on the workflow of PSO in detail.

2.2. XGBoost Model

XGBoost is an optimized implementation of Gradient Boosting Trees (GBDT), enhancing performance through the regularization of the objective function and parallel computation. By employing ridge regression, the model minimizes loss and variance while preserving objective function information.

The XGBoost model predicts final results through an additive expression composed of K (number of trees) base models, as follows:

$${\widehat{y}}_i={\displaystyle \sum _{k=1}^K{f}_k}(x_i)$$

(3)

where x_i represents the ith sample, ŷ_i denotes the predicted output for the ith sample, k indicates the number of base models, and f_k represents the kth base model.

The associated loss function takes the following form:

$$L={\displaystyle \sum _{i=1}^{n}l}(y_i,{\widehat{y}}_i)$$

(4)

where y_i is the true value corresponding to the ith sample, ŷ_i denotes the predicted output for the ith sample, and L is the loss function.

The objective function is defined as follows:

$$Ob{j}^{(k)}={\displaystyle \sum _{i=1}^{n}l}[y_i,{\widehat{y}}_i^{(k-1)}]+f_t(x_i)+{\displaystyle \sum _k\Omega }(f_k)$$

(5)

$$\Omega (f)=\gamma T+{\displaystyle \frac{1}{2}}\lambda \left|\right|w|{|}^2$$

(6)

where ŷ_i^(k−1) is the sum of the outputs from the previous (k − 1) trees; f_k(x_i) is the output of the kth tree; and l is the differentiable convex loss function measuring the discrepancy between the predicted value ŷ_i and the true value y_i. Ω(f) is the penalty term for model complexity; γ is the regularization parameter for the number of leaves; λ is the regularization parameter for leaf weights; w is the value of the leaf node; and T is the number of leaf nodes.

2.3. PSO-XGBoost Model

PSO-XGBoost is a hybrid model combining the PSO with XGBoost. It enhances model prediction performance by optimizing XGBoost hyperparameters via PSO. Its core idea is to leverage PSO’s global search capability to identify optimal XGBoost hyperparameter combinations, addressing the inefficiency and local optima trapping issues of traditional grid search. Figure 2 elaborates on the workflow of the PSO-XGBoost model.

2.4. Convolutional Long Short-Term Memory Model

ConvLSTM was proposed by Xingjian Shi et al. [44] to address spatio-temporal sequence prediction problems. Its core innovation replaces the fully connected operations in traditional LSTMs with convolutional operations, enabling the model to simultaneously capture temporal dependencies and spatial features. This makes it suitable for spatio-temporal structured data such as radar echo map prediction and video frame generation.

ConvLSTM inherits the gating mechanism of LSTM, but all inputs, hidden states, and cell states are 3D tensors (channel × height × width), enabling local feature interactions through convolutional operations.

The input gate i_t controls the proportion of current input data X_t stored in the memory cell C_i, as detailed in the following formula:

$$i_t=\sigma (W_{xi}\ast X_t+W_{hi}\ast H_{t-1}+W_{ci}°C_{t-1}+b_i)$$

(7)

where * denotes convolution; ∘ denotes Hadamard product; σ() denotes the sigmoid activation function; W_xi represents the weight for the current input X_t; W_hi denotes the weight for the previous hidden state H_t−1; W_ci denotes the weight for the previous memory cell C_t−1; and b_i represents the threshold for the input gate.

The forget gate f_t controls the proportion of the previous memory cell state C_t−1 preserved into the current memory cell C_t, with the specific formula as follows:

$$f_t=\sigma (W_{xf}\ast X_t+W_{hf}\ast H_{t-1}+W_{cf}°C_{t-1}+b_f)$$

(8)

where W_xf is the weight of the current input X_t; W_hf is the weight of the previous hidden state H_t−1; W_cf is the weight of the previous cell state C_t−1; and b_f is the respective thresholds for the forget gates.

The output gate o_t controls the proportion of the current memory cell state C_t stored in the ConvLSTM network’s output value H_t, as expressed by the following formula:

$$o_t=\sigma (W_{xo}\ast X_t+W_{ho}\ast H_{t-1}+W_{co}°C_t+b_o)$$

(9)

where W_xo is the weight for the current input X_t; W_ho is the weight for the previous hidden state H_t−1; W_co is the weight for the previous memory cell C_t−1; and b_o represents the threshold for the forget gate.

The memory cell C_t is obtained through a nonlinear transformation involving the output of the forget gate, the state of the memory cell at the previous time step, the input gate, the current input, and the hidden state at the previous time step:

$${\tilde{C}}_t=\tanh (W_{xc}\ast X_t+W_{he}\ast H_{t-1}+b_c)$$

(10)

$$C_t=f_t°C_{t-1}+i_t°{\tilde{C}}_t$$

(11)

where W_xc represents the weight of the current input X_t; W_he represents the weight of the previous hidden state H_t−1; and b_c denotes the threshold of the memory cell.

The hidden state H_t is obtained by applying the Hadamard product to the output of the output gate and the output of the Tanh activation function applied to the current memory cell state C_t.

$$H_t=o_t°\tanh (C_t)$$

(12)

Figure 3 illustrates the fundamental operational principle of the ConvLSTM architecture.

2.5. ConvLSTM-PSO-XGBoost Model

The prediction sets generated by ConvLSTM and PSO-XGBoost are fused using an entropy-weighted fusion method to combine the predictions from both models into new prediction data. The formulas for calculating the prediction error and weights in the fusion process are as follows.

$${\omega }_1={\displaystyle \frac{\frac{1}{er{r}_1}}{\frac{1}{er{r}_1}+\frac{1}{er{r}_2}}}, {\omega }_2=1-{\omega }_1$$

(13)

$$y_{ensemble}=w_1\times y_1+w_2\times y_2$$

(14)

where err₁ = |y_true − y₁| + 10⁻⁸ represents the prediction error of Model 1 and err₂ = |y_true − y₂| + 10⁻⁸ is the prediction error of Model 2. w₁ and w₂ represent the weights of the two models. y₁ and y₂ represent the predictions of the two models. y_ensemble is the final prediction value after fusion.

Figure 4 illustrates the comprehensive architecture of the ConvLSTM-PSO-XGBoost model in detail.

3. Model Training

3.1. Experimental Data Collection

The experimental data in this paper are all collected from a scientific research vessel operating along the coast of Zhoushan City, Zhejiang Province, in the East China Sea. The testing was conducted from 07:30 to 14:48 on 29 April 2025. Data were collected over a total sampling duration of approximately seven hours at fixed 10-s intervals, using a fixed frequency of 700 MHz (Band 28). The vessel’s voyage commenced within the dashed-line area depicted in Figure 5. The vessel completed a total voyage of 63.14 nautical miles, equivalent to 116.93 km. The experimental equipment is a maritime 5G terminal developed by Super Radio. This terminal is specifically engineered for maritime environments, featuring high precision and low power consumption. The 5G terminal automatically switches frequency bands based on current network conditions. It transmits real-time data from various sensors, including a 9-axis motion sensor (comprising a 3-axis gyroscope, 3-axis accelerometer, and 3-axis compass), and is equipped with a Global Positioning System (GPS).

3.2. Feature Variable Selection

To accurately predict the target variable RSRP, it is essential to both understand the vessel motion attitude (roll, pitch, yaw, surge, sway, heave, etc.) and surrounding environment information (temperature, field strength, distance, etc.). Fifteen relevant influencing factors are listed below, followed by a brief description.

x₁: surge, measured by an accelerometer to reflect the linear acceleration of the vessel in the X-axis direction, indicating the longitudinal motion state (forward/backward acceleration) of the vessel. The unit is usually g (9.8 m/s²).

x₂: sway, measured by an accelerometer to reflect the linear acceleration of the vessel in the Y-axis direction, reflecting the lateral motion state of the vessel (acceleration of left and right swaying), and is used to monitor the force on the hull and the stability of navigation.

x₃: heave, measured by an accelerometer to reflect the linear acceleration of the vessel in the Z-axis direction, reflecting the vertical motion state of the vessel (up and down jolting acceleration). It is commonly used for wave height estimation and navigation comfort assessment.

x₄: roll, the angular velocity of the ship’s rotation around the X-axis, reflecting the rate of change of the longitudinal tilt of the hull. The unit is typically degrees per second (°/s) or radians per second (rad/s).

x₅: pitch, the angular velocity of the ship’s rotation around the Y-axis, reflecting the speed of the ship’s heading change. The unit is usually degrees per second (°/s) or radians per second (rad/s).

x₆: yaw, the angular velocity of the ship’s rotation around the Z-axis, reflecting the speed of the ship’s lateral tilt change. The unit is usually degrees per second (°/s) or radians per second (rad/s).

x₇: roll angle, the angle of rotation around the X-axis, reflecting the longitudinal tilt of the hull. The unit is typically degrees (°).

x₈: pitch angle, the angle of rotation around the Y-axis, reflecting the degree of deviation from the vessel’s course.

x₉: heave angle, the angle of rotation around the Z-axis, reflecting the degree of lateral tilt of the hull.

x₁₀: longitudinal magnetic (LM) field, measured by a magnetometer to determine the intensity of the Earth’s magnetic field in the X-axis direction of the vessel’s coordinate system. It is commonly used for course calculation and attitude correction, with units typically being Tesla (T) or Gauss (Gs).

x₁₁: athwartship magnetic (AM) field, the intensity of the Earth’s magnetic field in the Y-axis direction of the vessel’s coordinate system.

x₁₂: vertical magnetic (VM) field, the intensity of the Earth’s magnetic field in the Z-axis direction of the vessel’s coordinate system. It can be used to detect electromagnetic interference in the environment and assist in the calibration of the heading sensor.

x₁₃: surrounding temperature (ST), the measurement of the air temperature around a vessel, typically expressed in °C or °F.

x₁₄: ship speed (SS), the speed at which a vessel moves through water, typically measured in knots.

To extract the most critical input variables for predicting 5G signal strength, a feature selection strategy combining Pearson correlation coefficient analysis with the PSO algorithm was employed. The Pearson correlation coefficient [45], also known as the Pearson product–moment correlation coefficient, measures the linear correlation between two variables X and Y, with values ranging from −1 to 1. A value of 1 indicates perfect positive correlation, −1 indicates perfect negative correlation, and 0 indicates no linear relationship between the variables.

For two variables X = {x₁, x₂, …, x_n} and Y = {y₁, y₂, …, y_n}, the Pearson correlation coefficient r is calculated as follows:

$$r={\displaystyle \frac{n{\displaystyle \sum x_i}{y}_i-{\displaystyle \sum x_i}{\displaystyle \sum y_i}}{\sqrt{n{\displaystyle \sum x_i^2}-{({\displaystyle \sum x_i})}^2}\sqrt{n{\displaystyle \sum y_i^2}-{({\displaystyle \sum y_i})}^2}}}$$

(15)

where r represents the correlation coefficient, x_i represents the feature value of the ith sample in the input features, and y_i represents the feature value of the ith sample in the output features.

As shown in Figure 6 (Pearson correlation heatmap), the LM field exhibits the highest correlation coefficient. Based on feature scoring, surge and heave were excluded in subsequent tests, retaining the remaining data.

3.3. Data Processing

Data preprocessing is preparatory work conducted before testing, including verifying dataset integrity, removing outliers, standardizing data, and partitioning the dataset. Standardization ensures each feature’s physical quantity is on the same order of magnitude, thereby simplifying subsequent experiments. The preprocessed dataset contains no missing values or outliers.

The data processing employed normalization, a technique prevalent in deep learning, which mitigates the likelihood of gradient explosion, accelerates convergence, stabilizes the training process, and ultimately enhances model performance. The objective of normalization is to transform the data into a distribution with a mean of zero and a standard deviation of one, thereby aligning it with the standard normal distribution. Consequently, prior to model training, all raw data—excluding binary variables and dummy variables—were normalized using the following formula:

$$z={\displaystyle \frac{x-\mu }{\sigma }}$$

(16)

where z is the standardized value, x is the raw feature value, μ is the feature mean and σ is the feature standard deviation.

The comparison between the standardized processed data and the raw data is shown in

Table A1. Comparison between standardized and raw data following data processing.

Feature	State	Mean	Standard Deviation	Minimum	Maximum	Median
Surge	Raw	−0.044	0.013	−0.098	0.01	−0.044
	Standardized	5.092	1.000	−4.089	4.093	0.001
Sway	Raw	−0.051	0.013	−0.097	−0.002	−0.052
	Standardized	4.368	1.000	−3.486	3.821	−0.025
Heave	Raw	1.012	0.018	0.941	1.081	1.013
	Standardized	−1.969	1.000	−3.836	3.663	0.020
Roll	Raw	−2.886	0.540	−4.839	−0.428	−2.884
	Standardized	−1.319	1.000	−3.615	4.552	0.004
Pitch	Raw	2.453	0.464	0.818	4.026	2.461
	Standardized	−1.756	1.000	−3.517	3.383	0.017
Yaw	Raw	74.586	1.630	70.763	82.524	74.586
	Standardized	−5.431	1.000	−2.345	4.868	−0.001
Roll angle	Raw	0.0156	0.314	−1.099	1.221	0.000
	Standardized	1.899	1.000	−3.543	3.832	−0.049
Pitch angle	Raw	−0.011	0.505	−1.892	1.892	0.000
	Standardized	1.305	1.000	−3.719	3.765	0.022
Heave angle	Raw	0.003	0.132	−0.549	0.671	0.000
	Standardized	2.136	1.000	−4.163	5.032	−0.025
LM field	Raw	20.224	4.274	11.230	27.539	22.168
	Standardized	0.000	1.000	−2.104	1.711	0.454
AM filed	Raw	7.261	1.237	3.32	10.254	7.812
	Standardized	−6.077	1.000	−3.184	2.417	0.444
VM field	Raw	−43.681	0.937	−46.973	−37.793	−43.457
	Standardized	−1.215	1.000	−3.510	6.278	0.238
ST	Raw	35.116	1.097	34.1	38.9	34.5
	Standardized	6.533	1.000	−0.926	3.449	−0.561
SS	Raw	16.901	8.944	0.000	91.129	20.651
	Standardized	−2.279	1.000	−1.889	8.300	0.419
Dis	Raw	10.833	11.423	0	46.286	4.980
	Standardized	7.600	1.000	−0.94849	3.103	−0.512

in Appendix A.

3.4. Hyperparameter Tuning

In the model training process, 20% of the data are designated as the test set, which is exclusively utilized for data prediction to preclude any interference from feature selection and model evaluation on the predictive outcomes. The remaining 80% of the data are allocated as the training set. A temporal partitioning approach is employed for dataset construction, with a window length of five time steps, thereby eliminating temporal overlap and leakage between the test set and the training set.

The optimization of hyperparameters proceeded in two phases. The initial phase focused on configuring the ConvLSTM architecture and its associated training parameters. Subsequently, the XGBoost model’s hyperparameters were automatically optimized via the PSO algorithm. The ConvLSTM model employed a three-layer architecture with progressively decreasing filter sizes (32, 16, and 8) to capture spatiotemporal features at multiple scales. Each ConvLSTM2D layer utilized a 3 × 3 kernel with appropriate padding to preserve spatial dimensions. The input tensor shape was configured as (5, 1, 15, 1) to accommodate the temporal sequence length and spatial dimensions. A fully connected layer with 64 ReLU-activated neurons followed by a single output neuron was employed for the final prediction. Training was conducted using the Mean Squared Error (MSE) loss function with a batch size of 32 and a maximum of 150 epochs. To prevent overfitting, we implemented early stopping with a patience of 10 epochs, monitoring the validation loss. The training would terminate if the validation loss failed to improve for 10 consecutive epochs.

For the XGBoost model, we implemented the PSO algorithm to automatically search for optimal hyperparameters. The PSO algorithm was configured with a swarm size of 20 particles and a maximum of 30 iterations. The algorithm parameters were set as follows: inertia weight, individual learning factor, and global learning factor. The minimum fitness threshold was set to 0.001.

Figure 7 displays the training curves for ConvLSTM. ConvLSTM employs one-dimensional feature vectors (15 features) and transforms them into a pseudo-spatial representation by reshaping the original three-dimensional tensor of dimensions (number of samples, 5, 15) into a five-dimensional tensor of dimensions (number of samples, 5, 1, 15, 1). The figure shows two curves: training loss represents iterations during training, while verification loss indicates iterations during prediction. It is evident that convergence is achieved around iteration 10, with MSE loss stabilizing in subsequent iterations.

By employing the PSO algorithm to optimize XGBoost hyperparameters, the optimal parameter ranges for XGBoost were ultimately determined, as shown in

Table 1. PSO-Optimized XGBoost parameters.

Parameter	Range	Physical Meaning
max_depth	[3, 10]	Tree depth, controls model complexity
learning_rate	[0.01, 0.3]	Learning rate, balancing iteration step size
n_estimators	[100, 500]	Number of trees, affecting fitting capability
min_child_weight	[1, 10]	Minimum sample weight sum for leaf nodes
subsample	[0.5, 1]	Sample subsampling ratio to prevent overfitting

.

Through iterative refinement, the optimal parameters for PSO-XGBoost and ConvLSTM are presented in

Table 2. Model hyperparameters.

Model Name	Parameter
PSO-XGBoost	n_estimators = 201, learning_rate = 0.03093
ConvLSTM	filters = 32, 16, 8, kernel_size = (3, 3), dropout = 0.3
Support Vector Regression	kernel = ‘rbf’, C = 1, gamma = ‘scale’, epsilon = 0.01
KNeighbors Regressor	n_neighbors = 7, weights = ‘distance’, algorithm = ‘auto’, leaf_size: 30
Linear Regression	intercept_: −69.0949790794979
Lasso Regression	alpha = 0.001, max_iter = 1000, tol = 0.0001
Ridge Regression	alpha = 0.001, max_iter = 1000, tol = 0.0001
ElasticNet Regression	alpha = 0.001, l1_ratio = 0.1, max_iter = 1000
Decision Tree Regression	criterion: absolute_error, max_dept: 10, min_samples_leaf: 4, min_samples_split: 10

. Other models employ network search methodologies to identify the parameter configurations that yield optimal performance in predictive tasks. For precise definitions of each hyperparameter, refer to the documentation of corresponding functions in scikit-learn.

3.5. Evaluation Metrics

This paper mainly adopts R², MAE, RMSE, and MAPE as the evaluation metrics of the prediction results. The calculation formulas are as follows:

$$R^2=1-{\displaystyle \frac{{\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}}{{\displaystyle \sum _{i=1}^n{(y_i-\overline{y})}^2}}}$$

(17)

$$MAE={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n\left|y_i-{\widehat{y}}_i\right|}$$

(18)

$$RMSE=\sqrt{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n{(y_i-{\widehat{y}}_i)}^2}}$$

(19)

$$MAPE={\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n\left|\frac{y_i-{\widehat{y}}_i}{y_i}\right|}\times 100\%$$

(20)

where y_i is actual value, ŷ_i is predicted value, $\overline{y}$ is average of actual values, and n is the number of test set samples.

3.6. ConvLSTM-PSO-XGBoost Model Fusion Approach

ConvLSTM effectively extracts spatio-temporal correlation features from data through convolutional operations and gating mechanisms. Meanwhile, PSO-XGBoost enhances the efficiency of traditional XGBoost in handling high-dimensional features and modeling nonlinear relationships by optimizing hyperparameters via the PSO algorithm. Due to the significant heterogeneity in model architecture and characteristics between the two, the fusion model demonstrates multiple outstanding advantages when dealing with complex data scenarios. Specifically, it exhibits complementarity. ConvLSTM can effectively capture long-term temporal dependencies within the data by virtue of its unique structure, delving into the underlying patterns of data changes over time. Meanwhile, PSO-XGBoost excels at handling complex interactions among high-dimensional features, accurately grasping the information embedded in different feature combinations. These two models complement each other, jointly providing a more comprehensive and in-depth perspective for the analysis of complex data. On the other hand, the fusion model demonstrates remarkable robustness. Single models are often prone to interference when confronted with noisy data, leading to deviations in analysis results. In contrast, the fusion model, by integrating the strengths of different models, can effectively reduce its sensitivity to noisy data. Even in the presence of a certain degree of noise interference in the data, it can still maintain relatively stable and reliable performance. Furthermore, the fusion model possesses strong generalization ability. Through the adoption of an ensemble strategy, it organically combines the advantages of multiple models, avoiding the overfitting issues that may occur in single models and reducing the risk of overfitting. As a result, it can achieve favorable application results across a wider range of data distributions and scenarios.

The fusion achieves information complementarity by concatenating or transforming intermediate features from ConvLSTM and PSO-XGBoost. Its core lies in constructing a unified feature space. An effective model fusion method directly impacts the fusion results of ConvLSTM and PSO-XGBoost. By testing different fusion approaches and comparing the MAE and RMSE of the final fused models, the optimal fusion method is selected. The optimal fusion method for ConvLSTM-PSO-XGBoost is entropy-weighted fusion. Comparison diagrams for each fusion method are shown in Figure 8.

4. Experiments and Analysis

4.1. Experimental Overview

The experiments were conducted on a PC running the Windows operating system and equipped with an AMD Ryzen 7 5800H processor (Lenovo, Beijing, China). Programming was performed using Python 3.13. To evaluate the performance of ConvLSTM-PSO-XGBoost, we compared it with seven other models.

A comparative analysis of 20 experimental trials demonstrates that the integrated model yields optimal predictive results.

▪

When compared to linear regression (LR), RMSE decreased by 0.3268 (58.42%); MAE decreased by 0.3076 (66.93%).

▪

When compared to Lasso Regression (Lasso), RMSE decreased by 0.3274 (58.46%); MAE decreased by 0.3080 (66.96%).

▪

When compared to ridge regression (Ridge), RMSE decreased by 0.3281 (58.52%); MAE decreased by 0.3086 (67.00%).

▪

When compared to ElasticNet Regression (ElasticNet), RMSE decreased by 0.3274 (58.46%); MAE decreased by 0.3080 (66.96%).

▪

When compared to Decision Tree Regression (DTR), RMSE decreased by 0.1109 (32.29%); MAE decreased by 0.0707 (31.75%).

▪

When compared to KNeighbors Regressor (KNR), RMSE decreased by 0.2892 (55.42%); MAE decreased by 0.2566 (62.80%).

▪

When compared to Support Vector Regression (SVR), RMSE decreased by 0.2071 (47.10%); MAE decreased by 0.1980 (56.57%).

The deviation between the predictions of the proposed model and the actual values compared against these seven models is illustrated in

Table 3. Performance evaluation of 10 models.

Model	R2	RMSE	MAE	MAPE
LR	0.674	0.559	0.460	8.669
Lasso	0.673	0.560	0.460	8.688
Ridge	0.672	0.561	0.461	8.712
ElasticNet	0.673	0.560	0.460	8.687
DTR	0.877	0.344	0.227	4.043
KNR	0.716	0.522	0.409	7.796
SVR	0.798	0.440	0.350	6.465
Proposed	0.944	0.233	0.152	2.745

.

The prediction curve of the proposed model is shown in Figure 9. The comparison between the ideal line, confidence interval, and actual prediction points in the figure demonstrates outstanding performance in regions with strong signal strength (−40 to −60 dBm).

Figure 10 presents the model evaluation metrics, highlighting that the ConvLSTM-PSO-XGBoost model achieved an R² value of 0.944, an RMSE of 0.233, an MAE of 0.152, and an MAPE of 2.745. These results demonstrate that this model outperforms the other nine models evaluated.

4.2. Analytical Study

To comprehensively validate the effectiveness of each component within the proposed ConvLSTM-PSO-XGBoost integrated model, extensive ablation experiments were conducted. All experiments were performed under identical training conditions to ensure a fair comparison. The necessity of the hybrid architecture was evaluated by comparing the performance of individual components with that of the fully integrated system. This study assessed the respective contributions of LSTM, ConvLSTM, PSO-optimized XGBoost, and their integrated framework.

Experimental process can be described as follows:

(1) XGBoost (PSO) only: evaluate the performance of XGBoost alone when optimized by PSO (without ConvLSTM).

(2) ConvLSTM only: evaluate the performance of ConvLSTM alone.

(3) XGBoost (random parameters): evaluate XGBoost with manually set (non-optimized) hyperparameters.

(4) LSTM only: evaluate LSTM (without convolutional layers) for spatiotemporal tasks.

(5) Complete system (ConvLSTM + PSO-XGBoost Fusion): evaluate the full hybrid model.

Table 4. Results of ablation experiments.

Model	R2	RMSE	MAE	MAPE
PSO-XGBoost	0.883	0.344	0.259	4.835
ConvLSTM	0.865	0.361	0.261	5.174
XGBoost	0.795	0.443	0.353	6.807
LSTM	0.795	0.444	0.363	6.783
ConvLSTM-PSO-XGBoost	0.944	0.233	0.152	2.745

presents a comparative analysis of the performance across various model architectures. The ConvLSTM-PSO-XGBoost ensemble achieved the optimal performance, with a Mean Absolute Error (MAE) of 0.1446. The use of XGBoost with PSO alone yielded an MAE of 0.2609, while the standalone ConvLSTM model attained an MAE of 0.2585. Among all variants, the LSTM model exhibited the lowest performance, with an MAE of 0.3633.

Figure 11 presents the prediction outcomes of various models from the ablation study. The ablation experiments revealed several important findings. Both models are necessary: neither XGBoost nor ConvLSTM alone can achieve the performance of the ensemble, confirming the complementarity of tree-based models and sequence-based models. Additionally, PSO optimization is effective: the performance improvement from random to PSO-optimized parameters demonstrate the effectiveness of particle swarm optimization in hyperparameter tuning. Besides, hybrid model outperforms single models: the fusion of ConvLSTM and XGBoost produced the best performance, validating the hypothesis that combining different modeling paradigms can capture more comprehensive patterns.

4.3. Feature Importance Analysis

This study employs the feature importance analysis method based on XGBoost to evaluate the contribution of each feature to RSRP prediction using the gain metric. This approach calculates the reduction value of the loss function brought by each feature at every split in the decision trees, with the computational formula as follows:

$$\mathrm{Importance}(f)={\displaystyle \sum _{t\in T}\frac{Gai{n}_t}{N_{\mathrm{f}}}}$$

(21)

where Gain_t represents the objective function improvement achieved by splitting on feature f in the i tree, while N_f denotes the total number of times this feature is utilized for splitting.

Table 5. The importance scores of the features.

Feature	Score
LM	0.0919
ST	0.0903
Dis	0.0873
AM	0.0840
Heave Angle	0.0823
SS	0.0803
VM	0.0780
Pitch Angle	0.0722
Roll Angle	0.0661
Roll	0.0610
Yaw	0.0559
Heave	0.0498
Surge	0.0446
Sway	0.0372
Pitch	0.0193

displays the importance scores of the features. According to the table, the top three contributing features are LM (9.2%), ST (9.0%), and Dis (8.7%).

Figure 12 presents the results of the feature importance scoring analysis.

4.4. Performance Analysis Under Different Conditions

To evaluate the model’s generalization capability and understand its performance boundaries, we conducted a comprehensive analysis of prediction accuracy under varying operational conditions.

Table 6. Scores of the model under various conditions.

Distance	Speed	R2	RMSE	MAE	MAPE
20	10	0.949	0.344	0.257	2.601
40	18	0.901	0.383	0.287	2.067
60	25	0.861	0.417	0.312	5.741
40	10	0.924	0.401	0.300	3.110
40	18	0.885	0.447	0.334	4.922
40	25	0.853	0.487	0.364	5.827
60	10	0.882	0.458	0.343	4.963
60	18	0.841	0.510	0.382	5.897
60	25	0.814	0.556	0.416	7.422

presents comparative experiments that were conducted at varying speeds while maintaining a fixed position relative to the shoreline. Specifically, at distances of 20, 40, and 60 km offshore, vessel operations were carried out at speeds of 10, 18, and 25 km/h, respectively. As illustrated by the data, the model’s prediction accuracy exhibits an overall declining trend as the distance increases from 20 to 60. The R² value decreases from 0.9494 to 0.8135, while error metrics such as RMSE, MAE, and MAPE all show an upward trajectory. Under a constant distance, an increase in speed (from 10 to 25) also contributes to a reduction in R² and an amplification of errors. For instance, at a distance of 40, R² declines from 0.9238 to 0.8525, and MAPE rises from 3.09% to 5.83%. Comprehensive analysis indicates that the model’s prediction accuracy is negatively correlated with both navigation distance and speed, with optimal performance observed under conditions of short distance and low speed. The experimental results are presented as follows.

4.5. Model Scoring

To compare the utility of the nine evaluation metrics, a model scoring process was designed. By categorizing metrics into positive and negative indicators, the scoring formula was designed as follows:

$$S_{\mathrm{P}}={\displaystyle \frac{x-\min (x)}{\max (x)-\min (x)+\epsilon }}\times 0.8+0.2$$

(22)

$$S_{\mathrm{N}}={\displaystyle \frac{\max (x)-x}{\max (x)-\min (x)+\epsilon }}\times 0.8+0.2$$

(23)

where S_P is the normalization formula for positive indicators and S_N is the negative indicator normalization formula (lower values yield higher scores). x represents the raw metric value, min(x) is the minimum value of that metric, max(x) is the maximum value of that metric, and ε is a small constant (e⁻¹⁰) to avoid division-by-zero errors.

The normalized score standard prioritizes values closest to 1.0 as optimal. Scoring results are shown in

Table 7. Model scoring results.

Model	R2	RMSE	MAE	MAPE
PSO-XGBoost	0.820	0.730	0.724	0.720
ConvLSTM	0.767	0.688	0.718	0.674
Linear Regression	0.204	0.203	0.203	0.206
Lasso Regression	0.202	0.202	0.202	0.203
Ridge Regression	0.200	0.200	0.200	0.200
ElasticNet Regression	0.202	0.202	0.202	0.203
Decision Tree Regression	0.803	0.730	0.817	0.826
KNeighbors Regressor	0.329	0.295	0.335	0.323
Support Vector Regression	0.571	0.495	0.487	0.501
ConvLSTM-PSO-XGBoost	1	1	1	1

.

Following extensive verification through more than five iterations, the findings shown in Figure 13 were obtained. The figure demonstrates that the ConvLSTM-PSO-XGBoost model significantly outperforms the other nine models.

5. Conclusions

This paper proposes a ConvLSTM-PSO-XGBoost fusion model that outperforms its individual components across all metrics. Evaluation encompassed feature processing, model optimization, model assessment, and predictive performance. Results demonstrate an accuracy rate of 94.4%, confirming the fusion model’s capability to accurately predict maritime 5G signals with superior performance compared to standalone models.

However, the current study has the following limitations.

▪

Limited dataset coverage: data were collected only in the Zhoushan Sea area of China, excluding offshore environments.

▪

No consideration of multi-base station handover: actual vessel navigation involves base station switching, necessitating the introduction of a handover prediction mechanism.

▪

Lack of extreme weather testing: data from extreme scenarios such as typhoons (wind speeds > 33 m/s) were not included.

Future research directions include the following:

▪

Integrating meteorological data (e.g., wave spectra, barometric pressure changes) to enhance prediction robustness.

▪

Introducing attention mechanisms to focus on critical features (e.g., environmental parameters during occlusion).

▪

Developing lightweight models (e.g., model compression techniques) for edge computing scenarios.

This study proposes a variable-weight fusion model combining ConvLSTM and PSO-XGBoost for predicting 5G signal strength at sea. Validation using field data from the East China Sea demonstrates superior overall performance compared to existing mainstream methods. The findings not only provide a novel approach for marine 5G signal prediction, but also offer a reliable data source for marine channel modeling and communication.