FTT: A Frequency-Aware Texture Matching Transformer for Digital Bathymetry Model Super-Resolution

Abstract

Deep learning has shown significant advantages over traditional spatial interpolation methods in single image super-resolution (SISR). Recently, many studies have applied super-resolution (SR) methods to generate high-resolution (HR) digital bathymetry models (DBMs), but substantial differences between DBM and natural images have been ignored, which leads to serious distortions and inaccuracies. Given the critical role of HR DBM in marine resource exploitation, economic development, and scientific innovation, we propose a frequency-aware texture matching transformer (FTT) for DBM SR, incorporating global terrain feature extraction (GTFE), high-frequency feature extraction (HFFE), and a terrain matching block (TMB). GTFE has the capability to perceive spatial heterogeneity and spatial locations, allowing it to accurately capture large-scale terrain features. HFFE can explicitly extract high-frequency priors beneficial for DBM SR and implicitly refine the representation of high-frequency information in the global terrain feature. TMB improves fidelity of generated HR DBM by generating position offsets to restore warped textures in deep features. Experimental results have demonstrated that the proposed FTT has superior performance in terms of elevation, slope, aspect, and fidelity of generated HR DBM. Notably, the root mean square error (RMSE) of elevation in steep terrain has been reduced by 4.89 m, which is a significant improvement in the accuracy and precision of the reconstruction. This research holds significant implications for improving the accuracy of DBM SR methods and the usefulness of HR bathymetry products for future marine research.

1. Introduction

Digital bathymetry models (DBMs) are critical tools for exploring and understanding the Earth [1]. Accurate bathymetric data assist scientists in studying marine geology [2] and biodiversity [3], revealing mechanisms within the Earth’s interior [4,5], and predicting geological hazards such as tsunamis and earthquakes [6].

There are three mainstream bathymetric methods: shipborne [7], airborne [8], and spaceborne [9,10]. Through a comparative analysis of advantages and disadvantages of three methods [11], as shown in

Table 1. Comparison of three prevailing bathymetry methods.

Methods	Sensors	Strength	Limitations
Shipborne	SBES	Highly reliable	Limited range
	MBES	High resolution	High operating costs
Airborne	ALB	High accuracy	Limited water depth
	SAR	Ignore clouds and fog	High uncertainty
Spaceborne	SA	Wide range	Low accuracy
	SDB	Wide range	Limited water depth

, we can conclude that each bathymetric method has limitations in terms of measurement accuracy, coverage range, and cost-effectiveness, which significantly hampers the generation of high resolution (HR) DBM. Currently, only about 20 percent of the seabed has been measured directly, and less than 1 percent has been measured with a horizontal resolution greater than 200 m [12]. Given the pressing demand for HR DBM in marine science, we are dedicated to developing a convenient and efficient method to improve the resolution of DBM.

Super-resolution (SR) is an image processing technique that predicts HR details from a single low resolution (LR) image [13]. Therefore, the SR method is expected to be an effective solution for generating high-quality HR DBMs. Earlier SR methods predominantly relied on traditional interpolation methods such as nearest neighbor, bilinear, and bicubic [14]. These methods are computationally simple but rely heavily on assumptions, which can lead to significant distortions, especially in complex terrains [15]. Recently, machine-learning based methods have shown superior performance over traditional interpolation, but they require a substantial amount of prior work [16]. With the advancement of deep learning, SR has made significant strides. Beginning with SR convolutional neural network (SRCNN) [17], numerous models utilizing CNNs have emerged. These can be categorized based on their structure as Residual Connections [18,19], Recursive Connections [20], Multi-Branch Connections [21,22], Densely Connected Connections [23,24], and Multiple Degradation Handling Connections [25,26]. As SR research advances, there is a growing emphasis on visual perception quality. Ledig et al. [27] pioneered the use of generative adversarial networks (GANs) to design SRGAN, markedly enhancing the visual quality of generated images. This innovation spurred the development of GAN-based models such as EDRSGAN [28] and RealGAN [29], which primarily focused on visual perception quality. Although CNN- or GAN-based SR methods have shown impressive results in natural images, they may not be suitable for DBM SR due to huge differences between DBM and natural images. Existing SR methods have three main drawbacks when applied to DBM:

(1)

Limited terrain feature-extraction capabilities: The local receptive field and translation invariance are powerful tools for CNNs to extract natural image features, but they may not be as effective for terrain features. A local receptive field has a limited capacity to capture large-scale topographic features [16,30], and translation invariance conflicts with the multi-scale spatial heterogeneity of terrain.

(2)

Loss of high-frequency priors: High-frequency priors refer to sharp changes in elevation, indicating steep terrain such as ridges or deep canyons. These topographic features are crucial for geological analysis and ocean resource management. However, convolutional operations and attention mechanisms may dampen high-frequency priors [31,32], which adversely affects the quality of DBM SR reconstruction.

(3)

Texture distortion and position shift: In our observations, increasing the depth of neural-network layers enhances the model’s ability to extract features but may lead to texture distortion and positional offsets. Although residual structures help to mitigate this phenomenon, further improving the fidelity of the generated HR DBM is a pressing problem given the high accuracy requirement of DBM.

To address the above challenges, we propose a frequency-aware texture matching transformer (FTT) for DBM SR, contributing to the following improvements:

(1)

We developed global terrain feature extraction (GTFE), which can effectively capture large-scale terrain features while maintaining sensitivity to spatial heterogeneity and spatial structure.

(2)

The high-frequency feature extraction (HFFE) is employed to alleviate the low-pass filtering characteristics of the swin transformer, thereby better preserving the high-frequency priors.

(3)

The terrain matching block (TMB) can integrate high-fidelity shallow features and deep features with rich semantics to improve fidelity.

2. Materials and Methods

In this section, we describe the FTT architecture and elaborate the principles and design philosophy of the core components.

2.1. Model Overview

The proposed FTT consists of basic terrain feature extraction, advanced terrain feature extraction, and high-quality terrain reconstruction (see Figure 1). Basic terrain feature extraction and advanced terrain feature extraction are employed to extract high-dimensional terrain features from the LR input. High-quality terrain reconstruction is used to upsample and get the reconstructed HR output.

Basic terrain feature extraction consists of a single CNN layer with a kernel size of $3\times 3$, designed for extracting basic terrain characteristics. Advanced terrain feature extraction includes several terrain feature extraction groups (TFEGs), each comprising one terrain feature refining block (TFRB) (see Figure 2a) and one CNN layer. The core components of TFRB are the GTFE (see Figure 2b) and HFFE (see Figure 2c). The output of GTFE and HFFE would be fused by cross-attention with ConvFNN [33]. High-quality terrain reconstruction consists of the pixel shuffle layer [34], TMB (see Figure 2d), and one CNN layer for adjusting the number of channels. A pixel shuffle is a common tool used for feature upsampling. The following is a detailed introduction to GTFE, HFFE, and TMB.

2.2. GTFE

The limited ability of CNNs to capture large-scale terrain details prompted us to introduce the swin-transformer layer (STL) of [35]. STL consists of swin-transformer blocks. To endow STL with spatial awareness capabilities, we developed spatial STL (SSTL), which replaced swin-transformer blocks with the proposed GTFE. Specifically, compared with the swin-transformer block, GTFE removed the layer norm (LN) and incorporated the coordinate attention (CA) [36] branch that uses the CA to extract the spatial structure information. The reasons for these modifications are as follows:

• Impact of LN: The LN reduces the contrast between different terrain features. For instance, normalized peaks may appear flattened, making it difficult for the model to capture spatial heterogeneity and resulting in overly smooth outputs. Removing LN helps the model retain more terrain detail;
• Limitations of Patch Embedding: The patch embedding in STL limits the model’s ability to explicitly understand the 2D spatial structure of the DBM. To address this issue, we added the CA branch too. As shown in Figure 3, the CA branch generates attention maps in both vertical and horizontal directions, enhancing the model’s ability to perceive 2D coordinates and better handle terrain information.

Overall, by removing LN and incorporating the CA, GTFE can more accurately capture and represent the details of large-scale terrain.

In GTFE, input $X\in {\mathbb{R}}^{C\times H\times W}$ (where C denotes the number of channels, and H and W represent the spatial height and width of the feature) is first transformed into $X\in {\mathbb{R}}^{(H\times W)\times C}$ by patch embedding. Then, $X\in {\mathbb{R}}^{(H\times W)\times C}$ is processed through two branches. One branch is the window multi-head self attention (WMHSA), and the other branch is the CA branch. WMHSA introduced the window partitioning and shifted window strategies to reduce the computational complexity. In here, for brevity, we introduce the process of MHSA. The details of window partitioning and shifted window strategies can be found in [35]. For the MHSA, three learnable matrices are used to map X to the Q ($query$), K ($key$), and V ($value$) matrices. Next, the dot product of Q and K is computed, and the attention scores are obtained using the softmax function. Finally, the attention scores are multiplied with V to produce the final result:

$$Attention\left(Q,K,V\right)= \mathrm{SoftMax} \left({\displaystyle \frac{Q{K}^T}{\sqrt{d}}}+B\right)V,$$

(1)

where B is the relative position information, and d is the dimension of the hidden layers.

As for the CA branch, the input X is first processed through average pooling to obtain the 1D vectors $Z_v$ and $Z_h$ for the vertical and horizontal directions. Then, $Z_v$ and $Z_h$ are concatenated to form f, which is compressed in channel dimensions using convolution to facilitate cross-channel information exchange. Finally, f is separated and used for normalization and weighting with X. The whole process is illustrated in Figure 3. Lastly, the terrain feature representation captured by two branches is enhanced through the multilayer perceptron (MLP). The overall formula for GTFE is given as follows:

$$Y=MLP\left(\mathrm{WMHSA}\right(X)+X+\mathrm{CA}(X\left)\right)+X,$$

(2)

where Y denotes the output of GTFE.

2.3. HFFE

The low-pass filtering property of transformers results in the loss of crucial high-frequency information. To address this issue, we designed HFFE. Specifically, there are two branches to extract spatial and high-frequency priors, respectively. In the high-frequency extraction branch, fast Fourier transform (FFT) is employed to convert the input X into the frequency domain. CNNs then identify and extract key high-frequency features, which are subsequently mapped back to the spatial domain using the inverse Fourier transform. Since frequency conversion can lead to loss of spatial information, the spatial information-extraction branch, composed of multiple CNNs, is used to retain important spatial location details. The formulas for the process described above are written as

$$\begin{array}{c}{X}_{fre}={\mathcal{F}}^{-1}\left(F_f\left(\mathcal{F}\left(X\right)\right)\right)+X,\hfill \\ X_{spa}=F_s\left(X\right)+X,\hfill \end{array}$$

(3)

where $F_f$ denotes alternating CNN layers. $F_s$ represents three CNN layers. $X_{fre}$ and $X_{spa}$ are the extracted high-frequency information and spatial information, respectively. Then the $X_{fre}$ and $X_{spa}$ will be concatenated along the channel dimension, and then fed into a CNN layer to obtain the output $X_h$.

After that, we use cross attention to fuse the $X_h$ with the output of SSTL $X_l$. Specifically, we treat $X_f$ as Q, and $X_l$ serves as K and V. We then use the cross-attention mechanism to implicitly integrate high-frequency information into spatial information, thereby enhancing the model’s ability to perceive high-frequency details. Specifically, we first process $X_l$ and $X_h$ separately through CNNs to obtain Q, K, and V, and then we use the following formula to compute the cross-attention.

$$Attention(Q,K,V)=Softmax\left(Q{K}^T\right)V.$$

(4)

Finally, we feed the fusion feature $X_{fusion}$ into the ConvFNN in [33]. The whole process can be formulated as

$$\begin{array}{c}{X}_{fusion}=CrossAttenion(LN\left(X_h\right),X_l),\hfill \\ Y=ConvFNN\left(LN\left(X_{fusion}\right)\right)+X_{fusion},\hfill \end{array}$$

(5)

where Y is the final output of HFFE.

2.4. TMB

Shallow terrain features, being closer to the input, have higher fidelity. Deep features, after multiple rounds of feature extraction, are rich in semantic information but deviate from the original inputs. As illustrated in Figure 2d, TMB aligns the deep terrain features with the shallow terrain features, thereby enhancing overall fidelity. Specifically, we first concatenate the shallow feature $X_{shallow}$ and the deep terrain feature $X_{deep}$, and then we employ the CNN to perform information fusion and generate attention weights by using the Sigmoid function. Next, the offsets and $X_{deep}$ are fed into the CNN layer to obtain the fused feature. The formula for TMB is expressed as

$$\begin{array}{cc}\hfill & \mathrm{offset}=\mathrm{Conv}\left(Concat\left(X_{shallow},X_{deep}\right)\right),\hfill \\ \hfill & X_{deep}=\mathrm{Conv}(\mathrm{Conv}\left(X_{deep}\right)\times \mathrm{Sigmoid}\left(\mathrm{offset}\right))\times W+X_{deep},\hfill \end{array}$$

(6)

where W is a scaling factor used to reduce the impact of noise in the shallow features.

3. Experiments and Results

In this section, we provide a comprehensive description of the experimental setup, including the datasets and elevation metrics, training details, and results.

3.1. Datasets and Elevation Metrics

3.1.1. Study Area

We selected the sea area located between 0° N to 45° N and 135° E to 180° E as the experimental region (see Figure 4). This region has diverse seabed topography, including ridges, trenches, seamounts, and basins. We have selected GEBCO as the data source, which is a publicly available 15 arc-second DBM dataset.

3.1.2. Data Preprocessing

To facilitate training, we preprocessed the dataset, including non-overlapping cropping, land–sea masking, normalization, and partitioning.

We cropped the original HR DBM data into 128 × 128 patches, leaving 7056 patches. Patches containing land were then removed. Considering that elimination of land is impractical near island areas, we performed land–sea masking:

$$Z=n-\alpha N,$$

(7)

where $\alpha$ is a constant that represents the proportion of land in each patch (set to 0.1 in our study); n is the number of points with DBM elevation values greater than 0; and N is the total number of data points in each patch. For a given patch, if $Z<0$, it indicates that the land area proportion of that region is less than 10%.

Finally, we obtained a DBM dataset containing 6908 patches. To obtain paired HR DBM and LR DBM, each patch was resized to 32 × 32 as LR DBM using nearest neighbor interpolation. Then the training, validation, and test sets were randomly assigned in a ratio of 8:1:1. Prior to training, due to the large difference between patches, it was necessary to normalize each patch value to $[-1,1]$,

$$DB{M}_i=2\times {\displaystyle \frac{\left(DB{M}_i-H_{Min}\right)}{(H_{Max}-H_{Min})}}-1,$$

(8)

where $H_{Max}$ and $H_{Min}$ are the respective maximum and minimum values in a single patch.

3.1.3. Elevation Metrics

To validate the performance of the FTT, we selected several representative SR methods as benchmarks, including SRCNN, SRResNet, SRGAN, and SwinIR. Previous studies [16,37,38,39] have used SRCNN, SRGAN, and SRResNet to evaluate the performance of SR models. We also added SwinIR [35] as a comparative method. Based on the swin-transformer, SwinIR has become SOTA.

We used the root mean square error (RMSE) of three key parameters (elevation, slope, and aspect) to quantitatively compare the performance of each method. Elevation RMSE directly reflects reconstruction accuracy, where a lower value indicates better reconstruction. Slope and aspect are important terrain factors, whose effectiveness of reconstruction can reflect a model’s ability to restore local terrain details. We calculate

$$\mathrm{RMSE}=\sqrt{{\displaystyle \frac{1}{n}}{\displaystyle \sum _{i=1}^n}{(x_i-\widehat{x_i})}^2},$$

(9)

where $x_i$ and $\widehat{x_i}$ denote the real HR DBM (including elevation, slope, and aspect) and the generated SR DBM at the i-th point, and n is the total number of points in each DBM.

SSIM was used to evaluate the fidelity of the HR DBM generated by different methods. The closer the SSIM value is to one, the higher the fidelity. SSIM is calculated as

$$SSIM(x,\widehat{x})={\displaystyle \frac{\left(2{\mu }_x{\mu }_{\widehat{x}}+c_1\right)\left(2{\sigma }_{x\widehat{x}}+c_2\right)}{\left({\mu }_x^2+{\mu }_{\widehat{x}}^2+c_1\right)\left({\sigma }_x^2+{\sigma }_{\widehat{x}}^2+c_2\right)}},$$

(10)

where ${\mu }_x$ and ${\mu }_{\widehat{x}}$ are the respective means of x and $\widehat{x}$, with corresponding variance ${\sigma }_x$ and ${\sigma }_{\widehat{x}}$; ${\sigma }_{x\widehat{x}}$ is the covariance between x and $\widehat{x}$; and $c_1={\left(k_{1}L\right)}^2$, $c_2={\left(k_{2}L\right)}^2$, and $c_3={\left(k_{3}L\right)}^2$ are constants to avoid a denominator of 0. $k_1$ and $k_2$ respectively default to 0.01 and 0.03, and L is the range of image pixel values.

3.2. Training Details

We used several Nvidia 3060 24 GB graphics cards for training. The hyperparameters set for FTT included 60 channels and 6 num-heads. Considering that the shallow features have more noise, the parameter $w=0.01$ in TMB was chosen. The convolution parameters in FTT were uniformly set with kernel size = 3, padding = 1, stride = 1. Considering the convergence rates of different models, and to ensure rigorous and fair experiments, we uniformly adopted the patience early stopping technique (patience value of 10) to train each model, so as to ensure convergence. The training hyperparameters set the learning rate to 0.00002 and the batch size to 16. The loss function was L1Loss, and the optimizer was Adam.

3.3. Results

The results of different SR methods on the test set are shown in

Table 2. Performance comparison results of DBM SR.

Model	RMSE of Elevation (m) ↓	RMSE of Slope (°) ↓	RMSE of Aspect (°) ↓	SSIM ↑
SRCNN	16.30	5.29	68.29	0.968
SRResNet	16.07	5.39	71.06	0.968
SRGAN	16.23	5.19	68.10	0.968
SwinIR	15.43	5.12	67.43	0.969
FTT (ours)	13.48	4.81	65.25	0.973

↓ indicates that a lower value of the indicator is better; ↑ indicates that a higher value of the indicator is better.

. To more intuitively show the performance of different models, we randomly selected a patch from the test set to show the reconstruction effect of elevation (Figure 5), slope (Figure 6), and aspect (Figure 7).

As shown in Table 2, FTT demonstrates the best performance across all four evaluation metrics, highlighting its effectiveness at enhancing the precision of generated HR DBM. Among the three methods using CNN, SRCNN performs relatively poorly in elevation RMSE due to the limitation of fewer convolutional layers, but its slope RMSE and aspect RMSE slightly exceed those of SRResNet and SRGAN, possibly because SRCNN takes a bicubic interpolated value as input. SRGAN outperforms SRResNet, suggesting that perceptual loss functions can improve reconstruction accuracy to some extent, but the SSIM metrics are not improved compared to the other two methods. SwinIR outperforms the methods using CNN, which demonstrates the superiority of transformers over CNNs, and shows the advantage of a larger receptive field in extracting terrain features.

4. Discussion

To facilitate a clear comparison of the performance across different models, we utilize boxplots to illustrate the distribution of errors on the test dataset. Specifically, to further exhibit the performance of the various methods under different topographic conditions, we divided the test set into 10 categories based on increasing elevation variance. Variance reflects the degree of terrain undulation. DBM with larger variance represents steeper and more rugged terrains, which typically pose greater challenges for DBM SR. The SR performance of different methods is displayed through box plots (see Figure 8). For ease of comparison, we present an RMSE line chart in the first subplot at the top-left corner of Figure 8. It is evident that all SR methods demonstrate optimal performance on flat terrain and significant limitations on steep terrain, which illustrates that steep and complex terrain presents a significant challenge. The proposed model exhibits superior performance across all terrain types, with a notable reduction in RMSE of 4.89 m compared to SwinIR in the most challenging terrain, indicating that FTT has stronger reconstruction capability and robustness on steep terrains.

To analyze the spatial distribution characteristics of errors in detail, we randomly selected a sample from the test data, and display its elevation error (Figure 9), slope error (Figure 10), and aspect error (Figure 11). By analyzing the correspondence between errors and actual values in spatial distribution, we conclude that areas with significant terrain undulations often exhibit larger errors and underestimate the true values. This reveals the limitations of CNNs and attention mechanisms regarding such terrains. In contrast, FTT demonstrates a clear advantage in SR reconstruction of steep terrains.

Ablation Experiments

From the above analysis, the proposed FTT method clearly exhibits significant performance advantages in DBM SR reconstruction tasks. To gain a deeper understanding of the specific impact of each key component of the FTT method on overall model performance, and to further validate the rationality of our proposed problem settings, we conducted a series of ablation experiments. We sequentially removed each key component from the FTT to observe the specific impact of these changes on the model’s performance. To ensure fairness and comparability across experiments, we employ identical hyperparameters during training.

Table 3. Result of the ablation experiment.

	CA	HFFE	TMB	RMSE of Elevation (m) ↓	RMSE of Slope (°) ↓	RMSE of Aspect (°) ↓	SSIM ↑
I	✓			13.90	4.91	65.74	0.972
II		✓		13.79	4.88	65.53	0.973
III			✓	13.93	4.92	66.11	0.973
IV	✓	✓		13.76	4.87	65.65	0.973
V	✓		✓	13.84	4.90	65.55	0.973
VI		✓	✓	13.76	4.87	65.51	0.973
VII	✓	✓	✓	13.48	4.81	64.97	0.973

↓ indicates that a lower value of the indicator is better; ↑ indicates that a higher value of the indicator is better; ✓ denotes that the component is in use.

shows that when only TMB is applied, its SSIM value exceeds that of CA alone. This indicates that TMB effectively enhances image fidelity by utilizing the positional offset generated in shallow features. The inclusion of HFFE results in substantial improvements across all four evaluation metrics, underscoring its effectiveness in extracting high-frequency prior information for DBM SR. The incorporation of CA enhances the model’s ability to perceive two-dimensional spatial features, further optimizing its performance. Collectively, each component functions as intended, contributing to the overall enhancement of performance.

5. Conclusions

Limitations in bathymetry equipment have prompted researchers to use SR technology to generate HR DBMs, but the significant differences between DBMs and natural images are often overlooked, which leads to serious distortions and inaccuracies. We have presented a novel SR method, FTT, to overcome these challenges. We conducted a detailed analysis of the disparities between natural images and terrain, identifying three primary issues prevalent in current SR methods. Subsequently, we proposed the FTT, which includes GTFE, HFFE, and TMB to address these issues. GTFE can aggregate global features and establish long-range dependencies. HFFE enhances the model’s ability to extract high-frequency information. TMB solves the problems of texture distortion and positional offset during upsampling and feature extraction. Experiments demonstrated that the proposed FTT shows significant improvement in elevation, slope, and aspect accuracy compared to commonly used SR models, and exhibits excellent performance, especially in steep terrain. Ablation experiments further validated the effectiveness of each component within FTT. FTT represents a substantial advancement in SR techniques tailored for DBM, offering robust solutions to improve the accuracy and fidelity of HR terrain data generation.

Notably, the error distribution on the test set reveals significant performance differences of all SR methods across various terrains, primarily due to terrain heterogeneity. We acknowledge this phenomenon, but have not extensively explored it, underscoring it as a pivotal area for future investigation. In the future, we intend to comprehensively address terrain differences, and to conduct more nuanced studies aimed at producing highly accurate global HR DBM products.