Stripe Extraction of Oceanic Internal Waves Using PCGAN with Small-Data Training

Abstract

Playing a crucial role in ocean activities, internal solitary waves (ISWs) are of significant importance. Currently, the use of deep learning for detecting ISWs in synthetic aperture radar (SAR) imagery is gaining growing attention. However, these approaches often demand a considerable number of labeled images, which can be challenging to acquire in practice. In this study, we propose an innovative method employing a pyramidal conditional generative adversarial network (PCGAN). At each scale, it employs the framework of a conditional generative adversarial network (CGAN), comprising a generator and a discriminator. The generator works to produce internal wave patterns as authentically as possible, while the discriminator is designed to differentiate between images generated by the generator and reference images. The architecture based on pyramids adeptly captures the encompassing as well as localized characteristics of internal waves. The incorporation of upsampling further bolsters the model’s ability to recognize fine-scale internal wave stripes. These attributes endow the PCGAN with the capacity to learn from a limited amount of internal wave observation data. Experimental results affirm that the PCGAN, trained with just four internal wave images, can accurately detect internal wave stripes in the test set. Through comparative experiments with other segmentation models, we demonstrate the effectiveness and robustness of PCGAN.

1. Introduction

Internal solitary waves in the ocean are captivating phenomena within the field of oceanography, occurring beneath the ocean’s surface and widely present in stably stratified oceans [1,2]. With amplitudes reaching up to 240 m [3], these waves carry substantial energy and have significant impacts on offshore drilling operations [4,5]. Simultaneously, they play a crucial role in the intricate interplay of energy within the marine ecosystem [6]. Hence, it is imperative to precisely ascertain the positions of oceanic internal waves.

Internal waves propagate beneath the ocean surface, and the marine environment is complex and dynamic, making it challenging to directly obtain parameters of internal waves. Utilizing multiple underwater gliders, as indicated in [7], allows for the reconstruction of three-dimensional regional oceanic temperature and salinity fields in the northern South China Sea. Observations of oceanic internal waves can be conducted through temperature and salinity profiles, providing corresponding parameters [8]. However, the data on internal waves obtained through this method remains very limited. Fortunately, ISWs induce convergence and divergence effects, leading to alterations in surface roughness and sun-glint reflection. This phenomenon results in the manifestation of alternating bright and dark stripes on satellite images. Since the early 1980s, this pattern has been detected in synthetic aperture radar (SAR) [9,10]. SAR remains unaffected by cloud cover and can capture high-resolution ocean surface imagery ranging from a few meters to tens of meters, irrespective of weather conditions, day or night [11]. Consequently, SAR has emerged as a robust tool for monitoring oceanic internal waves [12]. Automated segmentation of oceanic internal wave stripes within SAR images is necessary to ascertain the positions of the stripes and subsequently investigate their propagation or invert the parameters of oceanic internal waves.

Over the past few decades, there has been significant research into algorithms and techniques aimed at the automated detection of internal wave signatures from SAR imagery, employing fundamental image-processing methods. Rodenas and Garello [13] conducted oceanic internal wave detection and wavelength estimations using wavelet analysis. They introduced the creation of a suitable wavelet basis for the identification and localization of nonlinear wave signatures within SAR ocean image profiles. Subsequently, the application of continuous wavelet transform is employed to estimate energies and wavelengths within soliton peaks from the identified internal wave trains. Ref. [14] employs a 2D wavelet transform based on multiscale gradient detection for automated detection and orientation of oceanic internal waves in SAR images. Furthermore, it introduces a coastline detection approach to achieve sea-land separation, thereby enhancing the effectiveness of internal wave detection within the SAR image context. Simonin et al. [15] introduce a framework that combines wavelet analysis, linking, edge discrimination, and parallelism analysis for the automated identification of potential internal wave packets within SAR images. The framework has been demonstrated and tested using six satellite images of the Eastern Atlantic, affirming its capability to determine the signature type and wavelength of the internal wave. The study conducted by Zhang et al. [16] investigates the utilization of compact polarimetric (CP) SAR in detecting and identifying oceanic internal solitary waves (ISWs). CP SAR images are generated and 26 CP features are extracted from full-polarimetric Advanced Land Observing Satellite (ALOS) Phase Array type L-band SAR (PALSAR) images. The effectiveness of different polarization features in distinguishing ISWs from the sea surface is evaluated using Jeffries and Euclidean distances. Expanding upon this, an enhancement to the detection capabilities of ISWs is introduced through the implementation of a K-means clustering algorithm utilizing compact polarimetric (CP) features. Qi’s research [17] employed the Gabor transform to extract wave characteristics. The study further utilized the K-means clustering algorithm for stripe segmentation within SAR images. To distinguish the light and dark wave stripes from the background, morphological processing techniques were applied.

With the evolution of neural networks, machine-learning-based models for internal wave detection have been extensively explored in recent years. Machine learning has the capability to automatically extract deep features, providing a more convenient and effective approach. For instance, Wang et al. [18] developed a method for detecting oceanic internal waves, employing a deep learning framework known as PCANet. This method combines binary hashing, principal component analysis (PCA), and block-wise histograms. Following this, a linear support vector machine (SVM) classification model is utilized to accurately identify the locations of oceanic internal waves using a rectangular frame. In the study by Bao et al. [19], the Faster R-CNN framework is employed to achieve the detection of oceanic internal waves within SAR images. This model adeptly navigates the challenge of misidentifying features like ship wakes, prone to aliasing, while simultaneously accurately delineating regions responsible for generating internal waves.

However, the aforementioned deep learning-based methodologies are limited to detecting the positions of oceanic internal waves through rectangular bounding boxes; these approaches are unable to characterize the precise locations of the distinct stripes.

Ref. [20] presents a comprehensive algorithm designed to detect and identify oceanic internal waves. To address the pervasive speckle noise in SAR images, the initial step involves the application of the Gamma Map filtering technique. Subsequently, the classification of SAR images and identification of those containing oceanic internal waves are accomplished through feature fusion and SVM. Ultimately, the Canny edge detection method is employed for the detection and recognition of oceanic internal wave stripes within the SAR images. Within the framework proposed by Vasavi [21], U-Net is applied to carry out feature extraction and segmentation tasks, delving into wave parameters such as frequency, amplitude, latitude, and longitude. Following this, the Korteweg-de Vries (KdV) solver is applied, taking the internal wave parameters as inputs and providing density and velocity plots corresponding to the internal waves as outputs. Li et al. [22] applied a modified U-Net framework to extract ISW-signature information from Himawari-8 images in challenging imaging scenarios. They opted for $\alpha$-balanced cross-entropy as the loss function, deviating from the traditional cross-entropy, and achieved remarkable results. Zheng et al. [23] proposed an algorithm utilizing the SegNet architecture for segmenting oceanic internal waves. This approach proficiently detects the presence of oceanic internal waves within SAR images and determines the specific positions of both light and dark stripes. In [24], an algorithm for segmenting oceanic internal waves’ stripes was introduced, relying on Mask R-CNN. Additionally, they employed a separation and matching approach within the sector region (SMMSR). This method not only accomplishes the localization of internal waves but also allows for the extraction of crucial parameters, encompassing the width and directional angle, associated with every discernible light and dark stripe. In [25], Middle Transformer U²-net (MTU²-net) was introduced as an innovative model, combining a transformer and a unique loss function to enhance its ability to detect ISWs.

The aforementioned intelligent detection methods operate through a training process. The majority of machine-learning-based approaches for internal wave detection rely on substantial training datasets to ensure precise detection outcomes. However, obtaining a large number of labeled internal wave images remains a challenge. In addition to traditional data augmentation methods, there have been studies focused on training neural networks with a small number of samples. For instance, in [26], different-sized convolutional kernels were employed to extract features from a small sample set comprehensively. Ref. [27] employs a random forest-like strategy, achieving superior classification accuracy without overfitting, even with much smaller training datasets than commonly studied in deep learning literature for image classification tasks. In [28], a multi-scale model was applied to detect oil spills on the sea surface, achieving accurate detection results with very few training samples. In [29], a training strategy involving mutual guidance was used to create a powerful hyperspectral image classification framework trained on a small dataset.

Inspired by these methods, we apply the concept of training on a small dataset to the task of internal wave detection. Given the finer and less distinct features of oceanic internal wave stripes, we introduce a pyramidal conditional generative adversarial network (PCGAN) to achieve stripe segmentation of oceanic internal waves with limited training data. PCGAN is composed of a series of adversarial networks at different scales. At each scale, it undergoes training from coarse to fine using observed internal wave images and detection maps. At each scale, PCGAN includes an adversarial network consisting of a generator and a discriminator. The generator’s role is to capture the characteristics of the observed image and produce an internal wave detection map that closely mimics reality. Meanwhile, the discriminator is tasked with differentiating between real images and those generated by the generator. Each generator’s output becomes the input for the subsequent, more detailed-scale generator, as well as the current-scale discriminator. The training process is independently conducted at each scale, following the structure of a Conditional Generative Adversarial Network. This article’s primary contributions can be outlined as follows:

(1)

We introduce a pioneering Pyramidal Conditional Generative Adversarial Networks (PCGAN) structure designed for internal wave stripe extraction. This model integrates lightweight networks for both generators and discriminators.

(2)

We independently train adversarial network structures at every scale, allowing a cascade of internal wave features to flow from coarse to fine in the data processing.

(3)

We enhance the model’s capability to extract finer details of internal wave stripes from the images by incorporating upsampling.

(4)

We manually labeled a diverse set of internal wave images to train and validate the model’s performance across various characteristics.

The subsequent sections are organized as follows: Section 2 provides a detailed overview of the model architecture and training specifics of PCGAN. In Section 3, we present the experimental configurations and assessments conducted. Following that, Section 4 delves into a comprehensive discussion of the results of the experiments. Lastly, Section 5 summarizes the conclusions derived from the presented approach and its effectiveness.

2. Materials and Methods

2.1. Basic GAN and Its Variants

The foundational GANs model, initially presented by Goodfellow et al. in 2014 [30], operates as a framework for training deep generative models through a two-player minimax game. The primary objective of GANs is to train a generator distribution, denoted as $p_G$, to closely align with the distribution of real data, represented by $p_{data}$. In the context of GANs, the model entails the learning of a generator network G responsible for generating samples by transforming a latent noise vector z into a corresponding sample $G\left(z\right)$. This generator is subjected to training through an adversarial interplay with a discriminator network D, which is designed to discriminate between samples originating from the authentic data distribution $p_{data}$ and those generated by the distribution $p_G$. The core objective of the original GANs is articulated through the following objective function:

$$\underset{G}{min}\underset{D}{max}V(D,G)={\mathbb{E}}_{x\sim p_{data}\left(x\right)}\left[logD\left(x\right)\right]+{\mathbb{E}}_{z\sim p_z\left(z\right)}\left[log(1-D(G\left(z\right)\left)\right)\right].$$

(1)

Conditional Generative Adversarial Networks (CGAN) [31] are a modification of the basic GAN that integrates conditional information to effectively guide the generation process. In CGAN, both the discriminator and generator are augmented with conditional vectors, denoted as y, which can encompass auxiliary details like class labels or image descriptions. This inclusion empowers the generator to produce samples with enhanced accuracy by considering specific conditions. The objective function of CGAN can be represented as follows:

$$\underset{G}{min}\underset{D}{max}V(D,G)={\mathbb{E}}_{x\sim p_{data}\left(x\right)}\left[logD(x\mid y)\right]+{\mathbb{E}}_{z\sim p_z\left(z\right)}\left[log(1-D(G(z\mid y)\left)\right)\right].$$

(2)

The error function of conventional GANs might face challenges during training, stemming from their potential discontinuity concerning the generator’s parameters. As an alternative, the Wasserstein Generative Adversarial Network (WGAN) [32] adopts the Earth-Mover distance, also known as Wasserstein-1 distance, denoted as $W(q,p)$. This distance metric is loosely defined as the minimal cost needed to transport mass for converting the distribution q into the distribution p, where the cost is determined by the product of mass and transport distance. Subject to reasonable assumptions, $W(q,p)$ exhibits continuity everywhere and differentiability almost everywhere.

The WGAN loss function is formulated through the Kantorovich-Rubinstein duality to achieve:

$$\underset{G}{min}\underset{D}{max}V(D,G)={\mathbb{E}}_{x\sim p_{data}\left(x\right)}\left[D\left(x\right)\right]-{\mathbb{E}}_{z\sim p_z\left(z\right)}\left[D\left(G\right(z\left)\right)\right].$$

(3)

Since its introduction in 2014, GAN and its variants have achieved notable success in generative image modeling and demonstrated exceptional performance in semantic segmentation [33,34,35,36]. Adversarial learning has also been shown to be suitable for small-data training [37,38,39]. To obtain an effective method for high-precision internal wave detection in practical scenarios, the latter part of the current section presents an approach to detect internal wave stripes using a Pyramidal Conditional Generative Adversarial Network (PCGAN) trained with a restricted amount of data.

2.2. The PCGAN for the Extraction ISW

Firstly, the initial internal wave remote sensing image is denoted as $I_0$, the corresponding labeled image as $S_0$, and the internal wave identification map generated by our PCGAN is represented as ${\widehat{S}}_0$. Both $S_0$ and ${\widehat{S}}_0$ are binary images, where 0 indicates the presence of internal waves, and 1 indicates the absence of internal waves at the oceanic surface. Pyramidal representations spanning multiple scales are constructed for $I_0$, $S_0$, and ${\widehat{S}}_0$, commencing from the primitive scale.

We first represent the image $I_0$, $S_0$ in the form of an image pyramid by up and downsampling. Here, $I_n$ and $S_n$ denote the representations obtained after n successive downsamplings of the original images. In each downsampling step, the image size is reduced to $1/r$ of the previous level, where the coefficient r is typically set to 2. Correspondingly, $I_m^{+}$ and $S_m^{+}$ are obtained by upsampling the original image, and their image sizes are $r^m$ times those of $I_0$ and $S_0$. The collection of representations across $N+M+1$ scales forms the pyramidal representation set used in the PCGAN for internal wave detection, adopting a coarse-to-fine strategy.

Our proposed approach introduces a pyramid-based conditional generative adversarial network, a hierarchical structure composed of a series of discriminators and generators operating at various scales. Here, $D_n$ and $G_n$ denote the discriminator and generator at the n-th downsampling layer, respectively. The corresponding generated image by $G_n$ is denoted as ${\widehat{S}}_n$. Symmetrically, at the m-th upsampling layer, they are denoted as $D_m^{+}$, $G_m^{+}$, and ${\widehat{S}}_m^{+}$. Illustrating the PCGAN architecture (Figure 1), the core structure is represented in the central pyramidal section. Arrows on both sides indicate the flow of data from coarse to fine, while the upper corners depict the general structure of the generator and discriminator at each scale. The internal wave remote sensing image $I_0$ and its corresponding labeled image $S_0$ serve as the input, while the internal wave detection map ${\widehat{S}}_M^{+}$ represents the result generated by the entire PCGAN framework.

At every scale, taking the internal wave remote sensing image $I_n$ and the previously generated internal wave detection map ${\widehat{S}}_{n+1}$ as inputs, the generator $G_n$ produces the current-scale generated internal wave detection map $\widehat{S}n$. Simultaneously, discriminator $D_n$ is presented with inputs in the form of either the pair $(I_n,S_n)$ or $(I_n,{\widehat{S}}_n)$, leading to the generation of respective discrimination scores $X_n$. The generator aims to produce results that are as realistic as possible to deceive the discriminator, while the discriminator’s goal is to distinguish between the generated detection results and the reference internal wave detection map.

2.3. Discriminator Architecture of PCGAN

Figure 2 illustrates the structural design of the discriminator $D_n$ of PCGAN. The structure consists of five convolutional blocks, with the final block containing only a convolutional layer. In the preceding four blocks, batch normalization (BN) is applied after the convolutional layer, followed by activation using the LeakyReLU function.

The discriminator $D_n$ takes either the produced pair $(I_n,{\widehat{S}}_n)$ or the real pair $(I_n,S_n)$ as the input and produces a discrimination score $X_n$. The score $X_n$ is determined by computing the mean from the feature map produced by the last convolutional layer of $D_n$. Additionally, it serves as an indication of the likelihood that the image pair originates from a real image, with values closer to 1 indicating a higher probability of being from a real image. At each scale, the discriminator strives to assign a high score to the genuine image $S_n$, classifying it as real, and a low score to the generated image ${\widehat{S}}_n$, categorizing it as fake, thereby distinguishing between the two.

2.4. Generator Architecture of PCGAN

The structure of the generator $G_n$ of PCGAN is illustrated in Figure 3. It takes the observed remote sensing image $I_n$ at the current scale along with the extraction result ${\widehat{S}}_{n+1}$ generated by the generator $G_{n+1}$ at the previous scale as inputs. Then, the detection map ${\widehat{S}}_{n+1}$ is upsampled by a factor of r, resulting in ${\widehat{S}}_{n+1}^{\uparrow }$, having identical dimensions as $I_n$. Subsequently, the pair $(I_n,{\widehat{S}}_{n+1}^{\uparrow })$ is concatenated along the channel dimension and then processed by the convolutional network $C_n$. Similar to the discriminator, the convolutional network $C_n$ is composed of five convolutional modules. Each module consists of a convolutional layer, a BN layer, and an activation layer. The convolutional network’s output is pixel-wise blended with ${\widehat{S}}_{n+1}^{\uparrow }$, meaning that the grayscale values of features are combined. This process yields the output of the generator $G_n$ for stripe extraction of the internal wave, denoted as:

$${\widehat{S}}_n={\widehat{S}}_{n+1}^{\uparrow }+C_n(I_n,{\widehat{S}}_{n+1}^{\uparrow })=G_n(I_n,{\widehat{S}}_{n+1}).$$

(4)

Here, the upward arrow ↑ indicates the image after upsampling. At the coarsest scale, to ensure consistency in channels, ${\widehat{S}}_{N+1}$ is set as a blank image of the same size as $I_N$.

By utilizing the detection image from the previous layer ${\widehat{S}}_{n+1}$, the generator aims to enhance its expressive capacity and generate internal wave detection images that are as realistic as possible, thereby obtaining better scores in the discriminator.

2.5. The Training Process of PCGAN

The model PCGAN is trained hierarchically from the coarsest scale to the finest scale, with a CGAN trained at each scale. Once the training at a previous scale is completed, the generated internal wave detection images ${\widehat{S}}_{n+1}$ are utilized for training at the next scale. The loss function is based on WGAN-GP (Wasserstein GAN with gradient penalty) and incorporates an L1-norm regularization term, formulated as follows:

$$\mathcal{L}(G_n,D_n)={\mathcal{L}}_{WGAN}(G_n,D_n)+{\mathcal{L}}_{L_1}\left(G_n\right)+{\mathcal{L}}_{GP}\left(D_n\right),$$

(5)

where

$$\left\{\begin{array}{c}{\mathcal{L}}_{WGAN}(G_n,D_n)=\mathbb{E}\left[D_n(I_n,{\widehat{S}}_n)\right]-\mathbb{E}\left[D_n(I_n,S_n)\right],\hfill \\ {\mathcal{L}}_{L_1}\left(G_n\right)={\lambda }_1\mathbb{E}\left[{∥S_n-{\widehat{S}}_n∥}_1\right],\hfill \\ {\mathcal{L}}_{GP}\left(D_n\right)={\lambda }_2\mathbb{E}\left[{({∥{\nabla }_{\widetilde{S_n}}{D}_n(I_n,{\tilde{S}}_n)∥}_2-1)}^2\right],\hfill \end{array}\right.$$

where ${\lambda }_1$ and ${\lambda }_2$ are balancing parameters, the specific values for these parameters will be discussed in Section 3.4. Furthermore, $\tilde{S}n$ is a random variable sampled uniformly from either $S_n$ or $\widehat{S}n$. ${\mathcal{L}}_{WGAN}$ refers to the error function of WGAN, which provides more stable training compared to the logarithmic formulation in Equation (2). ${\mathcal{L}}_{L_1}$ represents the $L_1$-norm regularization term, which imposes a penalty based on the pixel-wise dissimilarity between the ground truth $S_n$ and the produced ${\widehat{S}}_n$. ${\mathcal{L}}_{GP}$ denotes the gradient penalty term, which prevents the gradients of the model from vanishing or exploding.

The training process of PCGAN, as shown in Algorithm 1, accepts ocean internal wave images along with their corresponding stripe label images as input, and outputs the trained generators and discriminators at various scales. For example, given a training image and its corresponding label image pair $(I_0,S_0)$, and the scale number is set to upsample and downsample once, we first generate an image pyramid $(I_1,S_1)$, $(I_0,S_0)$, and $(I_1^{+},S_1^{+})$. Training starts from the coarsest layer. Firstly, the output of $G_1$ is computed by ${\widehat{S}}_1=G_1(I_1,{\widehat{S}}_2)$, where ${\widehat{S}}_2$ is set to be a fully zero image with the same size as $I_1$. And $D_1$ takes $(I_1,{\widehat{S}}_1)$, $(I_1,S_1)$ as input. Next, update the parameters of $D_1$ and $G_1$ sequentially according to Equation (5). Secondly, concatenate ${\widehat{S}}_1$ and $I_0$ as the input to $G_0$ to obtain the output ${\widehat{S}}_0=G_0(I_0,{\widehat{S}}_1)$. Update $D_0$ and $G_0$ using Equation (5). Finally, similar to other scales, generate ${\widehat{S}}_1^{+}=G_1^{+}(I_1^{+},{\widehat{S}}_0)$ and then train $D_1^{+}$ and $G_1^{+}$ sequentially. Following this, the pair $(I_0,S_0)$ repeats the training process, iteratively updating the parameters of PCGAN. After completing the designated training epochs, the next image pairs in the training set will undergo the same training steps as $(I_0,S_0)$.

Algorithm 1 Training of PCGAN for Stripe Extraction of Oceanic Internal Waves

Input: Remote sensing images and their corresponding labeled internal wave images constitute the training set  1: for all training epochs do  2:   Create image pyramid and initialize ${\widehat{S}}_{N+1}$ to 0  3:   for all scales do  4:    Take the previously generated ${\widehat{S}}_{n+1}$ and the image $I_n$ as input.  5:    Generate ${\widehat{S}}_n$ using Equation (4)  6:    Maintain the parameters of $G_n$ and optimize $D_n$ utilizing Equation (5)  7:    Maintain the parameters of $D_n$ and optimize $G_n$ utilizing Equation (5)  8:   end for  9: end for Output: A collection of trained models at every scale.

3. Experiments and Results

3.1. Data Preparation

Figure 4 depicts a flow chart outlining the standard process of extracting ISW information from a SAR image [40]. The image, captured at 21:56 UTC on 8 March 2019, undergoes preprocessing steps from SAR image correction to geometric correction. Following this, the enhanced image is processed, leading to the manual extraction of the position information of ISWs. However, manual extraction is time-consuming. Our proposed PCGAN is designed to address the final step in Figure 4, employing machine learning techniques to automatically extract the accurate position of the wave crest of ISWs from local images.

To achieve this goal, the original SAR images are obtained from the northern section of the South China Sea, globally acknowledged as the “natural experimental field” for investigating oceanic Internal Solitary Waves [41]. Figure 5 shows the distribution of internal waves in the region from 2010 to 2020 in June and July [40]. Specifically, we use Sentinel-1A/B SAR images with the interferometric wide swath (IW), featuring an image width of 250 km and a spatial resolution of 20 m.

From the original SAR images, we initially compiled a dataset consisting of 86 local images containing internal waves for model training and validation. The dimensions of these images varied from 150 to 950 pixels in width and height. To ensure consistency, all these images were uniformly resized to 512 × 512 pixels and subsequently subjected to manual annotation.

From Figure 6 depicted below, we have selected four pairs of images for model training using a limited sample size. Each pair comprises the original SAR image containing internal waves and its corresponding labeled image, which serves as the ground truth for our training process.

The upper row of images in Figure 6 highlights the diverse features of internal waves, illustrating their distinctive characteristics. This inherent diversity within the images contributes to a comprehensive and diverse training process. Among these images, Figure 6a presents a heightened color contrast, whereas Figure 6b showcases a lower contrast. In Figure 6c, there exists a minor coda presence, whereas Figure 6d displays more prominent coda interference (‘coda’ means the secondary or trailing part of the stripes that follows the primary wave); however, only the most pronounced portion is annotated in the label image. This ensures that the model also prioritizes the primary wave within the internal wave image.

Eighty-two (82) additional images, featuring distinct content but matching the size of the four images in the training set, were compiled to form the initial test set. To further validate the effectiveness and generalization of the model, an extra 40 images from the South China Sea and Andaman Sea were incorporated into the test set. Importantly, these images were not resized, ensuring that they differ in image size from the training set. Subsequently, all image pairs in both the testing and training datasets underwent normalization, scaling their values to the range of $[0,1]$.

3.2. Experimental Environment and Basic Parameters

On a 64-bit Ubuntu 18.04.6 system, we utilized the PyTorch 1.12.1 machine learning framework for implementing and testing the model. The setup included the CUDA toolkit 9.1.85 software and an NVIDIA A100 GPU.

PCGAN underwent a training regimen comprising 5000 epochs, with each scale iteration set to 1. For the training of individual generators ($G_n$) and discriminators ($D_n$), we harnessed the Adam optimizer, employing ${\beta }_1=0.5$ and ${\beta }_2=0.999$. A learning rate of $0.0005$ was designated for each network, while a minibatch size was 1.

At each scale, both the generator and discriminator convolutional neural networks consist of five convolutional layers. These convolutional layers have a filter size of $3\times 3$, a stride of 1, and padding set to TRUE, ensuring that the dimensions of the input and output images remain unchanged. The kernel depth and activation functions for each convolutional layer are detailed in

Table 1. The kernel depth and activation functions for each convolutional layer.

Layers	Initial Layer	Middle Layers	Final Layer
			Generator	Discriminator
Kernel depth	64	32	$c=3$	1
Activation functions	LReLU	LReLU	Tanh	null

, which categorizes them into the initial layer (Layer 1), the middle layers (Layers 2 to 4), and the final layer (Layer 5).

Here, the generator and discriminator share the same parameters in the initial layer and the middle layers. They only differ in the final layer, where the generator’s kernel depth is set to c, which needs to match the input image’s channel. In our task, the input $I_n$ is an RGB three-channel SAR image, so c is set to 3. LReLU represents the Leaky ReLU function.

3.3. Evaluation Criteria

To assess the efficacy of the proposed model and make comparisons with other models, we use four commonly used evaluation indicators for segmentation models: MIoU, F1-Score, MACC, and FWIoU. MACC calculates the average of all class accuracies (6). The F1-Score, calculated as the harmonic mean of Recall and Precision (as shown in Formulas (7) and (8)), is a widely employed performance metric in binary classification tasks. MIoU (Mean Intersection over Union) measures the overlap between the predicted segmentation regions by the model and the actual labels. The process involves calculating the Intersection over Union (IoU) for each class, which in our task includes the internal wave class and the background class. Subsequently, the average of IoUs across these two classes is computed (as Formula (10)). FWIoU (Frequency Weighted Intersection over Union) is an enhanced metric derived from MIoU, which considers the occurrence frequency of each class. It is the weighted average of the IoUs for these two classes (11).

$$\begin{array}{ccc}\hfill \mathrm{MACC}& =& \frac{1}{2}(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}+\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}),\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill \mathrm{Recall}& =& \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}},\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}},\hfill \end{array}$$

(7)

$$\begin{array}{ccc}\hfill \mathrm{F}1\text{-}\mathrm{Score}& =& 2·\frac{\mathrm{Recall}·\mathrm{Precision}}{\mathrm{Recall}+\mathrm{Precision}},\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill \mathrm{Frequency}& =& \frac{\mathrm{TP}+\mathrm{FN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}},\hfill \end{array}$$

(9)

$$\begin{array}{ccc}\hfill \mathrm{MIoU}& =& \frac{1}{2}(\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}}+\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FN}+\mathrm{FP}}),\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill \mathrm{FWIoU}& =& \mathrm{Frequency}·\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}\hfill \\ & +& (1-\mathrm{Frequency})·\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}+\mathrm{FN}}.\hfill \end{array}$$

(11)

Here, the number of pixels correctly recognized as “internal wave” is denoted by TP (true positive), while FN (false negative) represents the pixels erroneously missed and not identified as “internal wave”. FP (false positive) corresponds to the number of pixels misclassified as “internal wave” by the model, although they are labeled as “non-internal wave” in the ground truth dataset. Lastly, TN (true negative) represents the accurate identification of pixels as “non-internal wave”, indicating the background pixels that were correctly classified.

3.4. Selection of Balancing Parameters

To determine the optimal balance parameters for the input image, we also conducted a series of preliminary experiments. The $L_1$-norm constraint parameter (${\lambda }_1$) and the weight assigned to the gradient penalty in the WGAN-GP loss term (${\lambda }_2$) were set to 5, 10, and 20, respectively. The model underwent training with the identical training dataset, and subsequent evaluation was carried out on the test dataset.

Table 2. Average MIoUs of different balancing parameters.

	${\mathit{\lambda }}_1$	5	10	20
${\mathit{\lambda }}_2$
5		0.6080	0.5641	0.5861
10		0.6130	0.6315	0.6189
20		0.5771	0.5950	0.6036

displays the average MIoU values corresponding to these parameter combinations on the test set. The highest values is highlighted in the cell with a gray background.

Based on the metrics, we opted to set the values of both balance parameters to 10 during the subsequent training process.

3.5. Selection of the Number of Upsample and Downsample Layers

To ascertain the ideal count of upsampling and downsampling layers, we conducted a series of preliminary experiments.

Table 3. Performance metrics and training duration across different scales.

Downsample Layer	Upsample Layer	MIoU	Training Time (min)
0	0	0.5398	50.29
	1	0.5803	241.44
	2	0.5851	1033.20
1	0	0.6177	85.13
	1	0.6238	273.94
	2	0.6278	1073.79
2	0	0.6128	108.19
	1	0.6224	295.12
	2	0.6274	1127.28

presents the training duration and MIoU values for the model with different configurations, where the upsampling and downsampling layers are set to 0, 1, and 2. Both the testing and training datasets comprise $256\times 256$ images, with the model trained for 5000 epochs. It can be observed that while the two-layer upsampling shows a slight advantage in evaluation metrics over the one-layer upsampling, its training time is nearly four times that of the one-layer upsampling model. Moreover, due to memory constraints, the model with two upsampling layers can only handle images of half the size compared to the one-layer model. Therefore, we opt for one layer of upsampling.

After determining the number of upsampling layers, we reverted the dimensions of both the training and testing set back to the original $512\times 512$ size.

Table 4. Fixed upsampling layers at one, different MIoU scores for various downsampling layers.

Downsample Layer	Upsample Layer	MIoU
0	1	0.6012
1		0.6003
2		0.6289
3		0.6102

displays the average model MIoU scores under different downsampling layer configurations. The highest values is highlighted in the cell with a gray background. As the number of downsampling layers increases, the size of the internal wave images to be identified, denoted as $I_N$, becomes progressively smaller, making it challenging to provide meaningful overall information. In such a context, the increasing number of downsampling layers N does not significantly enhance the model’s detection capability.

Therefore, guided by the outcomes of the preliminary experiments, we fix the number of upsampling layers M to 1, and choose 2 for the number of downsampling layers N.

3.6. Experimental Results and Comparative Analysis

In this segment, we conducted a performance comparison between PCGAN and four other methods, encompassing two traditional approaches: Adaptive Thresholding (AT) [42] and Canny edge detection [43], as well as two deep learning methodologies: Conditional Generative Adversarial Network (CGAN) and U-Net [44]. For a fair comparison, we utilize the identical training set, comprising only four pairs of images as depicted in Figure 6, to separately train these three deep learning models. Additionally, we use identical hyperparameters and conduct training for 5000 epochs.

Table 5. The performance of internal wave detection on nine representative images in the test set.

Approach	Metrics	I	II	III	IV	V	VI	VII	VIII	IX
AT	MACC	0.9776	0.9762	0.8894	0.8796	0.9093	0.8498	0.8741	0.9563	0.9738
	F1-Score	0.1918	0.1752	0.0736	0.0527	0.0790	0.1010	0.0980	0.1192	0.2459
	MIoU	0.5418	0.5361	0.4636	0.4531	0.4751	0.4508	0.4624	0.5098	0.5569
	FWIoU	0.9700	0.9694	0.8819	0.8700	0.8944	0.8334	0.8580	0.9487	0.9556
Canny	MACC	0.9839	0.9848	0.9801	0.9763	0.9797	0.9650	0.9655	0.9833	0.9746
	F1-Score	0.2722	0.2694	0.1720	0.2055	0.3207	0.3045	0.3017	0.1997	0.2244
	MIoU	0.5707	0.5702	0.5371	0.5454	0.5853	0.5722	0.5714	0.5471	0.5505
	FWIoU	0.9767	0.9784	0.9728	0.9669	0.9664	0.9498	0.9506	0.9760	0.9561
CGAN	MACC	0.9946	0.9966	0.9946	0.9918	0.9842	0.9797	0.9392	0.9907	0.9830
	F1-Score	0.6253	0.7561	0.6758	0.4636	0.1193	0.5179	0.3203	0.1806	0.4273
	MIoU	0.7247	0.8022	0.7524	0.6468	0.5238	0.6645	0.5645	0.5450	0.6273
	FWIoU	0.9899	0.9936	0.9905	0.9843	0.9687	0.9676	0.9245	0.9832	0.9675
U-Net	MACC	0.9945	0.9956	0.9951	0.9910	0.9877	0.9867	0.9841	0.9879	0.9796
	F1-Score	0.5934	0.6404	0.7676	0.5572	0.5471	0.5591	0.5518	0.4827	0.1416
	MIoU	0.7082	0.7333	0.8091	0.6886	0.6821	0.6873	0.6825	0.6530	0.5279
	FWIoU	0.9896	0.9916	0.9915	0.9844	0.9774	0.9753	0.9728	0.9822	0.9600
PCGAN	MACC	0.9941	0.9944	0.9954	0.9922	0.9870	0.9840	0.9835	0.9919	0.9812
	F1-Score	0.6638	0.6599	0.7171	0.6867	0.6372	0.5155	0.5845	0.5587	0.5390
	MIoU	0.7454	0.7434	0.7770	0.7575	0.7272	0.6656	0.6981	0.6897	0.6749
	FWIoU	0.9897	0.9905	0.9923	0.9871	0.9781	0.9719	0.9727	0.9867	0.9677

presents an assessment of the performance of AT, Canny, CGAN, U-net, and PCGAN on individual internal wave images, employing the metrics outlined in Section 3.3. Furthermore, Figure 7 visually presents the detection outcomes for four distinct test scenarios, providing insights into the comparative performance of the detection models. Figure 8 illustrates the recognition results of four internal wave images with different sizes, all from the Andaman Sea and featuring multiple wave crests within a single image.

Table 6. The mean metrics for test images with the same size across the five methods.

Method	MACC	F1-Score	MIoU	FWIoU
AT	0.9307	0.1154	0.4966	0.9174
Canny	0.9755	0.2034	0.5459	0.9626
CGAN	0.9735	0.2719	0.5759	0.9612
U-Net	0.9855	0.3065	0.5964	0.9734
PCGAN	0.9849	0.4177	0.6315	0.9739

provides the average metrics across all test images of the same size and compares the performance of the five methods.

Table 7. The mean metrics for test images with different sizes across the three deep learning methods.

Method	MACC	F1-Score	MIoU	FWIoU
CGAN	0.6176	0.2577	0.5435	0.8919
U-Net	0.6099	0.2985	0.5735	0.9244
PCGAN	0.7593	0.4538	0.6307	0.9314

presents the average metrics for an additional set of 40 images from the Andaman Sea and the South China Sea with varying sizes across three deep learning methods. In the table, the highest values for each metric are highlighted in cells with a gray background.

Figure 9 displays box plots illustrating the four evaluation metrics discussed in Section 3.3 for the five detection methods. While our method may have a slightly lower maximum value compared to CGAN and U-Net, it outperforms other models in terms of both mean performance and stability.

4. Discussion

4.1. Descriptive Assessment

Figure 7 showcases SAR images with distinct features along with the detection results obtained using five different models. In (a), there is a high contrast between the stripes and the background, while in (b), the stripes are similar to the background, resulting in low contrast. In (c), in addition to simple stripes, there is a minor coda presence, and in (d), there is a more pronounced coda wave interference.

In these scenarios, as depicted in Figure 7c, compared to other models, PCGAN captures finer details of internal waves, exhibiting a closer resemblance to the mask image. For Figure 7b, where the features of the internal wave image are less pronounced, other models perform poorly, while PCGAN provides a notably superior detection result. In Figure 7a,d, even in situations where CGAN and U-Net yield subpar results, PCGAN still produces detection results that closely align with the reference data.

In Figure 8, the recognition results of internal wave images with four different sizes are demonstrated under three distinct deep learning methods. These images all contain multiple wave crest lines. It can be observed from the figure that, in terms of recognition completeness and continuity, PCGAN is noticeably superior to the other two methods, exhibiting a closer resemblance to the reference data.

This qualitative evidence supports the superior performance of PCGAN over other approaches to comprehensiveness and stability.

4.2. Measurable Assessment

The performance metrics for nine internal wave images across the five segmentation methods are displayed in Table 5. In each column, the evaluation criteria with the highest scores are shaded in gray. Overall, the table indicates that PCGAN achieves higher values across the assessment metrics.

Table 6 provides the mean performance metrics of the five detection models, providing a comprehensive overview of their collective performance across the entire set of test images of the same size. PCGAN exhibits slightly lower MACC than U-Net, which can be attributed to the fact that internal wave stripes are often narrow and occupy only a small fraction of the entire image, potentially compromising the efficacy of the MACC metric. For the remaining three metrics, PCGAN consistently achieves higher average scores compared to the other four detection models.

Table 7 presents the average metrics for 40 images of different sizes under three deep learning methods, with 20 from the South China Sea and 20 from the Andaman Sea. From the results, it can be observed that PCGAN outperforms the other two models in all four evaluation metrics. This demonstrates the superior generalization capability of PCGAN, enabling it to handle images from different maritime regions and varying sizes effectively.

The data utilized for generating the boxplots in Figure 9 encompass all test images. Although PCGAN did not perform the best in terms of the maximum values across the four evaluation metrics, it consistently demonstrates higher median evaluation scores and greater overall detection robustness. This provides quantitative evidence that PCGAN improves internal wave detection across a range of contexts.

Through the above discussion, both visual and numerical assessments have validated the advantages of the pyramid architecture, reducing the model’s demand for extensive training data. Remarkably, PCGAN demonstrated exceptional proficiency in internal wave detection even with just four training data pairs. Consequently, PCGAN stands as an effective solution for addressing internal wave detection with constraints on training data availability.

5. Conclusions

In this research, a Pyramidal Conditional Generative Adversarial Network (PCGAN) was designed to facilitate the learning of an internal wave stripe extraction model, even when the training dataset is limited. The PCGAN possesses strong capabilities for extracting wave stripes due to the following features:

(1)

Pyramid Structure: The PCGAN incorporates a pyramid structure, integrating multiple scales of conditional generative adversarial networks. This architecture ensures a stable and efficient training process.

(2)

Data Flow: The model employs a data flow from coarse to fine across different scales, enabling it to effectively capture both global and local information.

(3)

Upsampling Enhancement: The introduction of upsampling significantly enhances the model’s ability to extract fine features, particularly effective in capturing wave crests.

(4)

Parameter Optimization: To achieve optimal performance, a series of preliminary experiments were conducted to fine-tune the model parameters.

Subsequently, the model’s performance on the training set was showcased and compared with other models. The experimental results conclusively affirm that, even when trained with only four sets of training data, the well-trained PCGAN excels in accurate stripe extraction of oceanic internal waves. Furthermore, it demonstrates robust stability and generalization capabilities.

Furthermore, there are areas for potential improvement in future research. For instance, enhancing the continuity of stripe generation and reducing noise in image generation, as well as enhancing the capability to recognize internal waves in remote sensing images from larger areas and more complex environments. Subsequent studies will also explore the use of extracted internal wave stripes for investigating internal wave propagation and inverting parameters.