A Deep Learning Phase Noise Compensation Network for Photonic Terahertz OFDM System

Abstract

To address the phase noise issue in terahertz OFDM system, this paper proposes a dual-branch deep learning phase noise compensation network named AdaPhaseNet. The Transformer branch of this network leverages the powerful modeling capability of Transformers for long-range dependencies to achieve long-range phase noise estimation and compensation, while the CNN branch is employed for local signal enhancement. Finally, an optimized signal is output through a confidence-driven adaptive fusion module. For experimental validation of the algorithm, we constructed a photonic terahertz communication system comprising 10 km of fiber and 5 m of wireless transmission. Experimental results show that, compared with multiple baseline models, AdaPhaseNet achieves relative BER reductions ranging from 37.0% to 57.9% and EVM gains ranging from 1.4 dB to 3.2 dB.

1. Introduction

Terahertz communication technology, with its vast available bandwidth, is widely regarded as one of the key enabling technologies for the future sixth-generation (6G) wireless communication systems. In recent years, it has attracted extensive research interest from both academia and industry [1]. To achieve extremely high data rates, orthogonal frequency division multiplexing (OFDM) has become the primary physical-layer waveform candidate for terahertz communication systems, owing to its high spectral efficiency and strong resistance to multipath fading [2]. However, at terahertz frequencies, communication systems face unprecedented phase noise challenges. The phase perturbations introduced by radio frequency oscillators not only cause a common rotation across all subcarriers, known as common phase error (CPE), but also destroy the orthogonality among subcarriers, leading to severe inter-carrier interference (ICI), which significantly limits system performance [3].

To address this critical challenge, researchers have developed various phase noise suppression techniques, which can be broadly categorized into two classes: model-driven traditional signal processing methods and data-driven machine learning approaches [4,5]. Model-driven methods typically rely on pilots and explicit assumptions. In broadband terahertz scenarios, such methods often require dense pilot patterns and incur high computational complexity. Learning-based approaches can learn the compensation mapping directly from data; however, under practical operating conditions, it remains challenging for a single network architecture to simultaneously achieve reliable long-range phase tracking and effective suppression of composite impairments.

To overcome the limitations of existing phase noise compensation schemes in photonic terahertz OFDM system, this paper proposes a novel dual-branch adaptive compensation network named AdaPhaseNet. In this architecture, the Transformer branch focuses on the long-range temporal estimation and compensation of phase noise, while the CNN branch is responsible for enhancing the local features of the received signal to suppress non-phase impairments such as additive noise. The outputs of the two branches are dynamically weighted and fused in an adaptive fusion module, based on confidence scores provided by the Transformer branch, to generate the optimized signal. System performance is evaluated using two key metrics, bit error rate (BER) and error vector magnitude (EVM), and compared with baseline models including recurrent neural networks, long short-term memory networks, a single CNN, and a single Transformer. Experimental results demonstrate that AdaPhaseNet significantly outperforms existing methods in both compensation accuracy and system robustness.

The main contributions are summarized as follows:

• A dual-branch adaptive phase noise compensation network named AdaPhaseNet is proposed and integrated into the digital signal processing chain at the receiver of a photonic terahertz OFDM system.
• The impact of different input window sizes on phase noise estimation accuracy is evaluated, determining the optimal receptive field for the Transformer branch in AdaPhaseNet.
• The impact of key parameters in AdaPhaseNet on system performance was analyzed, providing a basis for optimizing the network’s core components.
• The proposed algorithm was experimentally validated. A comprehensive comparison between AdaPhaseNet and multiple baseline models was conducted in terms of BER and EVM performance, validating its superiority.

The remainder of this paper is organized as follows. Section 2 reviews related works on phase-noise compensation for OFDM systems. Section 3 presents the proposed AdaPhaseNet and the methodology. Section 4 describes the experimental setup. Section 5 analyzes and optimizes the key parameters of AdaPhaseNet. Section 6 discusses the experimental results. Section 7 concludes the paper and outlines future research directions.

2. Related Works

As mentioned in Section 1, phase noise compensation techniques for OFDM systems can generally be categorized into two types: conventional model-driven signal processing methods and data-driven learning-based approaches. Therefore, this section systematically reviews the research progress of both categories, analyzes their inherent limitations in depth, and, based on this discussion, further highlights the advantages of our proposed method for phase-noise compensation.

Traditional signal processing methods are built upon rigorous mathematical derivations. Their core idea is to utilize pilots or training sequences to estimate and compensate for the impact of phase noise [6]. Among these, CPE compensation serves as a fundamental scheme, which estimates and corrects the common phase rotation across subcarriers. However, it completely ignores ICI, resulting in limited performance gains [7]. To suppress ICI, more advanced schemes have been proposed. For instance, ICI cancellation algorithms based on frequency-domain comb-type pilots and the minimum mean square error (MMSE) criterion, or methods that model phase noise as time-varying parameters and employ Kalman filtering for tracking [8,9,10,11]. Although these traditional algorithms have clear structures and have been extensively studied, their inherent limitations become particularly prominent in terahertz communication scenarios [12]. First, high-precision ICI compensation typically requires dense pilot patterns, which contradicts the core goal of pursuing extremely high spectral efficiency in terahertz communications. Second, the complex matrix operations involved lead to computational complexity that grows exponentially with the system bandwidth [13], making them difficult to apply in broadband terahertz transmission scenarios [14,15].

To overcome the bottlenecks of traditional methods, data-driven approaches based on machine learning have emerged. These methods move away from the idea of explicitly modeling phase noise mathematically. Instead, they directly learn an end-to-end recovery mapping from the impaired signal to the ideal signal from large volumes of data [16]. For example, deep neural network (DNN) have been employed to construct end-to-end receivers, implicitly performing joint channel equalization and phase noise suppression [17]. Generative adversarial network (GAN) have been explored for generating training data or directly participating in the compensation process [18]. Convolutional neural network (CNN) have been used to exploit the local correlations among subcarriers in the frequency domain to suppress ICI [19]. Although these data-driven schemes demonstrate significant potential, they commonly suffer from a key drawback: difficulty in effectively capturing the inherent temporal dependencies of phase noise. It is important to note that phase noise is not white noise; its variation process exhibits significant memory or long-range correlation [20]. This essential characteristic makes architectures like CNN, which primarily rely on local spatial feature extraction, less effective for achieving optimal compensation performance.

As shown in Figure 1, we analyze the temporal characteristics of phase noise under different laser linewidth conditions through simulations. Figure 1a illustrates the time-domain evolution of phase noise $\varphi \left(t\right)$. When the laser linewidth is ${10}^4$ Hz, the fluctuation amplitude of phase noise is constrained within approximately $\pm 0.03$ rad over short time scales, exhibiting a relatively smooth and slow-varying characteristic. As the linewidth increases to ${10}^5$ Hz and ${10}^6$ Hz, the fluctuation amplitude of phase noise intensifies significantly, exceeding $\pm 0.05$ rad and $\pm 0.25$ rad, respectively. Moreover, the time-domain waveform shows more high-frequency and severe random fluctuations, indicating that the dynamic variation rate of phase noise significantly increases with linewidth. Figure 1b further reveals the temporal memory characteristics of phase noise through the autocorrelation function (ACF) curves. It can be observed that a smaller laser linewidth leads to a slower decay of the ACF, implying that the phase error at the current moment maintains a high correlation with historical values over a longer period, i.e., phase noise has a longer “memory length”. Conversely, when the linewidth increases, the ACF decay rate accelerates sharply, indicating that the temporal correlation of phase noise weakens rapidly, its randomness enhances, and the effective memory length shortens.

Based on the analysis above, phase noise, especially when the linewidth is small, constitutes a non-stationary process with significant long-range dependencies. This key observation provides important theoretical justification for employing the Transformer architecture in this work, owing to its effectiveness in capturing long-range temporal dependencies. Given the inherent temporal memory characteristics of phase noise, recurrent neural network (RNN) and their improved variants, such as long short-term memory (LSTM) networks, have been introduced into this field to explicitly model historical dependencies [20]. Through their internal gated mechanisms, these networks aim to selectively remember and forget information, thereby theoretically enabling the capture of temporal dynamic features within sequences. However, RNN and LSTM often face challenges related to gradient vanishing or explosion during practical training. This essentially limits their effective memory span, making it difficult for them to stably learn long-term dependencies from distant history in long sequences.

In recent years, the Transformer architecture, based on the self-attention mechanism, has successfully overcome the limitations of RNN-based models in modeling long-range dependencies. Its core strength lies in its powerful capability for global sequence awareness—by computing the relevance weights between any positions in a sequence in parallel, it achieves dynamic and efficient integration of information across the entire sequence [21]. This ability to directly capture long-range contextual dependencies is particularly crucial for accurately estimating signals with significant temporal correlation, such as phase noise, thus providing a more ideal modeling tool for such problems.

3. Methodology

In this section, we will explain AdaPhaseNet in detail, including its advantages, workflow and the core methods of its sub-modules.

3.1. AdaPhaseNet

A single neural network architecture struggles to simultaneously accomplish the two critical tasks of “long-range phase tracking” and “local signal enhancement.” Specifically, while the Transformer excels at modeling the temporal dependencies of phase noise, it exhibits limitations in directly handling composite impairments such as additive noise and nonlinear distortion. Conversely, the CNN is proficient in local feature extraction and signal enhancement, but its inherently limited receptive field prevents it from effectively compensating for phase noise with long memory.

Building on this complementary nature, this paper innovatively proposes a novel dual-branch adaptive compensation network named AdaPhaseNet, the overall structure of which is shown in Figure 2. This architecture aims to integrate the strengths of the Transformer and the CNN, achieving an intelligent balance between the strategies of “accurate phase rotation correction” and “signal enhancement” through an adaptive fusion mechanism.

The workflow of this architecture comprises three core steps:

• Feature Extraction and Branch Input: The received noisy baseband signal undergoes preprocessing and is converted into two distinct feature representations. One path is used to generate a phase difference sequence, serving as the input to the Transformer branch to focus on the temporal characteristics of the phase noise. The other path retains the original I/Q waveform signals, which are fed into the CNN branch for local feature learning.
• Specialized Subnetwork Processing: The two branches process the features in parallel. Leveraging its global self-attention mechanism, the Transformer branch accurately estimates the long-range correlated phase noise while simultaneously outputting a confidence score for its estimate. The CNN branch, through its hierarchical convolutional structure, performs blind enhancement on the original signal to effectively suppress non-phase impairments such as additive noise and inter-symbol interference.
• Adaptive Fusion Output: The preliminary outputs from the two branches are combined in an adaptive fusion module. This module dynamically adjusts the contribution weight of each branch in the final output based on the real-time confidence provided by the Transformer branch, followed by a weighted fusion to generate the purified signal. This mechanism ensures the system maintains optimal compensation performance under various channel conditions.

3.2. Transformer

Transformer is a deep learning architecture based on self-attention. Its core lies in efficiently modeling global dependencies by computing correlation weights between any two elements in a sequence in parallel. This mechanism dynamically assigns attention weights to each position in the sequence by computing interactions between query (Q), key (K) and value (V) vectors. For a received symbol sequence of length L, $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,...,{\mathbf{x}}_L]$, the attention output Z is computed as follows:

$$\mathrm{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}}\right)\mathbf{V}$$

(1)

where $\mathbf{Q}=\mathbf{X}{\mathbf{W}}^{\mathbf{Q}}$, $\mathbf{K}=\mathbf{X}{\mathbf{W}}^{\mathbf{K}}$, $\mathbf{V}=\mathbf{X}{\mathbf{W}}^{\mathbf{V}}$ are obtained by linear projection of the input sequence, $d_k$ is the dimension of the key vector, and the scaling factor ${\displaystyle \frac{1}{\sqrt{d_k}}}$ is used to stabilize gradients. The softmax function normalizes the attention weights, enabling adaptive weighted aggregation of historical information. Due to its parallel computing characteristics, Transformer can directly capture interactions between any long-distance positions in the sequence, significantly improving the modeling capability of long-range dependencies. Although this architecture originated in the field of natural language processing, its powerful sequence modeling capability has rapidly extended to time-series data tasks such as computer vision, speech recognition, and communication signal processing. Phase noise estimation and compensation is essentially a typical time-series modeling problem, and the self-attention mechanism of Transformer provides an ideal tool for accurately characterizing the temporal dependencies of phase noise. As shown in Figure 3, the self-attention mechanism allows the current symbol to attend to all contextual information (both historical and future symbols) in the entire input sequence, where the color of the dots is light blue.

In this paper, we introduce the Transformer architecture into the phase noise estimation and compensation task in photonic terahertz OFDM systems. To satisfy the causality constraint of communication systems, a causal mask is used during both training and inference, ensuring that the estimation at time t depends only on historical information (from time 1 to time $t-1$), preventing any leakage of future information. Specifically, for an input sequence of length L, a causal mask matrix $\mathbf{M}\in {\mathbb{R}}^{L\times L}$ is defined as:

$$M_{ij}=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}j\le i\hfill \\ -\infty ,\hfill & \mathrm{if}j>i\hfill \end{array}\right.\mathrm{for}i,j=1,2,\dots ,L$$

(2)

where index i denotes the position of the Query ($\mathbf{Q}$) and index j denotes the position of the Key ($\mathbf{K}$). In the self-attention computation, this causal mask is added to the original attention score matrix. The modified attention score matrix $\tilde{\mathbf{S}}$ is calculated as:

$$\tilde{\mathbf{S}}={\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}}+\mathbf{M}$$

(3)

Finally, the attention weights $\mathbf{A}$ are obtained through the Softmax function:

$$\mathbf{A}=\mathrm{softmax}\left(\tilde{\mathbf{S}}\right)={\displaystyle \frac{exp\left(\frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}+\mathbf{M}\right)}{{\sum }_{j=1}^{L}exp\left({\left(\frac{\mathbf{Q}{\mathbf{K}}^{\top }}{\sqrt{d_k}}+\mathbf{M}\right)}_{i,j}\right)}}$$

(4)

Leveraging Transformer’s ability to capture long-range dependencies, we construct a phase noise estimator. This estimator takes a feature sequence composed of the instantaneous phase differences of the received signal as input, extracts deep temporal features through a multi-layer Transformer encoder, and finally outputs both the estimated phase noise and its confidence. Compared to traditional algorithms that rely on preset models or local stationarity assumptions, this data-driven method can directly learn the complex statistical characteristics and evolution patterns of phase noise from data, making it particularly suitable for handling high-intensity, non-stationary phase noise scenarios. This provides a new solution for enhancing the reliability of high-speed terahertz communication systems.

3.3. CNN

Convolutional Neural Network (CNN) is a type of deep learning architecture specifically designed to process data with a grid-like topology (such as images or time-series signals) that possesses the characteristics of local correlation and translation invariance. In this study, the CNN is constructed as a signal enhancement branch operating in parallel with the phase noise estimation branch. Its primary objective is to perform blind enhancement and purification of mixed impairments present in the received signal, excluding phase noise—such as additive noise, residual channel distortion, and nonlinear distortion. This branch takes the I/Q components of the original noisy signal as input. Through stacked one-dimensional convolutional layers, it automatically extracts the local structural features of the signal, thereby learning a nonlinear mapping function from the impaired signal to an enhanced signal.

The design core of this CNN branch lies in its hierarchical feature extraction and fusion mechanism. The network is composed of multiple stacked convolutional modules, with each module typically containing a one-dimensional convolutional layer, a batch normalization layer, and a ReLU activation function. The shallow convolutional kernels are responsible for capturing local, fine-grained features such as amplitude distortion and impulse noise. As the network depth increases, the receptive field progressively expands, allowing deeper layers to identify medium- to long-range waveform distortion patterns. Ultimately, the network reconstructs the enhanced signal through global feature aggregation. The entire processing flow does not rely on any explicit channel or noise models. It is entirely data-driven, thus enabling adaptive suppression of various unknown mixed impairments and improving the overall signal quality.

3.4. Adaptive Fusion Module

To intelligently integrate the outputs of the dual branches, we design an adaptive fusion module. This module takes the results from the front-end parallel processing as its input. The Transformer branch provides the signal $S_{\mathrm{trans}}$ along with its corresponding confidence score x. $S_{\mathrm{trans}}$ is the signal obtained after preliminary phase noise compensation and subsequent feature fusion with the CNN branch. Concurrently, the CNN branch provides the signal $S_{\mathrm{cnn}}$, which is the result after processing composite impairments. Subsequently, the fusion weight is dynamically calculated based on the confidence score x, and a weighted fusion of the two signals is performed to generate the final purified output signal $S_{\mathrm{out}}$.

The calculation mechanism for the confidence-based weight $w_t$ is defined as follows. A threshold is set at $\theta =0.7$. When the confidence is high ($x\ge \theta$), it indicates that the phase noise estimation is relatively reliable. In this case, the fusion weight leans towards the Transformer branch. Conversely, when the confidence is low ($x<\theta$), greater reliance is placed on the robust signal enhancement result provided by the CNN branch. The weight $w_t$ is calculated as follows:

$$w_t=\left\{\begin{array}{cc}0.3x+0.4,\hfill & \mathrm{if}x\ge \theta \hfill \\ 0.2x+0.1,\hfill & \mathrm{if}x<\theta \hfill \end{array}\right.$$

(5)

Finally, the output signal $S_{out}$ is generated by the following formula:

$${\mathbf{S}}_{\mathrm{out}}=w_t·{\mathbf{S}}_{\mathrm{trans}}+(1-w_t)·{\mathbf{S}}_{\mathrm{cnn}}$$

(6)

This design enables the system to smoothly and adaptively switch between the two strategies of “high-precision phase tracking” and “robust signal enhancement”, thereby maintaining excellent overall performance under various channel conditions.

4. Experimental Setup

For algorithm validation, we constructed an experimental platform for a photonic terahertz communication system. The link diagram is shown in Figure 4, and the equipment used in the experiment is presented in Figure 5. At the transmitter, test data generated by a PRBS generator is modulated using 16QAM and 128-point IFFT-based OFDM. The experimental link operates at a bit rate of 4 Gbps. The modulated symbols are processed by a DAC with 12× oversampling to produce the baseband signal. Lasers ECL1 and ECL2 emit continuous waves CW1 and CW2 at wavelengths of 1550 nm and 1547.44 nm, respectively. The baseband signal from an arbitrary waveform generator (AWG) is amplified by an electrical amplifier (EA) and then impressed onto CW1 via an I/Q modulator. The modulated optical signal is further amplified by an erbium-doped fiber amplifier (EDFA), and its polarization is adjusted by a polarization controller (PC) before being combined with CW2 in an optical coupler (OC). The coupled signal travels through 10 km of standard single-mode fiber (SSMF) and enters a variable optical attenuator (VOA) to adjust the optical power to a level suitable for the uni-traveling-carrier photodiode (UTC-PD). The attenuated signal is fed into the UTC-PD to generate the photonic terahertz signal via heterodyne beating. This signal is radiated by the HA1, propagates over 5 m of free space to the receiver, and is focused using lenses (L1 and L2) to mitigate free-space loss.Notably, to focus on validating the effectiveness of the proposed algorithm and to reduce the difficulty of reproduction, we intentionally do not employ a terahertz power amplifier in the experimental setup.

At the receiver, the terahertz signal captured by antenna HA2 is mixed with a 327 GHz sinusoidal wave from a local oscillator (LO), down-converted to a 7 GHz intermediate frequency (IF) signal, and digitized by a digital oscilloscope (DSO) operating at 100 GSa/s. Finally, the captured signal undergoes the RX-DSP steps, including down-conversion, resampling, OFDM demodulation, and AdaPhaseNet-based processing, to recover the original data. AdaPhaseNet is placed after parallel to serial conversion and before symbol demapping, where the CPE/ICI induced by phase noise and residual local distortions are most prominent. This deployment enables the network to directly compensate for phase noise and mitigate residual local distortions after OFDM demodulation, and output high-quality signals. Without AdaPhaseNet, the receiver has to rely on the conventional DSP chain and perform symbol demapping directly after OFDM demodulation, which results in an increase in residual phase errors and degradation of the system performance under the scenario of strong phase noise.

5. AdaPhaseNet Key Parameter Analysis and Optimization

In AdaPhaseNet, the hyperparameters of the algorithmic modules and the network architecture jointly determine the theoretical performance ceiling and the practical learning efficiency of the system. Specifically, within the design of the adaptive fusion module, the choice of the threshold parameter $\theta$ has a decisive impact on system performance. In the Transformer-based phase noise estimator, the window size determines the temporal receptive field and the amount of physical information available to the network, which directly influences the theoretical upper bound of estimation accuracy. Meanwhile, hyperparameters such as batch size, network depth, and number of attention heads collectively govern the model’s ability to effectively learn features.Therefore, in this chapter, we first analyze the influence of different threshold settings on the system’s BER performance to select the optimal threshold. Subsequently, we examine the impact of window size on estimation performance to determine the optimal input dimension. Finally, we conduct systematic ablation experiments on the network architecture hyperparameters to optimize the network’s convergence and stability.

5.1. Adaptive Fusion Threshold Optimization and Performance Analysis

To determine the optimal threshold, we examined the variation in the system’s BER as $\theta$ ranged from $0.4$ to $0.9$. The results are shown in Figure 6.

The BER curve exhibits a distinct “U-shaped” trend, revealing the trade-off mechanism inherent in threshold configuration. When the threshold is set too low ($\theta =0.4$), AdaPhaseNet activates the phase correction branch too frequently. However, in scenarios where channel conditions are acceptable and phase noise is not the dominant impairment, forced phase correction may introduce unnecessary estimation errors or amplify the residual additive noise from the CNN branch, leading to an increase in BER.As $\theta$ increases, phase correction becomes more conservative, avoiding overcorrection under favorable channel conditions. Consequently, the BER decreases rapidly. At $\theta =0.7$, AdaPhaseNet achieves the best decision balance, reaching the global minimum BER. At this setting, the system can accurately determine when to rely on the Transformer branch for precise phase rotation and when to depend on the CNN branch for signal enhancement.When $\theta$ is further increased to $0.9$, AdaPhaseNet tends to conservatively maintain the CNN-enhanced signal. However, in the presence of significant phase noise, it fails to fully leverage the correction capability of the Transformer, resulting in a slow rebound in BER.

The above analysis demonstrates that $\theta =0.7$ is the optimal threshold for the system under the tested conditions. This setting ensures that the adaptive fusion module can make nearly optimal fusion decisions in complex and varying impairment environments.

5.2. Impact of Window Size on Estimation Performance

As shown in Figure 7, we examine the variation of phase noise estimation error with training epochs under different window sizes. The results indicate that the configuration with a window size of 32 exhibits the slowest convergence speed and the highest steady-state error, suggesting that a limited temporal receptive field severely constrains the Transformer model’s ability to capture the long-term correlation characteristics of phase noise. As the window size increases from 32 to 128, the model performance improves significantly: the error curve descends faster and eventually stabilizes at a lower level, verifying that expanding the temporal context window helps the self-attention mechanism identify noise features more effectively. When the window size is further increased to 256 and 512, the model achieves lower initial error early in training and maintains a stable decreasing trend throughout the training process, ultimately reaching the lowest steady-state error. This phenomenon indicates that a sufficiently large window size is a necessary condition for the model to fully learn the long-memory characteristics of phase noise and achieve high-precision, robust estimation.

It is worth noting that although the window size of 512 achieves the best accuracy performance, its corresponding computational and memory costs increase linearly with the window size. Balancing estimation accuracy, training stability, and computational efficiency, we ultimately select a window size of 256 for all subsequent experiments, aiming to maintain high accuracy while controlling model complexity.

5.3. Ablation Study and Optimization of Transformer Architecture Hyperparameters

Figure 8 presents the sensitivity analysis results of the Transformer network to key architectural hyperparameters. We systematically conduct ablation studies on critical hyperparameters including batch size (b), number of attention heads (h), network layers (l), and dropout rate (d). The experimental results indicate that the stability of the training process significantly depends on the choice of batch size. Specifically, when $b=32$, the training exhibits noticeable high-variance oscillations, suggesting considerable noise in gradient estimation, which hinders effective convergence of the optimizer. In contrast, increasing the batch size to 64 results in a smoother and more stable convergence trajectory, demonstrating superior training characteristics.

Regarding regularization, the setting of the dropout rate has a significant impact on the model fitting performance. The results show that a relatively high dropout rate of $d=0.5$ is overly aggressive for the current regression task, leading to underfitting, as evidenced by its residual error being markedly higher than that of the configuration with $d=0.1$. Furthermore, model capacity plays a crucial role in estimation accuracy: increasing the network depth from 4 to 8 layers enables the model to better learn the complex nonlinear features within the phase noise, consequently achieving the lowest validation loss among all tested configurations.

Based on the aforementioned analysis, this work ultimately selects $b=64$, $h=8$, $l=8$, and $d=0.1$ as the optimal hyperparameter configuration for the Transformer-based phase noise estimator. This combination achieves the best balance between convergence stability and estimation accuracy.

6. Results and Discussion

After confirming the optimal parameter configuration ($\theta =0.7$, $windowsize=256$, $b=64$, $h=8$, $l=8$, $d=0.1$), we conducted a systematic evaluation of the BER and EVM performance of the proposed AdaPhaseNet network against multiple baseline models. EVM is a key metric for measuring the deviation between the received signal and the ideal reference signal, directly reflecting the recovery fidelity in the signal transmission and processing chain. Its computational procedure is as follows:

First, the EVM value in the linear domain (percentage form) is computed using the core formula:

$${\mathrm{EVM}}_{\mathrm{linear}}={\displaystyle \frac{\sqrt{\frac{1}{N}{\sum }_{i=1}^N{|r_i-s_i|}^2}}{\sqrt{\frac{1}{N}{\sum }_{i=1}^N{\left|s_i\right|}^2}}}\times 100\%$$

(7)

where $r_i$ denotes the complex amplitude of the i-th symbol actually detected at the receiver (including amplitude and phase information), $s_i$ represents the complex amplitude of the ideal reference symbol at the transmitter, and N is the total number of OFDM symbols counted. To convert the linear-domain EVM to the dB form (the standard representation in the communication field), the formula is:

$${\mathrm{EVM}}_{\mathrm{dB}}=20\times {log}_{10}\left({\displaystyle \frac{{\mathrm{EVM}}_{\mathrm{linear}}}{100}}\right)$$

(8)

The comparative results are shown in Figure 9. First, in the region of low input optical power to UTC-PD(denoted as $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$), e.g., 1–4 dBm, where channel conditions are relatively harsh and signal impairments are dominated by a combination of additive noise and phase noise, the strength of AdaPhaseNet lies in its adaptive fusion module. This module intelligently balances the outputs of the two branches. The CNN branch, leveraging its robust local feature extraction and filtering capabilities, effectively suppresses background noise, providing a reliable baseband signal for fusion. Concurrently, the Transformer branch focuses on tracking the trend of phase variation from the noise-corrupted signal. In this region, the bit error rate (BER) and error EVM performance of AdaPhaseNet is significantly and consistently superior to all baseline models. At a BER of $3.8\times {10}^{-3}$, AdaPhaseNet achieves a maximum $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ gain of approximately 1.4 dB compared to other baselines. In contrast, standalone RNN or LSTM models struggle to maintain stable long-range modeling capability under strong noise due to their inherent sequential processing defects and gradient issues. While the standalone Transformer model possesses stronger modeling capability, its ability to filter out local impairments like impulsive noise is inferior to that of CNN. The standalone CNN model, however, completely fails to handle long-range phase noise, exhibiting a clear performance bottleneck.

Second, in the high $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ region (e.g., 5–8 dBm), where signal quality improves and the impact of additive noise diminishes, phase noise gradually becomes the dominant factor limiting system performance. The performance advantage of AdaPhaseNet further expands. At a BER of $2\times {10}^{-4}$, AdaPhaseNet achieves $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ gains of 1.3 dB, 1.17 dB, 0.85 dB, and 0.4 dB compared to four baselines (RNN, LSTM, Transformer, and CNN), respectively. When the $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ is 8 dBm, the BER of these baselines and AdaPhaseNet are $1.32\times {10}^{-4}$, $1.07\times {10}^{-4}$, $9.52\times {10}^{-5}$, $8.83\times {10}^{-5}$ and $5.56\times {10}^{-5}$, respectively, corresponding to relative BER reductions of 57.9%, 48.0%, 41.6%, and 37.0%. Furthermore, Figure 9b shows that at an EVM of approximately −33 dB, AdaPhaseNet achieves EVM gains of 3.2 dB, 2.7 dB, 1.7 dB, and 1.4 dB over the same baselines mentioned above, respectively.

The above results strongly demonstrate:

• The designed Transformer branch possesses excellent sequential modeling capability for phase noise. Its causal masking and long-window design enable it to accurately estimate and compensate for phase perturbations with memory.
• The effectiveness of the adaptive fusion module. It correctly places high confidence in the Transformer branch’s high-precision phase estimation results and “calibrates” the enhanced signal from the CNN branch, thereby outputting a signal with purer phase. In contrast, other baseline models, due to either architectural limitations or a singular optimization objective, fail to simultaneously achieve local signal purification and global phase tracking, thus reaching their performance limits in the high $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ region.

Notably, although AdaPhaseNet achieves significant BER/EVM improvements over all baseline models across the entire power range, the BER observed in our experiment still remains on the order of ${10}^{-4}$ to ${10}^{-3}$. We attribute this primarily to the hardware noise floor limitation of the experimental link. Specifically, some receiver-side instruments used in our setup (e.g., the oscilloscope and the mixer) are relatively old, and thus exhibit higher inherent noise and a higher effective noise figure. As a result, more pronounced noise and distortion are introduced during the down-conversion and sampling stages, which elevates the BER “floor”. Under such noise-limited conditions, the benefit of the proposed algorithm is reflected more in a stable gain relative to the baseline schemes (i.e., a clear BER reduction and EVM improvement under the same hardware platform and link conditions).

7. Conclusions

This paper addresses the severe phase noise challenge in photonic terahertz OFDM systems by proposing a dual-branch adaptive noise compensation network named AdaPhaseNet. Through a functionally specialized design, this architecture enables the Transformer branch to focus on capturing the long-range temporal dependencies of phase noise, while simultaneously employing the CNN branch to suppress local impairments such as additive noise. The core adaptive fusion mechanism dynamically integrates the outputs of the two branches based on confidence scores, achieving an intelligent balance between the strategies of “precise phase tracking” and “robust signal enhancement.” Although the model training process requires a certain amount of time, once trained, the network can process received signals in real-time, meeting the low-latency requirements of practical communication systems.

Experimental results validate the superiority of the proposed network. In terms of BER, AdaPhaseNet achieves optimal performance under all tested $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ conditions, demonstrating significant $P_{\mathrm{UTC}-\mathrm{PD},\mathrm{in}}$ gains, especially in the low BER region. Moreover, it consistently outperforms all baseline models on the EVM metric, proving its effectiveness in enhancing signal modulation quality.

Future work will focus on the following directions: first, investigating model lightweighting techniques to reduce computational and memory overhead, making it suitable for resource-constrained terminal devices; second, exploring joint optimization of the model for end-to-end systems; third, investigating the generalizability of this architecture to other frequency bands.