Joint Timing and Carrier Synchronization with Integrated Modulation Quality Measurement for High-Order QAM Signals

Abstract

To address limitations in the modulation-quality analysis of high-order Quadrature Amplitude Modulation (QAM) signals, including insufficient timing synchronization accuracy, challenges in carrier recovery, and coupling between synchronization errors and parameter estimation, a cascaded digital baseband processing framework tailored for measurement scenarios is proposed. The proposed framework is designed to integrate synchronization recovery and parameter measurement. In the timing synchronization stage, a feedforward open-loop structure based on the Oerder–Meyr (OM) algorithm is employed to estimate the optimal sampling instants rapidly. In the carrier synchronization stage, a two-stage recovery structure is constructed, comprising coarse frequency offset estimation based on polarity decision and fine synchronization using an improved frequency–phase detector (FPD), thereby achieving both robust acquisition of large frequency offsets and high-precision compensation of residual errors. On this basis, a unified modulation quality evaluation model is established, enabling the joint estimation of the Error Vector Magnitude (EVM) and the Modulation Error Ratio (MER), as well as amplitude, phase, and frequency errors, within a consistent analytical framework. System-level validation of 256 QAM and 1024 QAM signals is conducted using a MATLAB R2021b-based simulation platform. The results demonstrate that stable synchronization recovery can be achieved under timing, frequency, and phase perturbations, yielding well-defined constellation diagrams. In terms of parameter estimation, the relative errors of all evaluated metrics are maintained within 2%, which is significantly below the conventional 5% measurement criterion. Further analysis indicates that the proposed method maintains strong robustness across varying signal-to-noise ratios (SNRs) and sampling rates. The results confirm that the proposed cascaded processing framework effectively unifies synchronization recovery and modulation quality analysis, significantly improving parameter estimation accuracy while maintaining high synchronization precision. This approach provides a practical and efficient solution for high-order QAM signal testing and measurement systems.

1. Introduction

The sustained growth of global Internet traffic has imposed increasingly stringent requirements on the transmission capacity and spectral efficiency of optical fiber communication systems. Coherent optical communication, together with digital signal processing (DSP), has become a key solution for ultra-high-speed and long-haul optical fiber transmission because of its strong capability for link impairment compensation and high receiver sensitivity [1]. High-order QAM is a fundamental approach for improving spectral efficiency. As the modulation order is increased from 64 QAM to 256 QAM and 1024 QAM, higher spectral efficiency can be achieved, and several breakthrough demonstrations have been reported for ultra-large-capacity and long-haul optical transmission systems [2,3].

Buchali et al. achieved simultaneous improvements in transmission rate and reach by using probabilistically shaped 64 QAM without modifying the hardware architecture [2]. Koizumi et al. demonstrated, for the first time, 150 km single-carrier coherent transmission of a 60 Gbit/s 1024 QAM signal, thereby laying a foundation for the practical implementation of ultra-high-order QAM [3]. Zhou et al. realized 580 km transmission of a 32 Tb/s polarization-division-multiplexed return-to-zero 8 QAM (PDM-RZ-8QAM) signal over ultra-low-loss fiber, confirming the feasibility of high-order modulation in ultra-large-capacity systems [4]. Schmidt-Langhorst et al. demonstrated time-division demultiplexing of a single-channel 5.1 Tb/s signal, while Salsi et al. achieved 7200 km long-haul transmission of 155 × 100 Gbit/s signals, further demonstrating the application potential of high-order modulation and coherent reception technologies [5,6].

In measurement scenarios such as vector signal analyzers and coherent receiver test platforms, synchronization recovery performance determines not only demodulation accuracy but also the measurement accuracy of modulation quality parameters. Error vector magnitude (EVM) is a key comprehensive metric for evaluating the modulation quality of high-order QAM signals. Shafik et al. established a quantitative relationship among EVM, bit error rate (BER), and signal-to-noise ratio (SNR) [7], and Schmogrow et al. validated the effectiveness of EVM for system performance evaluation under high-order modulation formats [8]. Therefore, an integrated investigation of synchronization recovery and modulation quality measurement for high-order QAM is of significant theoretical and engineering value for the development, testing, and standardization of next-generation high-speed optical communication systems.

For high-precision measurement scenarios, existing synchronization and measurement techniques for high-order QAM still face three major limitations. First, in terms of timing synchronization, conventional closed-loop algorithms suffer from slow convergence and large loop delay. In contrast, mainstream feedforward algorithms are mainly optimized for communication reception and have not been sufficiently adapted to the high-precision requirements of measurement scenarios. Second, in terms of carrier synchronization, existing single-stage structures cannot simultaneously provide a wide frequency-offset acquisition range and high phase compensation accuracy for high-order modulation, and their suitability for cascaded processing remains limited. Third, at the system level, most existing studies have focused on demodulation performance optimization, while insufficient attention has been paid to the coupling between synchronization recovery and modulation quality measurement. As a result, an integrated, cascaded processing framework remains lacking.

To address these limitations, a digital baseband cascaded processing method is proposed for high-order QAM synchronization recovery and high-precision modulation quality measurement. Timing synchronization, carrier synchronization, and parameter measurement are integrated into a unified processing chain. The main contributions of this work are summarized as follows:

• A measurement-oriented cascaded DSP framework integrating timing synchronization, carrier synchronization, and modulation-quality evaluation is established for high-order QAM signal analysis.
• A unified synchronization impairment model, including timing offset, carrier frequency offset, phase offset, and additive noise, is developed to analyze the influence of synchronization recovery on modulation-quality measurement accuracy.
• A two-stage synchronization architecture combining coarse carrier recovery and fine phase tracking is implemented to improve synchronization robustness under dense high-order constellations.
• System-level offline simulations for 256 QAM and 1024 QAM signals are performed under different SNR and sampling conditions, validating the stability and consistency of the proposed modulation-quality evaluation framework.

Unlike conventional studies focusing primarily on communication performance optimization, this work emphasizes synchronization-assisted modulation-quality evaluation for offline DSP-based measurement scenarios. Unlike conventional coherent communication receivers primarily optimized for BER performance, the present work focuses on synchronization-assisted modulation-quality evaluation in offline DSP-based measurement scenarios. The objective of the proposed framework is not full optical-link impairment compensation, but stable synchronization recovery and accurate extraction of modulation-quality parameters under synchronization impairments. The algorithmic novelty of this work should be understood at the system-integration and module-refinement levels, rather than as the invention of entirely new standalone synchronization primitives.

2. Related Work

2.1. Digital Signal Processing Techniques for High-Order QAM Coherent Optical Reception

The development of coherent optical receiver DSP techniques has provided essential support for the practical implementation of high-order QAM. In 2004, Taylor first proposed a DSP-based method for coherent detection and link impairment equalization, which laid the theoretical foundation for coherent receiver DSP architectures [9]. Ip and Kahn established an all-digital equalization theory for chromatic dispersion and polarization-mode dispersion (PMD), and derived the algorithmic limits of linear impairment equalization [10]. Savory et al. experimentally validated chromatic dispersion compensation in digital coherent receivers and further optimized digital filter design for coherent reception. A full-link DSP framework involving chromatic dispersion compensation, synchronization recovery, and channel equalization was thereby established [11,12].

At the system implementation level, Sun et al. realized real-time measurement and hardware implementation of a 40 Gb/s coherent system, thereby verifying the practical feasibility of DSP algorithms [13]. Crivelli et al. analyzed adaptive equalization algorithms in the presence of chromatic dispersion, PMD, and phase noise, and demonstrated that system performance improved under multiple simultaneous impairments [14]. Pfau et al. proposed a hardware-efficient coherent receiver architecture for M-ary quadrature amplitude modulation (M-QAM) constellations. By using a fully feedforward carrier recovery scheme, the hardware implementation complexity of synchronization algorithms was substantially reduced [15]. The classical work by Mengali and D’Andrea systematically established a comprehensive theoretical framework for synchronization techniques in digital receivers, providing authoritative theoretical support for the design and optimization of synchronization algorithms [1].

2.2. Advances in Timing Synchronization for High-Order QAM

Timing synchronization is the first synchronization stage in the coherent receiver DSP chain. Its primary purpose is to recover the optimal symbol sampling instant and eliminate timing errors caused by transmitter–receiver clock mismatch, thereby providing a prerequisite for reliable subsequent demodulation [1]. Existing timing synchronization techniques can generally be classified into two categories: closed-loop feedback schemes and feedforward open-loop schemes.

Among closed-loop feedback schemes, Gardner’s 1986 timing error detector is a representative method [16]. Without requiring a training sequence, timing error extraction can be achieved with only twofold oversampling, and the method has therefore been widely adopted in low- and medium-order QAM systems. For high-order QAM scenarios, Gao et al. proposed a low-complexity symbol-rate joint equalization and timing recovery scheme, in which timing recovery and adaptive equalization were jointly designed to reduce algorithmic complexity [17]. As early as 1992, Jablon proposed a closed-loop architecture that jointly incorporated blind equalization, carrier recovery, and timing recovery, enabling joint optimization of synchronization and equalization [18]. However, the inherent limitations of closed-loop algorithms, including slow convergence, large loop delay, and high parameter sensitivity, become more pronounced in high-order QAM scenarios, making them unsuitable for real-time measurement applications.

Among feedforward open-loop schemes, the OM digital square-filter timing recovery algorithm proposed in 1988 is a classical representative. In this method, modulation information is removed via squared-magnitude processing, and the symbol-rate spectral line is extracted for timing-error estimation. Because no feedback loop is required, fast convergence can be achieved, and inherent robustness to carrier frequency offset and phase offset is provided [19]. For ultra-high-speed transmission scenarios, Schmidt et al. proposed a fully digital parallel architecture for the OM algorithm, thereby addressing the symbol-rate processing bottleneck in high-speed systems [20]. However, existing OM-based algorithms have generally been optimized to minimize demodulation BER in communication receivers rather than to minimize the variance of timing-error estimation in measurement scenarios. Therefore, their estimation accuracy remains limited under low-SNR and high-order constellation conditions.

2.3. Advances in Carrier Synchronization and Phase Recovery for High-Order QAM

Carrier synchronization is used to compensate for local oscillator frequency offset and laser phase noise, thereby preventing rotation and diffusion of the received constellation. It is therefore a key stage for stable demodulation of high-order QAM signals. Because high-order QAM constellations are densely packed and highly sensitive to frequency offsets and phase errors, carrier synchronization algorithms are required to provide both a wide frequency-offset acquisition range and high phase compensation accuracy.

Among classical phase recovery algorithms, the nonlinear phase-shift keying (PSK) carrier phase estimation algorithm proposed by Viterbi and Viterbi in 1983 laid a theoretical foundation for carrier synchronization techniques [21]. The phase-frequency detector (PFD) structure proposed by Sari and Moridi in 1988 enables joint tracking of frequency offset and phase error, and has become a classical architecture for fine carrier synchronization [22]. For high-order QAM scenarios, Fatadin et al. proposed a quadrature phase-shift keying (QPSK) partitioning-based carrier recovery scheme, thereby improving the laser linewidth tolerance of 16 QAM systems [23]. Lu et al. proposed a constellation-rotation-based frequency-offset estimation algorithm for 32 QAM, reducing the implementation complexity of frequency-offset estimation [24]. Gagnon et al. proposed a fast carrier recovery algorithm for high-order QAM, which substantially improved convergence speed [25]. Câmpeanu et al. proposed an adaptive decision-directed extended Kalman filter (EKF)-based carrier synchronization algorithm for high-order QAM, thereby improving tracking performance under low-SNR conditions [26]. Tselniker et al. proposed a multiplier-free joint carrier recovery structure for 16 QAM that significantly reduced hardware implementation overhead [27].

More recently, carrier synchronization and carrier phase recovery (CPR) for high-order QAM have further evolved toward low-complexity two-stage estimation, improved CPR for ultra-high-order modulation formats, pilot-assisted phase recovery, and probabilistic-shaping-aware recovery. Peng et al. proposed a simple two-stage carrier phase estimation algorithm for 32 QAM coherent optical communication systems, in which coarse phase estimation and fine maximum-likelihood detection were combined to improve phase-estimation stability while reducing computational complexity [28]. Zhang et al. investigated an improved CPR method for high-capacity optical communication systems with high-order modulation formats, including DP-64 QAM and DP-256 QAM, highlighting the continued importance of phase recovery for dense QAM constellations [29]. In addition, Betancourt et al. proposed a pilot-assisted phase recovery method with robust locally weighted interpolation for coherent optical receivers, providing a low-complexity and reliable phase estimation solution for higher-order modulation scenarios [30]. The influence of probabilistic shaping on CPR has also been investigated. Zhang et al. experimentally compared CPR algorithms for uniform and probabilistically shaped QAM in a 324.1 Gb/s fiber-mm-wave integration system at W-band, and analyzed the performance of algorithms such as BPS and two-stage BPS under different symbol-distribution conditions [31]. These recent studies indicate that high-order QAM synchronization research increasingly emphasizes the trade-off among phase-estimation accuracy, computational complexity, modulation-format adaptability, and compatibility with probabilistically shaped constellations.

In summary, three major limitations remain in existing techniques. First, timing synchronization algorithms have not been sufficiently optimized for measurement scenarios, and their estimation accuracy remains inadequate under high-order and low-SNR conditions. Second, single-stage carrier synchronization structures cannot simultaneously achieve wide frequency-offset acquisition and high-precision phase compensation, and their adaptability to cascaded processing is limited. Third, an integrated framework for synchronization recovery and modulation quality measurement remains lacking, and the coupling between these processes has not been adequately addressed. The present study is therefore conducted to address these limitations and provide a complete solution for high-precision modulation quality measurement of high-order QAM signals.

3. Methodology

3.1. Equivalent Baseband and Impairment Models for QAM Signals

To address digital demodulation and modulation-quality analysis of high-order QAM signals in vector signal analyzers, the complete modulation–transmission–reception process is modeled using a unified complex baseband representation. This formulation provides a consistent theoretical foundation for the subsequent design of timing, carrier, and parameter measurement synchronization algorithms. In test and measurement systems, radio-frequency (RF) front-end downconversion and analog-to-digital conversion are primarily implemented in hardware. In contrast, in digital baseband processing, carrier frequency offset, phase offset, and timing errors—arising from local oscillator (LO) frequency mismatch, initial phase misalignment, channel impairments, and clock deviations—dominate both demodulation performance and measurement accuracy in high-order QAM systems. Therefore, establishing a unified baseband transmission model and a corresponding synchronization impairment model is essential for subsequent algorithmic analysis and system-level optimization.

As shown in Figure 1, the input bit stream is first converted via serial-to-parallel conversion and then mapped to a QAM constellation, yielding a complex-valued symbol sequence. Let the transmitted symbol sequence be denoted as $\left\{a_n\right\}$, where $a_n$ represents the $n$-th symbol, $T_s$ denotes the symbol period, and $g_T\left(t\right)$ denotes the impulse response of the pulse shaping filter. The transmitted equivalent complex baseband signal can then be expressed as a continuous-time superposition of time-shifted shaping pulses weighted by the symbol sequence:

$$s\left(t\right)=\sum _{n=-\infty }^{\infty }a\left(n\right)g_T\left(t-nT\right)$$

(1)

This formulation indicates that the time-domain structure of the ideal transmitted signal is jointly determined by the constellation mapping and the transmit pulse shaping filter. After propagation through the channel, the signal is impaired by carrier frequency offset, phase offset, and additive white Gaussian noise (AWGN). Accordingly, the received complex baseband signal before matched filtering can be expressed as the superposition of the transmitted signal, a complex exponential distortion term, and additive noise:

$$r\left(t\right)=\sum _{n=-\infty }^{\infty }a\left(n\right)g_T\left(t-nT\right)e^{j\left(2\pi \Delta fT+\phi \right)}+n\left(t\right)$$

(2)

Here, $\Delta f$ primarily arises from frequency mismatches between the local oscillators at the transmitter and receiver, as well as non-ideal effects such as system drift and temperature variations. The parameter ${\varphi }_0$ represents the initial phase offset and time-varying phase disturbances introduced during signal propagation. Compared with low-order modulation schemes, high-order QAM constellations exhibit significantly reduced symbol spacing, making them more sensitive to carrier frequency offsets and phase distortions. The received signal is processed using a matched filter, and the resulting signal can be expressed as:

$$z\left(t\right)=r\left(t\right)\otimes g_R\left(t\right)=\left[\sum _{n=-\infty }^{\infty }a\left(n\right)g\left(t-nT\right)\right]e^{j\left(2\pi \Delta fT+\phi \right)}+n\left(t\right)$$

(3)

In this expression, $g_R\left(t\right)$ denotes the impulse response of the matched filter, and $\epsilon =\tau T$ represents the timing offset induced by channel delay and asynchronous local clock oscillations between the transmitter and receiver. The convolution of the transmit and receive filters yields the overall system response, which effectively mitigates inter-symbol interference (ISI) caused by sampling mismatches. By sampling the continuous-time signal $z\left(t\right)$ at instants $t=k{T}_s-\tau T_s$, where the sampling rate is configured as an integer multiple of the symbol rate, the corresponding discrete-time baseband signal is obtained as:

$$z\left(k\right)=\left[\sum _{n=-\infty }^{\infty }a\left(n\right)g\left(K{T}_s-nT-\tau T\right)\right]e^{j\left(2\pi \Delta fK{T}_s+{\phi }_0\right)}+n\left(k\right)$$

(4)

Through sampling adaptation, the sampling rate is adjusted to an integer multiple of the symbol rate, yielding the discrete sequence $z\left[k\right]$. This sequence is directly utilized for timing synchronization, carrier synchronization, and symbol decision, and serves as the primary signal representation for the proposed synchronization algorithms.

As shown in Figure 2, synchronization impairments primarily manifest as timing, carrier, and phase offsets. To quantify the impact of these impairments on demodulation and measurement performance, a unified synchronization impairment model is established based on an ideal QAM signal:

$$z\left(t\right)=\left(z_I\left(t-{\tau }_I\right)+j\cdot z_Q\left(t-{\tau }_I\right)\right)e^{j\left(2\pi f_0+\phi \left(t\right)\right)}+n\left(t\right)$$

(5)

In this model, $z_I\left(t\right)$ and $z_Q\left(t\right)$ denote the in-phase and quadrature components of the reference signal, respectively; $f_0$ represents the carrier frequency offset; $\varphi$ denotes the carrier phase offset; ${\tau }_I$ and ${\tau }_Q$ represent timing offsets in the in-phase and quadrature branches; and $n\left(t\right)$ denotes AWGN. In this formulation, channel impairments, local oscillator instability, and clock deviations are modeled as a unified set of mathematical perturbations. Non-synchronization impairments, such as IQ gain imbalance, quadrature mismatch, and amplitude fading, are neglected, thereby enabling focused design, analysis, and validation of timing and carrier synchronization algorithms.

It should be noted that the present study focuses primarily on synchronization-related impairments and modulation-quality evaluation within an equivalent complex baseband framework. Transmission impairments commonly encountered in practical coherent optical communication systems, including chromatic dispersion (CD), polarization mode dispersion (PMD), polarization rotation, and adaptive MIMO equalization, are not considered in the current work. This simplification is intentionally adopted to isolate the influence of synchronization recovery on modulation-quality analysis.

The current work assumes a single-polarization equivalent baseband model and therefore does not include polarization rotation or polarization-diverse coherent reception. Consequently, adaptive polarization demultiplexing, chromatic dispersion compensation, and MIMO equalization are outside the scope of this study. Future work will extend the proposed framework toward dual-polarization coherent optical systems with adaptive equalization and polarization tracking.

In addition, it should be noted that the present work focuses primarily on synchronization-related impairments and modulation-quality evaluation within an equivalent complex baseband framework. Unlike full coherent optical receiver architectures, the current study does not include optical front-end components such as optical modulation, local oscillator phase noise modeling, fiber transmission links, polarization-diverse coherent reception, or optical hardware impairments. Transmission impairments commonly encountered in practical coherent optical communication systems, including chromatic dispersion (CD), polarization-mode dispersion (PMD), polarization rotation, and adaptive MIMO equalization, are not considered in the present work. The proposed framework assumes a single-polarization equivalent baseband model in order to isolate the influence of synchronization recovery on modulation-quality analysis. Consequently, adaptive polarization tracking and polarization demultiplexing are outside the scope of this study. Future work will extend the proposed framework toward dual-polarization coherent optical receiver architectures with adaptive equalization and polarization recovery.

3.2. OM-Based Timing Error Estimation Using Segmented FFT

As the modulation order of QAM increases, the minimum Euclidean distance between constellation points is significantly reduced, resulting in increased sensitivity to timing offsets. Consequently, conventional closed-loop timing synchronization methods are limited by slow convergence, significant loop delay, and insufficient real-time performance to meet the stringent requirements of instrumentation-based measurement systems.

To address these challenges, a fully digital non-data-aided (Non-Data-Aided, NDA) feedforward timing synchronization architecture is adopted. This architecture eliminates the need for training sequences, achieves high spectral efficiency, and avoids cumulative errors and latency associated with feedback loops. As a result, it is particularly suitable for rapid synchronization in high-order QAM systems, including 256 QAM, 1024 QAM, and beyond.

The OM algorithm, also referred to as the digital squaring timing synchronization method, is a representative open-loop symbol timing estimation technique. Its key advantage lies in its inherent robustness to carrier frequency offsets and phase distortions. By relying solely on signal amplitude information, the symbol-rate spectral component can be directly extracted, enabling accurate timing-error estimation. This characteristic makes the OM algorithm highly suitable for high-speed demodulation systems in which timing synchronization is cascaded with carrier synchronization.

As shown in Figure 3, the core idea is that the received signal $X\left(t\right)$ is first digitized using an analog-to-digital converter (ADC) with a sampling clock of $1/T$, followed by matched filtering to obtain a baseband signal containing timing offsets. The OM timing synchronization module subsequently processes the resulting signal. The received signal is oversampled at $S$ times the symbol rate, and the magnitude-squared operation is applied to the resulting sequence. The processed data are then segmented into blocks of $L$ symbols, with each segment containing $LS$ samples. Each segment of $LS$ samples is utilized to perform timing error estimation. A discrete Fourier transform (DFT) is applied to each segment, and the phase information at the symbol-rate frequency bin is extracted as the timing error estimate. Finally, spline interpolation is used to determine the optimal sampling instant accurately. The received signal $r\left(t\right)$ is resampled at a sampling rate. $F_s$, yielding an $N$-times oversampled discrete sequence $r\left[k\right]$. The magnitude-squared operation is subsequently applied, resulting in the sequence:

$$r_k =\sum _{n=-\infty }^{\infty }{a}_{n}g\left(k{\displaystyle \frac{T_s}{N}}-n{T}_s-\epsilon \left(t\right)T_s\right)e^{j\Delta \theta \left(k{\displaystyle \frac{T_s}{N}}\right)}+n\left(k{\displaystyle \frac{T_s}{N}}\right)=r\left(k{\displaystyle \frac{T_s}{N}}\right)$$

(6)

$$x_k={\left|\sum _{n=-\infty }^{\infty }{a}_{n}g\left(k{\displaystyle \frac{T_s}{N}}-n{T}_s-\epsilon \left(t\right)T_s\right)\right|}^2={\left|r_k\right|}^2$$

(7)

$$g\left(t\right)=g_T\left(t\right)\times g_R\left(t\right)$$

(8)

By transforming the signal from the time domain to the frequency domain, the magnitude-squared operation can be interpreted as a self-convolution of the signal spectrum. As a result, a distinct, discrete spectral line emerges at the normalized frequency ${\displaystyle \frac{2\pi }{T_s}}$, corresponding to the symbol rate. To extract the discrete spectral line, an $NL$-point discrete Fourier transform (DFT) is applied to the signal sequence, yielding the frequency-domain representation $X\left[k\right]$:

$$X\left(k\right)=\sum _{n=0}^{LN-1}x\left(n\right)e^{-j{\displaystyle \frac{2\pi }{LN}}nk},k=0,1,2,\dots .,\mathit{LN}-1$$

(9)

$$X\left(L\right)=\sum _{n=0}^{LN-1}x\left(n\right)e^{-j{\displaystyle \frac{2\pi \ast n}{N}}}$$

(10)

The spectral component corresponding to the symbol rate $f={\displaystyle \frac{1}{T_s}}$ is located at index $k=L$, and the corresponding component $X\left[L\right]$ represents the discrete spectral line containing timing error information. The entire signal sequence is further segmented into $N$ sub-blocks, each containing $L$ samples. A DFT is applied to each segment, and the spectral component at the symbol-rate frequency bin is extracted as $X_i\left[L\right]$. After normalization, the phase of this component is used to estimate the timing error for each segment:

$$X_m=\sum _{k=mLN}^{\left(m+1\right)LN-1}{x}_k{e}^{-j2\pi k/N}$$

(11)

$$\epsilon =-{\displaystyle \frac{1}{2\pi }}\mathit{arg}\left(\sum _{k=mLN}^{\left(m+1\right)LN-1}{x}_k{e}^{-j2\pi k/N}\right)$$

(12)

3.3. Two-Stage Carrier Synchronization Structure

The novelty of the proposed carrier synchronization method lies in the reliability-constrained coarse estimation and coarse-initialized residual tracking strategy, rather than in the independent invention of OM timing recovery or PLL tracking. The proposed carrier synchronization framework operates in a non-data-aided (NDA) mode without requiring pilot symbols or training sequences. Coarse carrier frequency recovery is achieved through blind estimation, while the fine carrier synchronization stage employs decision-directed phase tracking to compensate residual carrier phase fluctuations. Carrier synchronization constitutes a critical module in high-order QAM coherent demodulation systems, ensuring both reliable demodulation performance and accurate vector measurement. Frequency mismatches between transmitter and receiver local oscillators, Doppler effects, and phase disturbances introduce carrier frequency offsets and phase rotations, resulting in constellation rotation and spreading. These impairments directly lead to symbol decision errors and degradation of key measurement metrics, including EVM and Modulation Error Ratio (MER). Under ideal timing and equalization conditions, carrier impairments can be modeled as a multiplicative complex exponential distortion. Compared with low-order modulation schemes, high-order QAM constellations exhibit increased sensitivity to carrier impairments. Therefore, a wide acquisition range and high-precision two-stage carrier synchronization architecture are required to satisfy system performance requirements [30].

As shown in Figure 4, a two-stage synchronization structure is adopted to compensate for carrier frequency and phase impairments. In the first stage, coarse frequency offset estimation is performed to remove the majority of the carrier frequency offset. In the second stage, a phase-locked loop (PLL) is employed to track residual frequency offsets and phase errors with high precision. The overall architecture of the carrier synchronization algorithm, along with the detailed structure of the fine estimation PLL, is illustrated in the corresponding figures.

3.3.1. Coarse Carrier Frequency Offset Estimation Based on Polarity Decision

To achieve high-precision carrier synchronization for high-order QAM signals, a polarity decision-based open-loop algorithm, referred to as Carrier Polarity-Based Phase Difference Frequency Estimation (CP-PDFE), is employed for coarse carrier frequency offset estimation. In this method, corner constellation symbols that are less susceptible to noise are selectively used to estimate the frequency offset rapidly. As a result, low computational complexity and strong robustness to noise and interference are achieved.

Let the received baseband signal after matched filtering and timing synchronization be denoted as $r\left(t\right)$, which contains residual carrier frequency offset $\Delta f$ and phase offset $\Delta \varphi$. The signal can be expressed as:

$$r\left(t\right)=s\left(t\right)e^{j\left(2\pi \Delta ft+\Delta \varphi \right)}+n\left(t\right)$$

(13)

Here, $s\left(t\right)$ denotes the ideal baseband signal, and $n\left(t\right)$ represents AWGN. In the CP-PDFE algorithm, corner constellation symbols are first extracted using a power detection mechanism. Polarity decision is then applied to estimate the phase deviation. ${\theta }_i$ associated with each selected symbol. The carrier frequency offset is subsequently estimated based on the phase difference between two selected symbols separated by a time interval $\Delta t_{i,j}$:

$$\Delta f_{i,j}={\displaystyle \frac{{\theta }_i-{\theta }_j}{2\pi \Delta t}}$$

(14)

By averaging over all $N$ valid symbol pairs, the coarse carrier frequency offset estimate is obtained as:

$${\widehat{\Delta f}}_{\mathrm{c}\mathrm{o}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{e}}={\displaystyle \frac{1}{N}}\sum _{i,j}\Delta f_{i,j}$$

(15)

The coarse estimate is used to initialize the numerically controlled oscillator (NCO), enabling compensation for most of the carrier frequency offset. This provides a stable and reliable initial condition for the subsequent fine synchronization stage, thereby improving convergence speed and estimation accuracy.

As the modulation order increases, the proportion of corner constellation symbols decreases significantly. Consequently, the effective utilization of valid symbols in the conventional CP-PDFE algorithm is reduced, leading to degraded frequency offset estimation accuracy and reduced robustness to noise and interference. To address this limitation, two improvement strategies are proposed to enhance the performance of coarse frequency offset estimation under high-order QAM conditions:

• Hard-Decision Error Mitigation

In high-order QAM systems or under long-duration transmission conditions, corner constellation symbols may deviate from their nominal quadrants, leading to phase errors when hard-decision rules are applied. To mitigate such misclassification effects, a joint constraint based on quadrant consistency and temporal separation is introduced. Only symbols satisfying the following conditions are retained:

$$\left\{\begin{array}{c}\mathrm{Symbols}\mathrm{lie}\mathrm{within}\mathrm{the}\mathrm{same}\mathrm{quadrant}.\\ \Delta t_i<T_{\mathrm{t}\mathrm{h}}\end{array}\right.$$

(16)

Symbols that do not satisfy these conditions are treated as outliers and discarded, thereby reducing the impact of hard-decision-induced phase errors on frequency offset estimation.

2.

Histogram-Based Statistical Refinement

To address discrete fluctuations in the estimated frequency offset sequence, a histogram-based statistical refinement method is employed:

• The mean $\mu$ and variance ${\sigma }^2$ of the estimated frequency offset sequence $\left\{\Delta f_{i,j}\right\}$ are computed, and outliers exceeding $3\sigma$ from the mean are removed;
• The remaining valid frequency offset samples are partitioned into $K$ equal-width intervals, and the sample density within each interval is evaluated;
• The three intervals with the highest densities are selected, and a weighted average is computed based on their densities to obtain the final coarse carrier frequency offset estimate:

$${\widehat{\Delta f}}_{\mathrm{f}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}={\displaystyle \frac{{\displaystyle {\sum }_{k=1}^3}{w}_k{\stackrel{-}{\Delta f}}_k}{{\displaystyle {\sum }_{k=1}^3}{w}_k}}$$

(17)

where ${\stackrel{-}{\Delta f}}_k$ denotes the mean frequency offset within the $k$-th interval, and $w_k$ represents the corresponding weighting coefficient proportional to the interval density.

Compared with a conventional polarity-decision coarse frequency estimator, the present coarse synchronization stage introduces a reliability-constrained estimation strategy for high-order QAM signals. In dense constellations, especially for 256 QAM and 1024 QAM, the Euclidean distance between adjacent symbols is small, and the proportion of highly reliable corner symbols is relatively limited. Directly applying polarity-decision-based phase-difference estimation may therefore introduce biased frequency-offset estimates when noisy inner symbols or ambiguous boundary symbols are involved.

To improve the robustness of coarse carrier-frequency estimation, two reliability constraints are introduced before frequency-offset averaging. First, candidate symbols are selected according to quadrant consistency and amplitude reliability, so that only symbols with stable polarity decisions and sufficiently large in-phase and quadrature components are used for coarse estimation. Second, the candidate frequency-offset estimates are statistically refined by rejecting isolated outliers and retaining the dominant estimation cluster. This operation suppresses unreliable phase-difference samples caused by noise, constellation ambiguity, and occasional polarity-decision errors.

3.3.2. Carrier Fine Synchronization Based on an Improved Phase and Frequency Detector (PDF)

To achieve high-precision carrier fine synchronization for high-order QAM signals, a Phase and Frequency Detector (PDF)-based decision algorithm jointly estimates residual carrier frequency offsets and phase errors. This method is implemented within a phase-locked loop (PLL) framework. By incorporating a track-and-hold mechanism into polarity decision-based detection, continuous tracking of small residual frequency and phase errors is achieved, resulting in a wide synchronization range and improved loop stability, particularly for high-order modulation signals.

As shown in Figure 5, let the received complex baseband signal after matched filtering, timing synchronization, and coarse frequency compensation be denoted as $r\left[k\right]$, which still contains residual carrier frequency offset $\Delta f_r$ and phase offset $\Delta {\varphi }_r$. The signal can be expressed as:

$$r\left(k\right)=s\left(k\right)e^{j\left(2\pi \Delta f_{r}kT+\Delta {\varphi }_r\right)}+n\left(k\right)$$

(18)

where $s\left[k\right]$ denotes the ideal baseband symbol sequence, $T_s$ is the symbol period, and $n\left[k\right]$ represents complex AWGN. In the PDF-based detection process, the input signal is first multiplied by the compensation signal generated by the numerically controlled oscillator (NCO), resulting in a partially phase-corrected intermediate signal. A power detection mechanism is then applied to select valid symbols based on a predefined threshold, and the corresponding phase error is estimated using a polarity decision rule. When the input symbol satisfies the threshold condition, the phase error is updated according to the detection rule. Otherwise, the phase detector output is maintained from the previous time instant. The track-and-hold mechanism can be formulated as:

$$e(k)=\left\{\begin{array}{cc}{e}_{\mathrm{n}\mathrm{e}\mathrm{w}}\left(k\right),& \mid e_{\mathrm{n}\mathrm{e}\mathrm{w}}\left(k\right)-e(k-1)\mid <\delta \\ e(k-1),& \mid e_{\mathrm{n}\mathrm{e}\mathrm{w}}\left(k\right)-e(k-1)\mid \ge \delta \end{array}\right.$$

(19)

where $e_{\mathrm{new}}\left[k\right]$ denotes the phase error obtained at the current time instant, $e[k-1]$ is the previous output of the phase detector, and $\delta$ is the track-and-hold threshold. The resulting error signal is processed through a loop filter and smoothing stage before being fed back to the NCO, which generates an updated local compensation signal. In this way, closed-loop fine synchronization of residual frequency and phase offsets is achieved.

However, as the modulation order increases, the limitation of conventional PDF-based methods—relying solely on outermost corner constellation symbols—becomes increasingly pronounced. Due to the significantly reduced probability of corner symbols in high-order QAM constellations, the number of valid symbols participating in loop updates decreases, resulting in slower convergence and increased PLL acquisition time, thereby degrading synchronization efficiency. To address this limitation, an improved PDF-based detection strategy is proposed.

• Expansion of the Effective Detection Region

In conventional PDF-based detection, only a limited number of outermost corner constellation symbols are retained, resulting in an insufficient number of valid symbols for phase detection under high-order QAM conditions. To improve symbol utilization, the detection region of the power detector is adaptively expanded, allowing more high-energy outer-layer symbols to participate in the polarity decision process without significantly degrading synchronization accuracy. The output of the improved phase detector can be expressed as:

$$e\left(k\right)=f \left(e(k-1),\Delta \theta \left(k\right),s_k,\Gamma \right)$$

(20)

where $e\left[k\right]$ and $e[k-1]$ denote the phase detector outputs at the current and previous time instants, respectively; $\Delta \theta \left[k\right]$ represents the phase deviation obtained via polarity decision; $s_k$ denotes the constellation point, and Γ defines the effective detection region. By expanding the detection region, the number of valid symbols involved in phase error estimation is significantly increased, thereby improving the update rate of the phase-locked loop (PLL).

2.

Polar Sector-Based Symbol Selection Optimization

Although expanding the detection region improves symbol utilization, rectangular windows in the Cartesian domain may introduce adjacent non-target symbols, leading to increased phase errors and degraded loop acquisition performance. To address this issue, the detection region is redesigned as a sector-shaped window in the polar coordinate domain, which better matches the distribution characteristics of outer-layer constellation points in high-order QAM signals. The refined phase detector output can be expressed as:

$$e\left(k\right)=g\left(e(k-1),\Delta \theta \left(k\right),\rho ,\phi \right)$$

(21)

where $\rho$ and $\varphi$ denote the radial and angular bounds of the detection region in the polar coordinate system. Compared with conventional rectangular windows, the sector-shaped detection window increases the number of valid symbols while suppressing the inclusion of irrelevant symbols, thereby achieving a balance between loop convergence speed and phase detection accuracy.

After coarse carrier-frequency compensation, residual frequency offset and phase jitter still remain, particularly for high-order QAM signals whose constellation points are highly sensitive to small phase deviations. A conventional single-stage decision-directed PLL is required to simultaneously perform large-offset acquisition and residual phase tracking, which may lead to slow convergence or decision error propagation when the initial carrier-frequency offset is large. To avoid this problem, the present method separates wide-range acquisition from high-precision tracking.

Specifically, the coarse frequency estimate obtained in Section 3.3.1 is first used to initialize the numerically controlled oscillator and remove the dominant carrier-frequency offset. The fine synchronization stage then tracks only the residual frequency and phase errors through an improved phase-frequency detector and PLL structure. In this way, the PLL does not need to search over a large frequency range, but instead operates around a reduced residual-error region. This design improves convergence stability and reduces the probability that incorrect symbol decisions are fed back into the loop.

The algorithmic difference from a conventional PLL-based carrier recovery method is therefore twofold. First, the loop is initialized by the reliability-refined coarse frequency estimate rather than by a zero-frequency or blindly estimated initial state. Second, the fine loop is designed as a residual-error tracking module rather than a complete carrier-acquisition module. This division of labor is particularly important for dense QAM constellations because it reduces phase-rotation accumulation and improves the accuracy of subsequent EVM, MER, phase-error, and frequency-error measurements.

With the above enhancements, the proposed method significantly improves carrier fine synchronization performance for high-order QAM signals while maintaining the PLL’s stability. The proposed approach enables joint compensation of residual frequency and phase errors. By expanding the effective symbol-detection region and optimizing its geometric structure, both symbol utilization and loop convergence speed are significantly improved, providing a reliable fine-synchronization output for subsequent high-precision coherent demodulation.

3.4. Modulation Quality Parameter Evaluation

After timing and carrier synchronization are complete, the deviation between the recovered signal and the ideal reference signal can be used to evaluate modulation quality. Let the synchronized received symbol, the ideal reference symbol, and the corresponding error vector at the $k$-th time instant be denoted as:

$${\widehat{s}}_k={\widehat{I}}_k+j{\widehat{Q}}_k,k=\mathrm{1,2},\dots ,N , s_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}=I_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}+j{Q}_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}$$

(22)

$$e_k={\widehat{s}}_k-s_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}$$

(23)

Based on these definitions, the EVM and MER are defined as:

$${EVM}_{RMS}\left(\%\right)=\sqrt{{\displaystyle \frac{{\displaystyle {\sum }_{k=1}^N}{\mid s_k-s_k^{\left(ref\right)}\mid }^2}{{\displaystyle {\sum }_{k=1}^N}{\mid s_k^{\left(ref\right)}\mid }^2}}}\times 100\%$$

(24)

$$MER\left(dB\right)={10\mathrm{l}\mathrm{o}\mathrm{g}}_{10}\left({\displaystyle \frac{{\displaystyle {\sum }_{k=1}^N}{\mid s_k^{\left(ref\right)}\mid }^2}{{\displaystyle {\sum }_{k=1}^N}{\mid s_k-s_k^{\left(ref\right)}\mid }^2}}\right)$$

(25)

where $s_k$ and $s_k^{\left(ref\right)}$ denote the ideal and recovered symbols. Furthermore, let $A_k$ and ${\theta }_k$ denote the amplitude and phase of the recovered symbols relative to the reference symbols. The root mean square (RMS) amplitude error and phase error are defined as:

$$A_k=\mid {\widehat{s}}_k\mid ,A_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}=\mid s_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}\mid , {\theta }_k=\arg \left({\widehat{s}}_k\right),{\theta }_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}=\arg \left(s_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}\right)$$

(26)

$$A_{\mathrm{e}\mathrm{r}\mathrm{r},\mathrm{R}\mathrm{M}\mathrm{S}}=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{k=1}^N}{\left(A_k−A_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}\right)}^2} , {\theta }_{\mathrm{e}\mathrm{r}\mathrm{r},\mathrm{R}\mathrm{M}\mathrm{S}}=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum _{k=1}^N}{\left({\theta }_k−{\theta }_k^{\left(\mathrm{r}\mathrm{e}\mathrm{f}\right)}\right)}^2}$$

(27)

For frequency error, let ${\widehat{f}}_{\mathrm{r}\mathrm{e}\mathrm{s}}$ denote the residual frequency offset after carrier synchronization, and $f_{\mathrm{r}\mathrm{e}\mathrm{f}}$ the reference value. The RMS frequency error is defined as:

$$f_{\mathrm{e}\mathrm{r}\mathrm{r},\mathrm{R}\mathrm{M}\mathrm{S}}=\sqrt{{\displaystyle \frac{1}{M}}{\displaystyle \sum _{m=1}^M}{\left({\widehat{f}}_{\mathrm{r}\mathrm{e}\mathrm{s},m}−f_{\mathrm{r}\mathrm{e}\mathrm{f},m}\right)}^2}$$

(28)

These parameters are all derived from statistical deviations between the recovered and ideal reference signals. They characterize modulation quality from multiple perspectives, including overall distortion, amplitude deviation, phase deviation, and residual frequency offset, thereby providing a unified evaluation framework for the experimental analysis presented in Section 4.

4. Experimental Results and Analysis

4.1. Simulation Configuration and Signal Impairment Model

To validate the proposed synchronization and modulation-quality evaluation framework, a MATLAB-R2021b-based offline DSP simulation platform was established. At the transmitter, pseudo-random binary sequences (PRBS) were generated and mapped into 256 QAM and 1024 QAM constellations. Root-raised-cosine (RRC) pulse shaping with a roll-off factor of 0.4 was employed to suppress inter-symbol interference. To emulate synchronization impairments encountered in practical coherent measurement systems, timing offset, carrier frequency offset, and carrier phase offset were digitally introduced according to the following equation:

$$r\left(t\right)=s(t-\tau )ej(2\pi \Delta ft+\varphi )+n\left(t\right)$$

where $\tau$ denotes fractional timing offset, $\Delta f$ represents carrier frequency offset, $\varphi$ denotes carrier phase offset and $n\left(t\right)$ denotes additive white Gaussian noise (AWGN).

The timing offset was generated using fractional-delay interpolation with normalized offsets ranging from 0.05 UI to 0.25 UI. Carrier frequency offsets ranging from 500 Hz to 2 kHz were digitally inserted through complex exponential rotation in the baseband domain. Carrier phase offsets were initialized randomly within $[0,2\pi ]$ to emulate asynchronous acquisition conditions. The signal-to-noise ratio (SNR) was varied from 10 dB to 30 dB to evaluate synchronization robustness under different noise conditions. At the receiver, the DSP chain consisted of matched filtering, timing synchronization, coarse carrier synchronization, fine carrier synchronization, symbol decision, and modulation-quality parameter extraction. All simulations were performed offline in MATLAB without optical transmission hardware or polarization-diverse coherent receiver front-end components.

Table 1. Simulation Parameters.

Parameter	Value
Modulation format	256 QAM/1024 QAM
Pulse shaping	RRC
Roll-off factor	0.4
Oversampling factor	8
Timing offset	0.05–0.25 UI
Frequency offset	500 Hz–2000 Hz
Phase offset	Random
SNR	10–30 dB
Channel model	AWGN

presents the key parameter information for the MATLAB model.

To further improve reproducibility, additional reproducibility-oriented implementation parameters are summarized in

Table 2. Additional Reproducibility Parameters.

Parameter	Value
Symbol rate	25 MBaud
Number of symbols per run	131,072
Number of independent runs	10
Random seed setting	Fixed random seeds
Sampling rate	200 MHz
Filter span	16 symbols

, including the symbol rate, number of symbols per run, number of independent runs, random seed setting, sampling rate, and filter span.

4.2. Experimental Setup and Evaluation Metrics

To validate the effectiveness of the proposed high-order QAM synchronization and parameter measurement method, a MATLAB simulation platform is established. A systematic analysis is conducted of timing synchronization, coarse carrier synchronization, fine carrier synchronization, and their cascaded performance. In addition, measurement results obtained from Vector Signal Explorer Software Version 20.0 (VSE) developed by Rohde & Schwarz are used for comparative validation. The evaluation is performed on 256 QAM and 1024 QAM signals to assess the synchronization recovery capability and parameter measurement accuracy of the proposed method under different modulation orders. During the simulation, high-order QAM baseband signals are generated at the transmitter. Pulse shaping is performed using a root-raised cosine (RRC) filter, after which the signal is transmitted through an AWGN channel. Timing offsets, carrier frequency offsets, and phase deviations are then introduced to emulate dominant synchronization impairments encountered in practical reception scenarios. At the receiver, after matched filtering, OM-based feedforward timing synchronization, coarse carrier synchronization, and fine carrier synchronization are sequentially performed. Subsequently, constellation recovery and parameter measurement are completed. For parameter measurement experiments, ten groups of 256 QAM and 1024 QAM signals are generated and evaluated under an AWGN channel with an SNR of 25 dB.

To comprehensively characterize system performance, both BER and multiple modulation quality parameters are used as evaluation metrics, including EVM and Modulation Error. Among these metrics, EVM and MER quantify the overall distortion of the demodulated signal relative to the ideal reference, while amplitude error, phase error, and frequency error characterize residual deviations in the amplitude, phase, and frequency domains, respectively [32]. For performance evaluation, the VSE measurement results serve as reference values, while MATLAB simulation results are treated as measured values. The RMS relative error is adopted as the final evaluation criterion. The proposed method is considered to meet measurement requirements when the RMS relative error of all evaluated parameters is below 5%.

4.3. Performance Analysis of Timing Synchronization

To evaluate the applicability of the proposed OM-based feedforward timing synchronization algorithm in high-order QAM scenarios, 256 QAM and 1024 QAM signals are generated in MATLAB and transmitted through an AWGN channel with an SNR of 25 dB. A RRC filter with a roll-off factor of 0.4 is employed for matched filtering at the receiver. The timing synchronization performance of the OM algorithm is then systematically evaluated.

As shown in Figure 6, the experimental results demonstrate that high-precision timing error estimation can be achieved without relying on feedback loops. The optimal sampling instants are accurately recovered via spline interpolation. For 256 QAM signals, the constellation diagrams before and after synchronization indicate that the Euclidean distance between constellation points is significantly increased after OM processing. Clear symbol clustering boundaries are observed, enabling reliable symbol decisions. Even in the presence of residual phase or combined frequency–phase offsets, the optimal sampling instants can still be reliably recovered, indicating that the OM algorithm is highly robust to carrier impairments. Furthermore, the BER of the 256 QAM signal after timing synchronization reaches $4.7\times {10}^{-5}$, satisfying the timing recovery accuracy requirements for high-order QAM systems.

As shown in Figure 7, for 1024 QAM signals, due to the denser constellation distribution, timing errors have a more pronounced impact on detection performance. Initial results indicate that, under default parameter settings, the separation between adjacent constellation points remains insufficient after synchronization, resulting in unstable symbol decisions. Further analysis reveals that the FFT segment length $L$ in the OM algorithm directly affects timing estimation accuracy. As the segment length increases, the relative timing estimation error decreases significantly, indicating improved estimation accuracy in high-order QAM scenarios. However, excessively long segment lengths increase computational complexity, necessitating a trade-off between accuracy and efficiency. Based on this observation, the OM algorithm parameters are optimized for 1024 QAM signals by increasing the segment length from 50 to 200 and the oversampling rate from 8 to 16.

The improved results show that constellation point separation becomes more distinct, enabling accurate recovery of the optimal sampling instants. The BER is reduced to $6.7\times {10}^{-5}$. These results confirm that the proposed OM-based timing synchronization algorithm achieves effective timing recovery for both 256 QAM and 1024 QAM signals, demonstrating strong applicability and stability under high-order modulation conditions.

Overall, the OM feedforward timing synchronization algorithm avoids the high complexity and limited real-time performance associated with conventional Gardner-based closed-loop structures, while maintaining high synchronization accuracy under high-order QAM conditions. In particular, for 1024 QAM signals, appropriately adjusting the FFT segment length and oversampling rate further enhances performance, providing high-quality input for subsequent carrier synchronization and parameter measurement stages.

4.4. Performance Analysis of Carrier Synchronization

In high-order QAM systems, the reduced Euclidean distance between adjacent constellation points makes the signal highly sensitive to carrier frequency offsets and phase deviations. These impairments lead to constellation rotation, dispersion, and decision distortion. To address these challenges, a two-stage carrier synchronization framework is adopted, comprising coarse synchronization for large-frequency-offset compensation and fine synchronization for residual frequency and phase error suppression, thereby enabling high-precision carrier recovery.

In the coarse synchronization stage, the FFT-based frequency offset estimation method is compared with the polarity decision-based open-loop algorithm, and the latter is further improved. For 1024 QAM signals, three representative constellation cases with frequency and phase offsets are evaluated. The estimated carrier frequency offsets are 2000.7 Hz, 1997.7 Hz, and 2006.7 Hz, respectively, all of which closely match the true offset. Further analysis indicates that approximately 99% of the normalized carrier frequency offset is compensated after coarse synchronization, and the relative estimation error is controlled within 1%. Consequently, the fine synchronization stage is only required to compensate for residual normalized frequency offsets on the order of 0.01 and phase errors.

Performance comparisons show that the polarity decision-based open-loop algorithm achieves a normalized frequency offset estimation that closely follows the true offset, significantly outperforming the FFT-based method in high-order QAM scenarios. Further evaluations on 256 QAM and 1024 QAM signals demonstrate that the proposed method maintains high estimation accuracy across different normalized frequency offset conditions, indicating strong adaptability to varying modulation orders and frequency offset ranges.

As shown in Figure 7 and Figure 8, in the fine synchronization stage, the improved PDF-based algorithm effectively compensates for residual frequency and phase errors after coarse synchronization. The resulting BER after fine synchronization reaches $5.8\times {10}^{-4}$. Convergence analysis shows that for 256 QAM signals, convergence is achieved within 15,000 samples, while for 1024 QAM signals, convergence occurs within 30,000 samples. In both cases, the processing time remains within 1 s, demonstrating efficient convergence.

As shown in Figure 9 and Figure 10, Overall, the proposed two-stage carrier synchronization scheme—coarse synchronization for large-offset suppression followed by fine synchronization for residual-error compensation—effectively resolves the trade-off between a wide acquisition range and high-precision phase tracking in high-order QAM systems. As shown in Figure 9 and Figure 10, Experimental results demonstrate that the improved coarse synchronization achieves approximately 99% initial frequency offset compensation accuracy, while the refined PDF-based fine synchronization achieves BER performance on the order of ${10}^{-4}$ with fast convergence, thereby validating the effectiveness and stability of the proposed carrier synchronization framework.

To further clarify the algorithmic contribution of the proposed carrier synchronization strategy, the coarse and fine synchronization stages were analyzed from the perspective of functional decomposition. The coarse stage is responsible for wide-range frequency-offset acquisition, whereas the fine stage focuses on residual phase and frequency tracking. This design differs from a conventional single-stage synchronization structure, in which the loop must jointly handle large initial frequency offsets and fine residual phase fluctuations.

The results show that the reliability-constrained coarse frequency estimator provides a more stable initial frequency-offset estimate before PLL tracking. As a result, the subsequent fine synchronization loop starts from a smaller residual-error region and converges more smoothly. This is especially beneficial for 1024 QAM, where the constellation spacing is much smaller and small phase deviations can easily cause symbol decision errors. After the proposed coarse-to-fine synchronization process, constellation rotation is effectively suppressed, symbol clustering becomes more compact, and the residual frequency and phase errors are significantly reduced.

From a measurement perspective, the improvement in carrier synchronization directly benefits the accuracy of modulation-quality parameter extraction. Since EVM, MER, phase error, and frequency error are all computed after synchronization recovery, residual carrier mismatch may be incorrectly interpreted as modulation distortion if synchronization is incomplete. The proposed cascaded structure reduces this error propagation by first removing the dominant frequency offset and then tracking the residual phase/frequency variations. Therefore, the proposed contribution is not limited to constellation recovery, but also improves the reliability of subsequent modulation-quality measurement.

4.5. Cascaded Synchronization Performance Analysis

To further validate the applicability of the proposed method in a complete demodulation chain, a cascaded simulation is conducted by integrating OM-based timing synchronization, coarse carrier synchronization, fine carrier synchronization, and parameter measurement modules. In the experiments, the received signal simultaneously contains timing offsets, carrier frequency offsets, and phase deviations. First, timing error estimation and optimal sampling recovery are performed using the OM algorithm. Next, the majority of the carrier frequency offset is compensated using the improved polarity decision-based open-loop coarse synchronization algorithm. Finally, the improved PDF-based algorithm is employed to suppress residual small frequency offsets and phase errors, achieving stable synchronization and demodulation for high-order QAM signals.

As shown in Figure 11, the cascaded synchronization results show that the constellation evolves from an initially distorted, rotated distribution to a clear, stable, clustered pattern after timing and two-stage carrier synchronization, indicating successful full-chain signal recovery.

As shown in Figure 12, the results illustrate the progressive recovery process from severely distorted constellations to a final constellation that closely matches the ideal reference, demonstrating the effectiveness of the proposed synchronization chain.

From a functional perspective, the OM timing synchronization module provides accurate sampling instants, ensuring optimal temporal alignment; the coarse synchronization module rapidly eliminates large frequency offsets, reducing the burden on fine synchronization; and the fine synchronization module precisely compensates for residual frequency and phase errors. The experimental results indicate that the cascaded modular design exhibits strong inter-stage compatibility, with stable transitions between modules and no significant error propagation or amplification.

Combined with the results from previous sections, the proposed cascaded synchronization method achieves BERs on the order of ${10}^{-5}$ for 256 QAM and ${10}^{-4}$ for 1024 QAM signals. In addition, accurate constellation recovery is achieved, providing reliable inputs for subsequent modulation quality analysis. These results demonstrate that the proposed cascaded synchronization framework not only performs well at the module level but also exhibits strong robustness and effectiveness in full system-level validation.

4.6. Chromatic-Dispersion Signal Model and Compensation Description

To make the signal-impairment model more representative of coherent optical measurement scenarios, chromatic dispersion (CD) is additionally considered at the signal-model level. CD is a typical linear impairment in fiber-optic transmission and may broaden the received waveform before synchronization recovery. In the frequency domain, the CD-impaired channel can be described as:

$$H_{\mathrm{C}\mathrm{D}}\left(\omega \right)=\exp \left(-j{\displaystyle \frac{{\beta }_{2}L}{2}}{\omega }^2\right)$$

(29)

where $L$ is the fiber length, $\omega$ is the angular frequency, and ${\beta }_2$ is the group-velocity dispersion parameter. The relationship between ${\beta }_2$ and the dispersion coefficient, $D$ is expressed as:

$${\beta }_2=-{\displaystyle \frac{D{\lambda }^2}{2\pi c}}$$

(30)

where $D$ is the dispersion coefficient, $\lambda$ is the optical wavelength, and $c$ is the speed of light. Accordingly, the received signal with CD and AWGN can be written as:

$$r_{\mathrm{C}\mathrm{D}}\left(t\right)={\mathcal{F}}^{-1}\left\{\mathcal{F}\left\{s\left(t\right)\right\}{H}_{\mathrm{C}\mathrm{D}}\left(\omega \right)\right\}+n\left(t\right)$$

(31)

where $s\left(t\right)$ is the transmitted baseband signal, $n\left(t\right)$ denotes additive white Gaussian noise, and $\mathcal{F}\{\cdot \}$ represents the Fourier transform.

In a standard coherent receiver DSP flow, CD is commonly compensated before carrier and timing synchronization. Therefore, a frequency-domain inverse CD equalizer can be introduced as:

$$G_{\mathrm{C}\mathrm{D}}\left(\omega \right)=H_{\mathrm{C}\mathrm{D}}^{-1}\left(\omega \right)$$

(32)

After this front-end CD compensation stage, the subsequent processing chain remains the same as that used in the AWGN case, including matched filtering, OM timing synchronization, polarity-decision-based coarse carrier-frequency estimation, PLL-based fine carrier synchronization, symbol decision, and modulation-quality parameter extraction. This description clarifies that the proposed synchronization and modulation-quality evaluation framework is intended to operate after standard front-end linear impairment compensation, rather than to replace a complete coherent optical receiver DSP system.

4.7. Parameter Measurement Accuracy Under Nominal Conditions

After synchronization and demodulation, modulation-quality parameters for high-order QAM signals are measured from the recovered constellations. The results obtained from MATLAB simulations are compared with those measured by R&S FSV3030 Signal Analyzer Software to evaluate the accuracy of the proposed method. The evaluated parameters include EVM, MER, amplitude error, phase error, and frequency error. The RMS relative error is adopted as the evaluation criterion. The method is considered valid when the error of each parameter is below 5%.

As shown in

Table 3. Parameter Measurement Accuracy for 256 QAM under Nominal Conditions (mAP %).

Parameter	RMS Relative Error (%)	Criterion	Conclusion
EVM	0.547	5%	Qualified
MER	0.589	5%	Qualified
Amplitude Error	1.045	5%	Qualified
Phase Error	1.212	5%	Qualified
Frequency Error	1.422	5%	Qualified

, the RMS relative errors are 0.547% for EVM, 0.589% for MER, 1.045% for amplitude error, 1.212% for phase error, and 1.422% for frequency error. It is observed that the EVM and MER errors, as comprehensive indicators, are both below 0.6%, while amplitude, phase, and frequency errors remain below 1.5%, which is significantly lower than the 5% threshold. These results demonstrate that the proposed method achieves both stable synchronization and high-precision parameter measurement under 256 QAM conditions.

As shown in

Table 4. Parameter Measurement Accuracy for 1024 QAM under Nominal Conditions (mAP %).

Parameter	RMS Relative Error (%)	Criterion	Conclusion
EVM	0.947	5%	Qualified
MER	1.189	5%	Qualified
Amplitude Error	1.645	5%	Qualified
Phase Error	1.812	5%	Qualified
Frequency Error	1.922	5%	Qualified

, the RMS relative errors are 0.947% (EVM), 1.189% (MER), 1.645% (amplitude error), 1.812% (phase error), and 1.922% (frequency error). Compared with 256 QAM, the measurement errors are slightly higher but remain below 2%, which is still significantly lower than the 5% threshold. This increase is mainly attributed to the reduced number of effective symbols participating in carrier synchronization under higher-order modulation, which slightly weakens robustness to noise and residual synchronization errors. Nevertheless, the obtained error levels remain within acceptable limits, indicating good measurement stability under higher modulation orders.

Overall, strong consistency is observed between the MATLAB simulation results and the VSE reference measurements for both 256 QAM and 1024 QAM signals. The RMS relative errors of EVM and MER are maintained below 1.2%, while amplitude, phase, and frequency errors are all below 2%. All evaluated parameters satisfy the 5% performance criterion. Therefore, the proposed synchronization and parameter measurement method achieves high accuracy and reliability in the analysis of high-order QAM modulation quality. The parameter measurement experiments further verify the consistency of the overall processing chain, including synchronization recovery, constellation stabilization, and parameter extraction. The front-end synchronization ensures accurate constellation clustering, while the back-end measurement module enables stable extraction of EVM, MER, and amplitude, phase, and frequency errors. All measurement results satisfy the predefined criteria, demonstrating that the proposed method provides a reliable foundation for modulation quality evaluation of high-order QAM signals.

4.8. Robustness of Parameter Measurement Under Varying Sampling Rates and SNR Conditions

To comprehensively evaluate the robustness and reliability of the proposed algorithm, two groups of QAM signals with varying sampling rates and SNRs are generated in MATLAB, As shown in

Table 5. Test configuration for 256 QAM signals.

No.	Sampling Rate (MHz)	SNR (dB)	No.	Sampling Rate (MHz)	SNR (dB)
01	200	30	07	100	30
02	200	20	08	100	20
03	200	10	09	100	10
04	150	30	10	50	20
05	150	20	11	50	20
06	150	10	12	50	10

and

Table 6. Test configuration for 1024 QAM signals.

No.	Sampling Rate (MHz)	SNR (dB)	No.	Sampling Rate (MHz)	SNR (dB)
01	200	30	07	100	30
02	200	20	08	100	20
03	200	10	09	100	10
04	150	30	10	50	20
05	150	20	11	50	20
06	150	10	12	50	10

. The generated signals are processed using both the proposed method and R&S FSV3030 Signal Analyzer Software, and the measurement errors are evaluated through comparison. At the methodological level, the OM-based feedforward timing synchronization effectively eliminates the delay and error accumulation inherent in conventional closed-loop structures, enabling fast and stable recovery of optimal sampling instants under dense high-order constellations.

The designed two-stage carrier synchronization scheme achieves an effective balance between wide-range frequency offset acquisition and high-precision residual phase compensation by combining polarity-based coarse estimation and improved PDF-based fine synchronization. On this basis, a unified modulation quality parameter evaluation scheme is established, enabling accurate measurement of EVM, Modulation Error Ratio (MER), and amplitude, phase, and frequency errors.

As shown in Figure 13, the relative errors of EVM and MER increase as both the sampling rate and SNR decrease. It is observed that SNR has a more significant impact than sampling rate. The relative errors of amplitude, phase, and frequency are generally larger than those of EVM and MER, indicating higher sensitivity to residual fluctuations after demodulation. Among these parameters, phase error is the most sensitive to noise and sampling degradation, leading to faster error growth. Nevertheless, all five parameters remain below the 5% threshold, satisfying the performance requirements of vector signal analyzers.

As illustrated in Figure 14, similar trends are observed for 1024 QAM signals. The increased measurement error is primarily due to the reduced probability of corner constellation symbols in 1024 QAM, leading to fewer valid points participating in carrier synchronization. The RMS relative errors of EVM and MER are higher than those in 256 QAM and are more sensitive to SNR and sampling rate variations. Similarly, amplitude, phase, and frequency errors are larger than those for 256 QAM, with phase error increasing most rapidly under degraded conditions. Despite these increases, all parameter errors remain below the 5% threshold, satisfying the performance criteria.

Overall, the proposed method demonstrates strong robustness across varying sampling rates and SNR conditions. All evaluated parameters remain within acceptable limits under different operating conditions. These results confirm the stability and reliability of the proposed synchronization and parameter-measurement framework for high-order QAM signals under practical, non-ideal conditions.

5. Conclusions

In this paper, a digital baseband cascaded processing framework integrating timing synchronization, carrier synchronization, and parameter measurement is proposed to address key challenges in synchronization recovery and modulation quality analysis for high-order Quadrature Amplitude Modulation (QAM) signals. A unified synchronization error model and processing framework are established, enabling joint optimization from signal recovery to parameter evaluation.

At the methodological level, the OM-based feedforward timing synchronization effectively eliminates the delay and error accumulation inherent in conventional closed-loop structures, enabling fast and stable recovery of optimal sampling instants under dense high-order constellations. The designed two-stage carrier synchronization scheme achieves an effective balance between wide-range frequency offset acquisition and high-precision residual phase compensation by combining polarity-based coarse estimation and improved PDF-based fine synchronization. On this basis, a unified modulation quality parameter evaluation scheme is established, enabling accurate measurement of EVM, MER, as well as amplitude, phase, and frequency errors.

Experimental results demonstrate that stable synchronization recovery and clear constellation reconstruction are achieved under both 256 QAM and 1024 QAM conditions. In terms of parameter measurement, the relative errors of all evaluated metrics are significantly below the 5% engineering threshold, verifying the measurement accuracy and consistency under high-order modulation conditions. Furthermore, robust performance is maintained under varying sampling rates and signal-to-noise ratios, indicating strong adaptability to practical operating conditions.

Overall, the proposed cascaded framework overcomes the conventional separation between synchronization recovery and parameter measurement, achieving unified design and cooperative optimization of both processes. This provides a practical and effective solution for modulation quality analysis of high-order QAM signals. Future work may focus on three aspects: reducing computational complexity to support real-time measurement requirements; improving robustness under low-SNR conditions; and incorporating data-driven or deep learning-based approaches to enhance adaptability and generalization in complex channel environments.