Performance Evaluation of Post-Quantum Digital Signature in QPSK- and 16QAM-Based WDM Communication Systems

Abstract

The integration of post-quantum digital signature (PQDS) algorithms into coherent wavelength-division multiplexing (WDM) optical networks introduces a non-negligible cryptographic overhead that fundamentally alters physical-layer performance characteristics. Unlike conventional studies that treat security and transmission independently, this work provides a cross-layer evaluation of PQDS-induced payload expansion and its direct impact on coherent optical system behavior under realistic, DSP-aligned conditions. A structured and reproducible evaluation framework is proposed to systematically analyze this interaction across multiple transmission scenarios, ranging from a single-channel QPSK baseline to a 16-channel WDM system employing both QPSK and 16QAM modulation formats. Key system parameters—including launch power, local oscillator power, bit rate, and fiber length—are jointly optimized, while performance is rigorously assessed in terms of bit error rate (BER), Q-factor, and maximum transmission reach. The results demonstrate a clear performance degradation trend driven by both spectral efficiency scaling and cryptographic payload expansion. The single-channel QPSK system achieves a maximum reach of 203 km, which decreases to 194 km in the 16-channel WDM QPSK configuration due to inter-channel interference and nonlinear effects. In contrast, the 16-channel WDM 16QAM system exhibits a significantly reduced reach of 103 km, reflecting its heightened sensitivity to noise, chromatic dispersion, and fiber nonlinearities. Furthermore, increased payload size associated with PQDS schemes is shown to exacerbate transmission impairments by extending frame duration and intensifying inter-channel interactions. These findings identify PQDS-induced overhead as a critical system-level constraint that directly governs transmission efficiency, scalability, and performance limits. The study highlights the necessity of cross-layer co-design strategies, where cryptographic mechanisms and physical-layer parameters are jointly optimized to enable efficient, reliable, and quantum-safe coherent optical communication systems.

1. Introduction

The rapid advancement of quantum computing poses a serious threat to the security of modern communication systems. Contemporary secure communication frameworks rely heavily on classical cryptographic mechanisms to guarantee data confidentiality, integrity, and authenticity. However, these guarantees are expected to be undermined in the presence of quantum-capable adversaries [1,2]. In particular, widely used public-key cryptosystems, such as RSA and Elliptic Curve Cryptography (ECC), are vulnerable to quantum algorithms, highlighting the urgent need for quantum-resistant cryptographic solutions. In this context, post-quantum digital signature (PQDS) schemes have emerged as a key enabler of next-generation secure communication systems. These schemes are specifically designed to resist quantum attacks while preserving essential security properties, including authenticity and non-repudiation [3,4]. At the same time, optical fiber networks, particularly wavelength division multiplexing (WDM) systems, have become the backbone of global communication infrastructure by enabling ultra-high-capacity transmission over long distances. These systems are designed to maximize spectral efficiency, throughput, and transmission reach under increasingly demanding operating conditions. As communication networks evolve toward quantum-safe operation, ensuring data authenticity and integrity becomes as critical as maintaining transmission performance. Consequently, the adoption of PQDS is no longer a theoretical option but a practical requirement for future secure optical networks. Nevertheless, this transition introduces important system-level challenges, since PQDS schemes typically involve larger public keys and longer signatures than conventional digital signature schemes, resulting in payload expansion, higher processing latency, and increased computational overhead during signing and verification. Recent standardization efforts by the National Institute of Standards and Technology (NIST) have identified several leading PQDS candidates, including CRYSTALS-Dilithium, FALCON, and SPHINCS+ [5]. While these schemes provide strong security guarantees, they introduce trade-offs that affect system performance, particularly in terms of signal characteristics, processing complexity, and overall transmission efficiency [6]. In high-capacity WDM systems, even modest increases in cryptographic overhead can significantly impact system performance. Specifically, additional payload expansion leads to reduced effective throughput and increased performance degradation, especially in dense multi-channel configurations where impairments accumulate across wavelengths [7]. As a result, an inherent design tension arises: coherent WDM systems are optimized for high transmission efficiency, whereas PQDS integration increases payload size and processing burden, potentially degrading system performance in latency-sensitive and high-throughput optical networks [8,9]. Most existing studies in optical communications focus primarily on physical-layer optimization, including bit error rate (BER), Q-factor, optical signal-to-noise ratio (OSNR), transmission reach, and tolerance to dispersion and nonlinear impairments. In such studies, security-induced overhead is often neglected or implicitly assumed to be insignificant [9]. Conversely, research in post-quantum cryptography typically evaluates PQDS schemes in isolation, emphasizing computational metrics such as key size, signature size, and signing/verification latency across different platforms, including 5G [10,11], IoT and resource-constrained devices [12,13], hardware implementations [14], networking and performance evaluation [15,16], algorithm optimization [3,17], and emerging 6G systems [18]. As a consequence, the interaction between cryptographic overhead and transmission-layer constraints remains insufficiently explored. This gap is particularly critical in coherent optical systems, where performance is highly sensitive to physical-layer design parameters. PQDS-induced payload expansion reduces the proportion of useful information carried per transmitted symbol, thereby lowering the effective spectral efficiency [19]. The problem becomes more severe when higher-order modulation formats are employed, because densely packed constellation points are more vulnerable to noise, phase distortion, and transmission impairments, which increases BER and reduces performance margins [20].

Despite the rapid progress in post-quantum cryptography and coherent optical communication systems, the interaction between cryptographic overhead and physical-layer high-order modulation performance remains insufficiently quantified. Existing studies typically assess either PQDS computational cost (e.g., latency, signature size) or optical transmission performance (e.g., BER, Q-factor) in isolation, without establishing a direct system-level relationship between them.

Consequently, it remains unclear how PQDS-induced payload expansion affects key transmission characteristics in coherent WDM systems under realistic operating conditions, particularly when combined with factors such as modulation format, channel density, and transmission distance. This lack of integrated analysis limits the ability to design efficient quantum-safe optical systems.

Accordingly, the main problem addressed in this work is the absence of a unified framework that captures the cross-layer interaction between PQDS overhead and physical-layer performance.

To address this problem, this work proposes a structured cross-layer evaluation framework that quantitatively links PQDS overhead to physical-layer performance metrics. The framework enables systematic analysis across multiple system configurations, providing insight into the trade-offs between security overhead, spectral efficiency, transmission reach, and system robustness.

Furthermore, common attempts to compensate for degraded performance, such as increasing launch power, may intensify nonlinear effects in WDM links, including inter-channel interference, cross-phase modulation (XPM), and four-wave mixing (FWM), while also increasing accumulated noise [21]. Therefore, the overall impact of PQDS integration cannot be understood independently of key optical design parameters such as modulation format, channel spacing, and launch power. Previous studies have already shown that reduced channel spacing and improper power allocation can significantly degrade BER performance and constellation quality in coherent detection environments [22]. A limited number of recent studies have started to investigate the integration of PQDS into optical communication systems. For example, Hamood et al. (2025) examined the impact of PQDS on optical-system performance by considering factors such as signature size, verification latency, and payload overhead in relation to BER, Q-factor, and transmission reach [23]. To address this gap, this work proposes a unified cross-layer framework for evaluating the impact of PQDS integration in coherent WDM optical systems. Unlike prior studies that mainly assess cryptographic cost or optical-layer performance separately, the proposed framework explicitly links PQDS-induced payload overhead to key transmission metrics, including BER, Q-factor, transmission reach, and spectral efficiency, under consistent operating conditions. The analysis covers multiple system configurations, namely single-channel QPSK, 16-channel WDM QPSK, and 16-channel WDM 16QAM, together with a comparative evaluation of representative PQDS schemes. Through this framework, the study identifies operating regimes in which security overhead remains tolerable, as well as regimes in which performance degradation becomes significant. In this way, the work provides a clearer understanding of the trade-offs among security overhead, spectral efficiency, and transmission reliability, and offers practical guidance for the design of scalable quantum-safe coherent optical networks.

Unlike existing studies that evaluate cryptographic performance and physical-layer transmission in isolation, the proposed framework explicitly links PQDS-induced overhead to optical system behavior under realistic operating conditions. Prior works have focused on computational aspects of PQDS schemes, such as key size, signature size, and latency, without considering their impact on transmission characteristics in coherent WDM systems. Similarly, optical communication studies have extensively analyzed modulation formats, nonlinear effects, and channel scaling, while largely neglecting the influence of security-related payload expansion.

In contrast, this work provides a structured cross-layer evaluation that quantitatively captures how PQDS-induced payload expansion affects key physical-layer metrics, including BER, Q-factor, and transmission reach. Furthermore, the proposed framework enables a systematic comparative analysis across multiple system configurations and modulation formats, revealing system-level trade-offs between spectral efficiency, transmission robustness, and security overhead.

To the best of our knowledge, this work represents one of the first attempts to quantitatively link PQDS-induced payload expansion to physical-layer transmission limits in coherent WDM systems under realistic operating conditions.

The main contributions of this work are summarized as follows:

1.

A unified cross-layer framework is developed to evaluate PQDS: A structured cross-layer evaluation framework that explicitly links PQDS-induced overhead to physical-layer performance in coherent WDM systems.

2.

The impact of PQDS-induced payload expansion on BER, Q-factor, transmission reach, and spectral efficiency is quantitatively analyzed across multiple coherent optical-system configurations.

3.

The interaction between security overhead, modulation order, and WDM channel scaling is investigated to reveal critical trade-offs between transmission robustness and spectral-efficiency enhancement.

4.

Practical PQDS-aware design insights are derived to support the development of efficient and scalable quantum-safe coherent optical communication systems.

2. Related Work

The development of quantum-safe coherent optical communication systems requires the integration of two research domains that have traditionally evolved along separate paths: post-quantum cryptography and physical-layer optical transmission. On one side, post-quantum cryptographic research has concentrated on the security, computational efficiency, and implementation cost of candidate algorithms. On the other side, optical communication research has focused on maximizing transmission performance through modulation design, wavelength-division multiplexing (WDM) scalability, and impairment mitigation. Although both domains are essential for future secure high-capacity networks, their interaction remains insufficiently understood. In particular, the transmission-level consequences of post-quantum digital signature (PQDS) overhead in coherent WDM optical systems have not yet been systematically characterized. Existing literature relevant to this problem can be broadly categorized into three main directions: performance evaluation of post-quantum cryptographic schemes, physical-layer limitations of high-order modulation in coherent systems, and design constraints in dense WDM transmission. However, these directions remain only weakly connected in prior work, leaving an important cross-layer gap.

2.1. Post-Quantum Cryptography

Recent studies in post-quantum cryptography have primarily examined the practical feasibility of quantum-resistant algorithms from a computational and protocol perspective. In this context, researchers have evaluated latency, bandwidth cost, key size, signature size, and processing overhead associated with deploying post-quantum mechanisms in modern communication environments. For example, Montenegro et al. investigated post-quantum handshake mechanisms in TLS and QUIC under both ideal and realistic networking conditions, demonstrating that the introduction of post-quantum algorithms can significantly affect communication efficiency, particularly when protocol overhead is large [24]. Their results showed that hybrid approaches may introduce considerable latency and bandwidth penalties, while pure post-quantum solutions may provide a more balanced trade-off between security and performance. In parallel, a number of studies have focused specifically on post-quantum digital signature schemes, especially with respect to their implementation characteristics and comparative efficiency. These studies typically assess metrics such as public-key size, signature length, signing time, verification time, and computational complexity across different hardware and software environments. For instance, one study evaluated the properties of multiple PQDS candidates and showed that substantial differences exist among schemes in terms of key and signature size, as well as execution latency [25]. Such findings are highly relevant when selecting a signature algorithm for secure communication systems, as the overhead associated with a given PQDS family can vary significantly depending on whether the scheme is lattice-based or hash-based [26].

2.2. High-Order Modulation

High-order modulation formats in coherent optical communication systems are widely employed to enhance spectral efficiency; however, this improvement is inherently accompanied by increased sensitivity to transmission impairments. In particular, higher-order schemes such as 16QAM enable higher data rates by transmitting more bits per symbol, but at the cost of reduced robustness to noise, chromatic dispersion, and nonlinear effects. Previous studies have investigated the impact of chromatic dispersion and phase noise on QPSK and 16QAM modulation formats. These studies demonstrated that QPSK maintains relatively stable performance under varying dispersion conditions due to its wider constellation spacing, whereas 16QAM exhibits significantly higher sensitivity to both chromatic dispersion and phase noise. This is primarily because its densely packed constellation points are more susceptible to distortion, leading to increased symbol errors and higher bit error rates (BER) [27]. Although various compensation techniques have been proposed to mitigate these impairments, including advanced digital signal processing (DSP) methods, most existing studies focus exclusively on optimizing physical-layer performance [28]. The influence of higher-layer factors—particularly security-induced overhead—on modulation performance remains largely unexplored. In addition, the impact of noise on coherent optical systems has been extensively studied through constellation-based analysis. Prior work has shown that accumulated noise leads to constellation distortion, which directly degrades BER performance even in relatively robust modulation schemes such as QPSK [19,23]. These findings further indicate that, despite the use of coherent detection and advanced DSP techniques, system performance remains highly sensitive to noise and channel impairments. However, these studies primarily consider physical-layer impairments in isolation, without accounting for additional factors such as increased payload size and processing latency introduced by security mechanisms. Consequently, the interaction between high-order modulation formats and PQDS-induced overhead in coherent WDM systems remains insufficiently understood, particularly under conditions where systems operate near their performance limits.

2.3. Dense WDM Optical Systems

The evolution toward dense wavelength-division multiplexing (WDM) systems has introduced significant challenges due to both linear and nonlinear fiber impairments. As channel spacing is reduced to improve spectral efficiency, the interaction between co-propagating channels becomes increasingly complex, leading to performance degradation. Several studies have analyzed the combined impact of nonlinear effects, including self-phase modulation (SPM), cross-phase modulation (XPM), four-wave mixing (FWM), and stimulated Raman scattering (SRS), on high-density WDM transmission. These studies demonstrated that reducing channel spacing intensifies inter-channel interference (ICI) and nonlinear interactions, particularly under high launch power and long transmission distances, resulting in increased BER and degraded system performance [29]. In addition to channel spacing, system performance in coherent optical networks is highly sensitive to operating conditions. Variations in key parameters such as polarization control, optical signal-to-noise ratio (OSNR), and nonlinear effects can significantly accelerate performance degradation, especially in high-order modulation systems. This sensitivity is further amplified in dense WDM environments, where impairments accumulate across multiple channels, leading to compounded performance limitations [30]. To achieve optimal system performance, coherent optical systems require joint optimization of critical parameters, including launch power, channel spacing, and digital signal processing (DSP) operations. However, existing studies primarily focus on physical-layer optimization and do not account for additional constraints introduced by higher-layer mechanisms, particularly security-related overhead in PQDS-enabled systems [31] Although substantial progress has been made in both post-quantum cryptography and optical communication system design, these research domains have evolved largely in isolation. PQC research predominantly emphasizes computational efficiency, security strength, and protocol-level performance, whereas optical communication studies focus on improving physical-layer metrics and mitigating transmission impairments [32]. As a result, there is a lack of a unified cross-layer perspective that captures the interaction between cryptographic overhead and physical-layer performance. This limitation becomes particularly critical in coherent optical systems operating near their performance limits, where even minor increases in payload size or processing complexity can significantly affect system behavior. Therefore, a systematic evaluation framework is required to bridge this gap by jointly analyzing the impact of PQDS-induced overhead on coherent optical transmission performance. This need forms the foundation of the methodology proposed in this work.

Despite the growing adoption of post-quantum digital signature schemes in secure communication systems, their impact on physical-layer performance in coherent WDM optical networks remains unclear. In particular, the effect of PQDS-induced payload expansion on key metrics such as BER, Q-factor, and transmission reach has not been systematically quantified.

3. Secured High-Order Modulation WDM Optical Systems

The rapid growth of data-intensive applications has significantly increased the demand for high-capacity and secure transmission in optical systems. The individual channel capacity of fiber communication links using on–off keying (OOK) has steadily increased to 10 Gb/s rates per wavelength. Combined with dense wavelength division multiplexing technology, this has resulted in transmission rates of $B_{\mathrm{tot}}={\sum }_{i=1}^N{B}_N$.

To increase the single fiber capacity even more [33], higher-order modulation formats should be used to increase the spectral efficiency. The transmitters for a coherent system differ from those of direct detection because they need to generate an optical signal with proper encoding of both the amplitude and phase of the electric field. The modulator of these transmitters maps a sequence of k binary symbols, which in the case of equiprobable symbols, carries k bits of information into a set of corresponding signal waveforms, $1\le m\le M$, where $M=2^k$.

We assume that these signals are transmitted at every $T_s$ seconds, where $T_s$ is called the signaling interval. This means that in each second, $R_s={\displaystyle \frac{1}{T_s}}$ symbols are transmitted. Parameter $R_s$ is called the symbol rate [34]. Since each signal carries k bits of information, the bit interval $T_b$, i.e., the interval in which one bit of information is transmitted, is given by $T_b={\displaystyle \frac{T_s}{k}}={\displaystyle \frac{T}{{log}_{2}M}}$ and the bit rate R is given by $R=k{R}_s$ [34].

$$R=k{R}_s=R_s{log}_{2}M$$

(1)

A coherent optical receiver typically incorporates a 90° hybrid and two balanced receivers using four identical photodiodes. In this technique, the local oscillator output and the incoming optical signal are usually combined using a four-port (i.e., 3 dB) single-mode fiber coupler. The signal in one fiber in this device suffers a $\pi /2$ phase shift upon transfer to the other fiber. The two detector output voltages are thus given by:

$$V_A=E_s{E}_{L}sin({\omega }_s-{\omega }_L)t$$

(2)

$$V_B=E_s{E}_{L}sin({\omega }_L-{\omega }_s)t$$

(3)

It may be observed that these output voltages are similar but of opposite sign in that $V_A=-V_B$. The two output voltages are operated upon by the combiner function $(\Sigma )$, and as one is a positive input and the other is a negative input, then the output from the combiner function $V_o$ will form the difference between the two inputs, such that:

$$V_o=V_A-V_B=2V_A$$

(4)

Equation (5) indicates that twice the voltage, or four times the power, is provided in comparison with the single optical detector scheme. This technique therefore gives a 6 dB improvement over the single optical detector. These systems, signal quality and reliability, are typically evaluated using performance metrics such as SNR, Q-factor, and BER. In an optical coherent detection receiver, the SNR is predominantly influenced by shot noise, as the power of the local oscillator typically exceeds that of the received optical signal significantly. Consequently, taking into account solely shot noise and thermal noise, the signal-to-noise ratio (SNR) of the receiver is [7,30]

$$\mathrm{SNR}={\displaystyle \frac{i_s^2\left(t\right)}{i_{th}^2\left(t\right)+i_{shot}^2\left(t\right)}}$$

(5)

where $i_s^2\left(t\right)$, $i_{th}^2\left(t\right)$, and $i_{shot}^2\left(t\right)$ are the mean-square values of signal current, thermal noise current, and shot-noise current, respectively. In homodyne detection with optical phase locking between the received optical signal and the local oscillator, the signal power is [30] ${\left(i_s^2\left(t\right)\right)}_{\mathrm{homodyne}}=R^2{P}_s\left(t\right)P_{LO}$, where $P_s$ is the receiver sensitivity, $P_{LO}$ is the local oscillator optical power, and R is the responsivity.

The thermal noise power and the shot noise power in Equation (1) [33] are given by $i_{th}^2={\displaystyle \frac{4K_{B}T{B}_e}{R_L}}$, where $B_e$ is the receiver electrical bandwidth, $K_B$ is the Boltzmann constant ($1.38054\times {10}^{-23}$ J/K), T is the temperature, and $R_L$ is the load resistor.

Meanwhile, $i_{shot}^2=2qR\left({\displaystyle \frac{P_s\left(t\right)}{2}}+P_{LO}\right)B_e$, where q is the electron charge $1.60218\times {10}^{-19}$ C. Therefore, the SNR will equal [33]

$$\mathrm{SNR}={\displaystyle \frac{R^2{P}_s\left(t\right)P_{LO}}{4K_{B}T/R_L+2qR\left[{\displaystyle \frac{P_s\left(t\right)}{2}}+P_{LO}\right]}}·{\displaystyle \frac{1}{B_e}}$$

(6)

If the LO power is strong compared to the signal power, then the shot noise created by the local oscillator dominates the thermal noise [9,33].

$$\mathrm{SNR}\approx {\displaystyle \frac{R}{2q{B}_e}}{P}_s\left(t\right)$$

(7)

The relationship between electrical signal-to-noise ratio (SNR) and optical signal-to-noise ratio (OSNR) can be derived by comparing the definitions of noise spectral density in optical and electrical domains. Considering the reference optical bandwidth $B_{\mathrm{ref}}$, the OSNR can be expressed as [7]

$$\mathrm{OSNR}={\displaystyle \frac{R_b}{2B_{\mathrm{ref}}}}\mathrm{SNR}$$

(8)

where $R_b$ is the bit rate and $B_{\mathrm{ref}}$ denotes the reference optical bandwidth, typically $0.1$ nm (≈12.5 GHz). This relation establishes a direct link between electrical performance metrics and optical signal quality.

For coherent detection, $Q=\sqrt{\mathrm{SNR}}$, and under Gaussian noise assumptions, the BER is given by [9]

$$\mathrm{BER}={\displaystyle \frac{1}{2}}\mathrm{erfc}\left({\displaystyle \frac{Q}{\sqrt{2}}}\right)$$

(9)

Digital signal processing (DSP) plays a critical role in mitigating transmission impairments and enabling reliable high-order modulation. Typical DSP operations include:

• Chromatic dispersion compensation.
• Polarization demultiplexing.
• Carrier phase recovery.
• Frequency offset correction.
• Symbol timing recovery.

However, it is important to note that DSP operations introduce additional computational overhead and latency, particularly when the transmitted data structure is modified due to higher-layer mechanisms such as post-quantum cryptographic encoding [28]. Therefore, any modification in the transmitted frame structure, such as increased payload size due to PQDS integration, may directly influence DSP efficiency, processing latency, and overall system performance.

4. System Model and Proposed Architecture

This study proposes a unified cross-layer system model for evaluating the performance of high-order modulation schemes in post-quantum digital signature (PQDS)-enabled coherent wavelength-division multiplexing (WDM) optical systems. The proposed framework is designed to capture the interaction between physical-layer transmission characteristics and security-induced overhead within a realistic communication environment. Unlike conventional approaches that treat cryptographic processing and optical transmission independently, the proposed model integrates both domains into a single evaluation framework. This enables direct analysis of how PQDS-related overhead affects key system performance metrics. The system incorporates both transmission parameters such as modulation format, launch power, and channel configuration and cryptographic parameters, including signature size and processing latency. The performance is evaluated in terms of bit error rate (BER), Q-factor, and maximum transmission reach under varying system conditions.

The overall workflow of the proposed system is illustrated in Figure 1. The process begins with an input text file, which is processed using post-quantum digital signature (PQDS) algorithms including FALCON, Dilithium, and SPHINCS+. The algorithms are evaluated based on key generation and signing time, and the optimal scheme (FALCON-512) is selected.

The generated digital signature is appended to the original data, resulting in payload expansion due to the PQDS overhead. The secured data are then converted into a binary bitstream and mapped into high-order modulation formats (QPSK and 16QAM).

The modulated signal is transmitted through a simulated coherent optical communication system implemented in the OptiSystem environment, including WDM multiplexing, fiber propagation, dispersion compensation, amplification, and coherent detection. To evaluate system performance, a parameter sweep is conducted by varying key transmission parameters such as launch power, LO power, bit rate, and channel spacing. Multiple simulation runs are performed, and the resulting BER, Q-factor, and transmission reach are recorded to construct the dataset. At the receiver side, the signal is demapped and decoded, followed by signature verification through hash recomputation. Finally, the results are analyzed to quantify the impact of PQDS-induced payload overhead on optical transmission performance across different system configurations.

4.1. Secured Payload Framing

Secured payload framing in the proposed system is achieved by integrating post-quantum digital signatures (PQDS) into the transmitted data prior to optical modulation. This integration introduces additional payload overhead, which directly influences transmission efficiency and must therefore be explicitly considered in the system model. The total transmitted payload can be expressed as the sum of the original data and the appended signature:

$$L_{\mathrm{total}}=L_{\mathrm{data}}+L_{\mathrm{sig}}$$

(10)

where $L_{\mathrm{data}}$ denotes the original payload size and $L_{\mathrm{sig}}$ represents the size of the PQDS signature. The increase in payload length results in a higher transmission load, which may affect system performance, particularly in bandwidth-constrained and high-capacity optical environments [23].

To ensure a consistent and fair evaluation, multiple PQDS schemes are assessed under identical operating conditions. The evaluation focuses on key cryptographic parameters, including signature size, key generation time, and signing and verification latency, as these factors play a critical role in determining the suitability of each scheme for high-speed optical communication systems, as illustrated in Figure 2.

At the transmitter side, the input data are first processed using a cryptographic hashing function (SHA-256), producing a fixed-length digest. This digest is then signed using a selected PQDS algorithm, and the resulting signature is appended to the original data to form a secured payload. The secured payload is subsequently mapped onto modulation symbols using coherent modulation formats such as quadrature phase shift keying (QPSK) or 16-level quadrature amplitude modulation (16QAM), preparing the signal for transmission through the WDM optical channel.

At the receiver side, coherent detection is performed followed by digital signal processing (DSP) operations to recover the transmitted data and their associated signature. The received payload is reprocessed to generate a new hash using the same SHA-256 function, while the received signature is verified using the sender’s corresponding public key associated with the selected PQDS scheme. Data integrity and authenticity are confirmed by comparing the reconstructed hash with the verified signature.

To ensure compatibility with DSP operations, the input data sequence is adjusted to the nearest power-of-two length. Specifically, an input payload of 8144 bits is extended to 8192 bits, which enables efficient block-based processing and consistent symbol mapping across different modulation formats. The aligned payload length can be expressed as:

$$L_{\mathrm{aligned}}=2^k$$

(11)

where k is an integer such that $L_{\mathrm{aligned}}$ is the smallest power of two greater than or equal to the original payload length.

This alignment improves processing stability, simplifies implementation, and enhances the accuracy of performance evaluation metrics such as BER and Q-factor. Furthermore, it ensures uniform symbol mapping, where the number of transmitted symbols is determined by:

$$N_{\mathrm{sym}}={\displaystyle \frac{L_{\mathrm{aligned}}}{{log}_{2}M}}$$

(12)

where M represents the modulation order.

Overall, the secured payload framing mechanism establishes a direct link between cryptographic processing and physical-layer transmission behavior. This enables a systematic investigation of how PQDS-induced overhead impacts key performance metrics, providing a foundation for cross-layer optimization in coherent WDM optical systems, as illustrated in Figure 3.

4.2. Proposed Coherent WDM System (QPSK/16QAM)

Coherent wavelength-division multiplexing (WDM) systems are implemented using both single-channel and multi-channel configurations to enable a comprehensive evaluation of system performance under varying complexity levels. In this study, two primary configurations are considered: single-channel QPSK and single-channel 16QAM, followed by their extension to multi-channel WDM systems. This structured approach allows systematic investigation of the combined effects of modulation format and WDM scaling on overall system performance. The progression from a baseline single-channel configuration to more complex multi-channel architectures provides clear insight into how increasing system complexity influences transmission behavior, performance limitations, and sensitivity to impairments. In addition, the integration of post-quantum digital signatures (PQDS) introduces controlled payload overhead, enabling direct evaluation of its impact across different system configurations. This facilitates the analysis of cross-layer interactions between cryptographic processing and physical-layer transmission characteristics. System performance is evaluated using key physical-layer metrics, including bit error rate (BER), Q-factor, and transmission reach. These metrics provide a comprehensive assessment of the trade-offs between spectral efficiency, transmission reliability, and security overhead in coherent optical systems.

4.2.1. Single-Channel Optical System

A single-channel coherent optical system is employed as a baseline configuration to evaluate the impact of PQDS-induced overhead under controlled transmission conditions. This configuration enables isolation of physical-layer effects while maintaining a simplified and well-defined system structure. The system architecture, illustrated in Figure 4 for QPSK and Figure 5 for QPSK and 16QAM respectively, consists of the following main components.

The main components of the single-channel system are summarized as follows:

• User-Defined Bit Sequence Generator—Generates the input binary data sequence representing the transmitted payload. This sequence can be adapted to simulate realistic data streams with varying structure and size.
• Coherent Optical Transmitter (QPSK/16QAM)—Maps the input bit sequence into modulation symbols (2 bits/symbol for QPSK and 4 bits/symbol for 16QAM) and modulates the optical carrier at a wavelength of 1550 nm.
• Optical Transmission Link—Models signal propagation through a single-mode fiber (SMF). Chromatic dispersion is compensated using dispersion-compensating fiber (DCF), ensuring signal integrity over long distances.
• Optical Coherent Receiver and Local Oscillator (LO)—Performs coherent detection using a local oscillator, enabling recovery of both amplitude and phase information from the received signal.
• Universal DSP Block—Executes digital signal processing operations, including filtering, synchronization, dispersion compensation, and carrier recovery.
• M-ary Threshold Detector—Performs symbol decision according to the selected modulation format.
• Decoder (QPSK/16QAM Decoder)—Converts detected symbols back into binary data.
• Performance Evaluation Unit—Computes key performance metrics, including BER and Q-factor, and supports constellation analysis for signal quality assessment.

This baseline configuration provides a controlled environment for isolating the effects of PQDS-induced overhead, serving as a reference for comparison with multi-channel WDM systems.

The two systems shown in Figure 4 and Figure 5 where operate with the parameter values listed in

Table 1. Simulation parameters of the optical communication system.

Parameter	Value
Transmitter
Reference Wavelength (nm)	1550
Linewidth (MHz)	0.1
Polarization type	Single
Power (dBm)	−30–30
Fiber Attenuation (dB/km)	0.2
Dispersion (SMF) (ps/nm/km)	16
Dispersion (DCF) (ps/nm/km)	−80
Gain (dB)	20
Noise figure	4
Sequence length	8192
Modulation type	QPSK/16QAM
Filter type	Gaussian

. Unless otherwise stated, other parameter values were used in the simulation.

4.2.2. 16-Channel WDM System

The Wavelength Division Multiplexing (WDM) system in Figure 5—the 16-channel QPSK (Quadrature Phase Shift Keying)—is an example of how the basic single-channel structure can be extended to this type of multi-channel structured system, which provides a practical representation of real optical network environments. In addition, the multiple-channel architecture allows multiple channels to be transmitted simultaneously, providing realistic deployment scenarios. Some of the principal enhancements over the single channel include:

• Multi-wavelength transmission employs 16 independent channels ($\lambda$1–$\lambda$16), each carrying distinct PQC-encoded data streams concurrently.
• Multiplexing/Demultiplexing Modules: These modules utilize wavelength-selective components, such as Arrayed Waveguide Gratings, to combine and separate channels, thereby assessing inter-channel crosstalk and bandwidth performance.
• Channel-Specific Dispersion Compensation: Dispersion Compensating Fiber (DCF) modules are deployed for each channel to counteract.
• Parallel Bit Error Rate (BER) analysis uses specific tools for each channel to measure performance, both individually and together. This helps to identify problems in the implementation of multi-channel Post-Quantum Cryptography (PQC).

Figure 6 was extended to 16channel-16QAM by replacing the modulation and demodulations to 16QAM. Both systems in 16channel-QPSK/16QAM were operated with the parameter values listed in

Table 2. WDM system simulation parameters.

Parameter	Value
Wavelength (nm)	1550–1556
Channel spacing	0.8–0.4–0.2
Linewidth (MHz)	0.1
Fiber Attenuation (dB/km)	0.2
Dispersion (SMF) (ps/nm/km)	16
Dispersion (DCF) (ps/nm/km)	−80
Gain (dB)	20
Noise figure of reference wavelength	4
Number of channels	16
MUX bandwidth (GHz)	160
Sequence length	8192
Constellation type (QAM)	Square

. Unless otherwise stated, other parameter values were used in the simulation.

Multiple parameters from Table 1 and Table 2 were sweeps conducted to evaluate the system’s performance and determine the optimal operating conditions. Initially, Tx/Rx sweeps were performed to identify the optimal operating points. Subsequently, these optimized values were then used to evaluate the system transmission reach, ensuring fairness and consistency across all tested scenarios. Unless otherwise specified in Table 1 and Table 2, all other parameters of the components of Optisystem were kept at their default settings.

5. Results and Discussion

This section presents a comprehensive analysis of the simulation results obtained to evaluate the performance of the proposed PQDS-enabled coherent wavelength-division multiplexing (WDM) system. The evaluation focuses on key physical-layer performance metrics, including bit error rate (BER), Q-factor, and constellation behavior, under varying system conditions. The analysis is conducted across multiple modulation formats and WDM configurations to systematically investigate the impact of system complexity and security-induced overhead. In particular, the study examines how the integration of PQDS affects transmission performance in terms of spectral efficiency, signal integrity, and transmission reach. Furthermore, the results are analyzed to identify key performance trade-offs associated with modulation scaling, channel density, and payload expansion. This enables a deeper understanding of the interaction between cryptographic overhead and physical-layer constraints in coherent optical systems. A detailed discussion is provided for each configuration, highlighting the observed performance trends and the extent to which PQDS integration influences overall system behavior.

Although the simulations are conducted under idealized OptiSystem conditions, the observed performance trends remain physically consistent with practical optical communication systems. In real-world deployments, hardware impairments such as phase noise, component mismatches, and higher-order nonlinear effects may introduce additional performance degradation. To account for this, a practical performance margin can be considered when interpreting the results. Importantly, such impairments primarily affect absolute performance levels rather than relative system behavior. Therefore, the comparative trends observed in this study—such as the performance gap between QPSK and 16QAM, as well as between single-channel and WDM configurations—are expected to remain valid under realistic operating conditions.

In addition, a sensitivity-oriented evaluation is implicitly incorporated through systematic parameter variation (e.g., launch power, channel spacing, and transmission distance), providing a margin-based understanding of system robustness under different operating conditions.

Furthermore, the impact of PQDS-induced payload expansion is fundamentally structural, as it directly influences sequence length and transmission duration. As a result, its effect on system performance is largely independent of hardware-specific imperfections, reinforcing the general validity of the conclusions drawn in this work.

The optical transmission channel was modeled using a standard single-mode fiber (SMF) with practical parameter settings. The reference wavelength was set to $1552.5\mathrm{nm}$, with a fiber attenuation of $0.2\mathrm{dB}/\mathrm{km}$. Chromatic dispersion was defined as $16.75\mathrm{ps}/\mathrm{nm}/\mathrm{km}$ with a dispersion slope of $0.075\mathrm{ps}/{\mathrm{nm}}^2/\mathrm{km}$, while the group velocity dispersion parameter (${\beta }_2$) was set to $-20{\mathrm{ps}}^2/\mathrm{km}$. Nonlinear effects were characterized using a nonlinear refractive index ($n_2$) of $2.6\times {10}^{-20}{\mathrm{m}}^2/\mathrm{W}$ and an effective area of 80 μm². The differential group delay was set to $0.2\mathrm{ps}/\mathrm{km}$. Raman scattering effects were modeled using a fractional Raman contribution of $0.18$ and an orthogonal Raman factor of $0.75$. The maximum nonlinear phase shift was limited to $3\mathrm{mrad}$, and periodic boundary conditions were applied in the simulation.

5.1. Key Generation and Signing Time at the Transmitter Side

This section evaluates the computational overhead associated with key generation and signing processes at the transmitter side of the proposed PQDS-enabled coherent WDM system, with emphasis on its impact on latency-sensitive optical communication performance. The results demonstrate clear variation across PQDS schemes, primarily driven by their underlying cryptographic structures. Hash-based schemes, such as SPHINCS+, exhibit significantly higher latency due to their reliance on hierarchical hash-tree constructions and large key sizes, resulting in substantial processing overhead. In contrast, lattice-based schemes, including FALCON-512 and FALCON-1024, achieve significantly lower latency, making them more suitable for high-speed coherent optical systems. More importantly, the results reveal that cryptographic latency extends beyond computational cost and directly influences system-level performance. In multi-channel WDM environments, increased key generation time introduces buffering and synchronization overhead, reducing framing efficiency and limiting overall system throughput. This effect becomes more pronounced as system complexity increases, highlighting the cumulative impact of parallel processing across multiple channels. These findings demonstrate that PQDS selection must consider system-level constraints rather than purely cryptographic performance. In particular, computationally efficient schemes such as FALCON provide a more suitable balance between security and transmission efficiency in high-capacity optical systems. Figure 7 illustrates the comparative key generation performance across PQDS schemes, clearly highlighting the scalability limitations of computationally intensive approaches.

These results confirm that key generation latency varies significantly across PQDS schemes and plays a critical role in system-level performance, particularly in high-speed optical communication environments. A detailed evaluation of signature generation time reveals clear performance differences among the examined PQDS schemes. In general, an increase in payload size leads to longer signing times across all algorithms, as illustrated in Figure 8. However, the extent of this increase varies significantly depending on the underlying cryptographic structure. Hash-based schemes, particularly SPHINCS+, exhibit the highest latency due to their reliance on complex hash-tree constructions, resulting in considerable computational overhead. In contrast, lattice-based schemes demonstrate more efficient performance. CRYSTALS-Dilithium provides a balanced trade-off between efficiency and security, while FALCON achieves the lowest signing latency, with FALCON-512 outperforming other variants. These results indicate that signing latency is a critical factor in system-level performance. In high-throughput optical systems, excessive signing time can introduce processing delays that impact overall transmission efficiency. Based on these observations, FALCON-512 is selected for subsequent transmission analysis, as it provides an optimal balance between computational efficiency and system compatibility, enabling clearer evaluation of cross-layer interactions with minimal overhead, illustrated in Figure 8.

The variation in time is primarily determined by the underlying cryptographic structure, security level, and file size of each PQDS scheme. Lattice-based schemes (e.g., FALCON and Dilithium) exhibit lower latency due to efficient algebraic operations, whereas hash-based schemes (e.g., SPHINCS+) incur significantly higher computational cost due to their iterative hash-based construction.

When combined with key generation results, it is evident that overall cryptographic latency is dominated by the choice of PQDS scheme, with lattice-based approaches offering a significantly lower computational overhead compared to hash-based alternatives.

5.2. Optical Transmitter Optimization

This section analyzes the impact of transmitter operating conditions on the performance of PQDS-enabled coherent WDM systems, focusing on the interplay between spectral efficiency, launch power, and nonlinear impairments. The results in Figure 9 confirm that system performance is governed by a fundamental trade-off between noise-limited and nonlinearity-limited regimes. At low launch power levels, performance is constrained by insufficient optical signal-to-noise ratio (OSNR), leading to reduced Q-factor. As launch power increases, performance improves until an optimal operating point is reached. Beyond this point, nonlinear effects such as self-phase modulation (SPM) and cross-phase modulation (XPM) dominate, causing signal distortion and performance degradation. This trade-off becomes more critical with increasing system complexity. In multi-channel WDM systems, the total transmitted power increases with the number of channels, intensifying nonlinear interactions. Consequently, the optimal launch power shifts toward lower values as both channel count and modulation order increase.

The relationship between spectral efficiency and performance follows Shannon capacity principles, where higher spectral efficiency requires higher SNR. As a result, higher-order modulation formats become increasingly sensitive to channel impairments. While 16QAM enables higher data rates, its reduced constellation spacing significantly limits robustness compared to QPSK. All configurations exhibit a clear optimal operating point at approximately 40 Gb/s. Below this value, inefficient bandwidth utilization leads to reduced Q-factor, while operating beyond this point results in performance degradation due to chromatic dispersion and inter-symbol interference (ISI). Quantitatively, the single-channel QPSK system achieves the highest Q-factor (≈25), which decreases to ≈21 in the 16-channel WDM QPSK configuration due to inter-channel interference, and further drops to ≈5 in the 16-channel WDM 16QAM system due to the combined effects of high modulation order and dense WDM operation. The optimal launch power also varies significantly across configurations. For single-channel QPSK, the optimal value is approximately 15 dBm. In contrast, it shifts to approximately −7 dBm per channel for 16-channel WDM QPSK and further decreases to approximately −10 dBm for 16-channel WDM 16QAM. This shift reflects the increasing dominance of nonlinear effects as total transmitted power increases. Importantly, the results demonstrate that PQDS-induced payload overhead amplifies system sensitivity by increasing transmission duration and symbol density, which accelerates the accumulation of nonlinear impairments and reduces performance margins. These findings highlight that transmitter optimization in PQDS-enabled coherent systems requires a cross-layer approach, where modulation format, launch power, and cryptographic overhead must be jointly considered to achieve an optimal balance between spectral efficiency and transmission reliability.

The effect of input signal power is illustrated in Figure 10. The results indicate that the optimal operating point varies significantly across different system configurations due to the interplay between noise and nonlinear impairments. For the single-channel QPSK system, optimal performance is achieved at a launch power of approximately $15\mathrm{dBm}$, where the Q-factor reaches its maximum value. In contrast, for the 16-channel WDM QPSK configuration, the optimal launch power shifts to approximately $-7\mathrm{dBm}$ per channel. This shift is primarily attributed to the increase in nonlinear effects as the number of channels increases $\left(N\right)$ [35,36], where the total optical power can be expressed as $P_{\mathrm{tot}}={\sum }_{i=1}^N{P}_{\mathrm{ch}}$. As total power increases, nonlinear interactions such as cross-phase modulation (XPM) and four-wave mixing (FWM) become more pronounced, leading to performance degradation. For the 16-channel WDM 16QAM system, the optimal launch power is further reduced to approximately $-10\mathrm{dBm}$ per channel. This reflects the higher sensitivity of high-order modulation formats to both noise and nonlinear impairments. At low launch power levels, system performance is limited by insufficient signal-to-noise ratio (SNR), resulting in a reduced Q-factor. As the launch power increases, the Q-factor improves until it reaches an optimal point. Beyond this point, further increases in launch power intensify nonlinear effects, which degrade signal quality and lead to a reduction in Q-factor. These results highlight the critical role of launch power optimization in coherent WDM systems and demonstrate the trade-off between noise-limited and nonlinearity-limited operating regimes.

5.3. Optical Receiver and LO Power Analysis

At the receiver side, coherent detection is performed using photodiodes that combine the received optical signal with a local oscillator (LO) signal, as described in Equation (6). This mixing process enables the extraction of both amplitude and phase information from the received signal.

The detected photocurrent is proportional to the square root of the product of the signal power and the LO power, and can be expressed as I $\propto \sqrt{P_s{P}_{LO}}$, where $P_s$ is the signal power and $P_{LO}$ is the local oscillator power.

Increasing the LO power enhances the efficiency of the mixing process, effectively improving signal detection and increasing the signal-to-noise ratio (SNR) at the receiver. However, the results indicate that this improvement is limited.

As shown in Figure 11, system performance improves with increasing LO power up to a certain threshold value. Beyond this point, the performance saturates and no further improvement in Q-factor is observed.

This behavior can be explained by the transition of the receiver into a shot-noise-limited regime. Once the LO power becomes sufficiently high, the dominant noise source shifts to shot noise, as described in Equation (7). In this regime, further increases in LO power do not significantly enhance system performance, as the Q-factor becomes primarily constrained by signal power and transmission impairments rather than receiver sensitivity.

These findings highlight that an optimal LO power exists for coherent receivers, beyond which additional power does not yield performance gains. Therefore, careful optimization of LO power is required to balance detection efficiency and noise limitations in coherent WDM systems.

The selection of the receiver filter represents a critical design consideration in coherent optical systems, as the filter response directly influences noise shaping and, consequently, bit error rate (BER) performance. To evaluate this effect, three receiver filter types, Gaussian, Bessel, and Chebyshev, are analyzed using BER versus optical signal-to-noise ratio (OSNR) for both QPSK and 16QAM modulation formats. The results demonstrate the expected inverse relationship between OSNR and BER, where increasing OSNR leads to improved signal quality and reduced BER across all configurations. However, the sensitivity to noise varies significantly between modulation formats. For QPSK, the Gaussian filter achieves the lowest BER, indicating superior noise suppression capability compared to the Bessel and Chebyshev filters. In contrast, the 16QAM modulation format exhibits consistently higher BER values across all OSNR levels, reflecting its increased sensitivity to noise due to the reduced spacing between constellation points. Despite these observations, the performance differences among the evaluated filter types remain relatively small when compared to the impact of modulation format. This indicates that, while filter selection contributes to performance optimization, the dominant factor governing system robustness is the modulation scheme rather than the specific filter response. These findings suggest that receiver filter design should be considered as a secondary optimization parameter, with primary emphasis placed on modulation format and system operating conditions in PQDS-enabled coherent optical systems (Figure 12).

It is evident that variations in receiver filter type do not significantly affect BER performance, indicating that system behavior is primarily governed by modulation characteristics rather than filtering.

5.4. Optical System Evaluation

The performance of the proposed PQDS-enabled optical system is evaluated using OptiSystem 15 to analyze the impact of different PQDS schemes on transmission performance and spectral efficiency. The evaluation focuses on quantifying how security-induced overhead influences key system-level characteristics. The analysis considers several critical performance metrics, including maximum transmission reach, optimal channel spacing, and the number of supported WDM channels. These metrics provide a comprehensive assessment of system capacity, efficiency, and scalability under varying operating conditions. To ensure a fair and consistent comparison, three representative system configurations are evaluated under identical conditions: single-channel QPSK, 16-channel WDM QPSK, and 16-channel WDM 16QAM. This structured approach enables a systematic investigation of the combined effects of modulation format, channel scaling, and PQDS integration on overall system performance.

5.4.1. PQD Signature Algorithms in Single-Channel Coherent Systems

The impact of PQDS-induced payload expansion on transmission time is evaluated as a function of modulation format, system configuration, and data sequence length, as shown in Figure 13. The results demonstrate a consistent increase in transmission time with increasing data size across all configurations, confirming the direct influence of payload expansion on system throughput. For the single-channel QPSK system, transmission time increases from approximately 0.58 ms to 248 ms as the sequence length grows, indicating strong sensitivity to payload size. This effect becomes more pronounced in the 16-channel WDM QPSK configuration, where transmission time increases from 38 ms to 740 ms due to the combined impact of payload expansion and multi-channel processing overhead. In contrast, the 16-channel WDM 16QAM system achieves lower transmission times (15 ms to 580 ms) for equivalent data sizes, primarily due to its higher spectral efficiency, which enables more bits per symbol and reduces transmission duration. However, this improvement is accompanied by reduced robustness. The results highlight a fundamental trade-off between transmission efficiency and system reliability: while 16QAM reduces transmission time, it is significantly more sensitive to noise and channel impairments, whereas QPSK provides more stable performance under increased payload conditions. Importantly, the results demonstrate that PQDS-induced payload overhead directly affects transmission efficiency and cannot be neglected in system design. This confirms that performance evaluation in quantum-safe optical systems must consider both transmission time and reliability metrics jointly, rather than treating them as independent factors.

All simulations were conducted using OptiSystem 15 on a workstation equipped with an Intel^® Core™ i7-10750H processor (2.60 GHz) and 40 GB RAM, running Windows 10 Home (64-bit). The simulation environment was kept consistent across all experiments to ensure fair comparison and reproducibility of results.

It is important to note that the observed variations in computational time across PQDS schemes (as shown in Figure 7) are influenced by both the underlying algorithmic complexity and the associated file size (payload length). Schemes with higher computational complexity and larger parameter sizes require more processing time, which directly affects the overall simulation duration under a fixed computational environment.

Furthermore, the reported latency results are based on software-level execution and do not account for hardware acceleration. In practical deployments, hardware implementations such as FPGAs and optimized DSP platforms can significantly reduce cryptographic latency through parallel processing and dedicated arithmetic units. As a potential direction for future work, hardware acceleration approaches (e.g., FPGA- or DSP-based implementations) may be considered to enhance computational efficiency and reduce latency.

These results clearly demonstrate that PQDS overhead introduces a direct trade-off between security and transmission efficiency, particularly in high-capacity WDM systems.

5.4.2. Impact of PQDS Overhead on Transmission Reach

The impact of PQDS-induced overhead on transmission reach is evaluated using QPSK and 16QAM modulation formats under both single-channel and 16-channel WDM configurations, while maintaining a constant payload size. This controlled setup ensures that variations in performance are directly attributed to cryptographic overhead rather than data size. Transmission reach, defined as the maximum achievable distance between transmitter and receiver, is analyzed across different PQDS schemes, as shown in Figure 14. All simulations are conducted using a fixed data sequence length of 8144 bits, enabling a fair comparison in which differences in reach are primarily governed by signature size and resulting payload expansion. The results demonstrate a clear dependency between PQDS structure and transmission reach. Lattice-based schemes consistently achieve superior performance, with FALCON-512 and FALCON-1024 reaching approximately 204 km and 200 km, respectively. This advantage is attributed to their compact signature sizes, which limit payload expansion and reduce the number of transmitted symbols, thereby mitigating the accumulation of noise, chromatic dispersion, and nonlinear impairments. In contrast, CRYSTALS-Dilithium achieves moderate transmission reach (approximately 180–182 km), reflecting a balance between security strength and system efficiency. Its larger signature size increases transmission duration, leading to higher susceptibility to physical-layer impairments. Hash-based schemes, particularly SPHINCS+, exhibit significantly reduced transmission reach, with a maximum distance of approximately 76 km. This degradation is primarily due to substantial payload expansion, which increases symbol transmission and accelerates the accumulation of channel impairments, resulting in rapid performance degradation. These results confirm that PQDS-induced payload overhead is a dominant factor in determining transmission reach and cannot be neglected in system design. More importantly, they demonstrate that cryptographic parameters directly influence physical-layer limits, establishing a clear dependency between security mechanisms and transmission performance. This finding establishes that transmission reach in quantum-safe coherent optical systems is not solely constrained by physical-layer impairments, but is also fundamentally limited by cryptographic overhead.

This confirms that PQDS-induced payload expansion directly limits transmission reach, particularly for schemes with large signature sizes. Overall, the results highlight that the choice of PQDS scheme plays a decisive role in determining both transmission efficiency and achievable communication distance in secure optical systems.

5.4.3. Optimize Channel Spacing and Channels Capacity

Channel spacing is a key design parameter in coherent wavelength-division multiplexing (WDM) systems, directly affecting both transmission reach and spectral efficiency. While reducing channel spacing increases channel density and improves spectrum utilization, it also intensifies inter-channel interference (ICI) and nonlinear effects, leading to a fundamental trade-off between system capacity and transmission performance. To analyze this trade-off, three channel spacing values (0.2 nm, 0.4 nm, and 0.8 nm) are evaluated for a 16-channel QPSK WDM system, as shown in Figure 15. The results demonstrate that transmission reach is highly sensitive to channel spacing. At 0.2 nm spacing, the system achieves a limited transmission reach of approximately 95 km due to strong nonlinear interactions, including cross-phase modulation (XPM) and four-wave mixing (FWM), as well as increased spectral overlap between adjacent channels. As the channel spacing increases, nonlinear coupling is reduced, leading to improved signal integrity. Consequently, the transmission reach increases to approximately 194 km at 0.4 nm and further to around 200 km at 0.8 nm. However, this improvement comes at the expense of spectral efficiency, as wider spacing reduces the number of channels that can be supported within a fixed bandwidth. Based on these results, a channel spacing of 0.4 nm represents an optimal operating point, achieving a balanced trade-off between transmission reach and spectral efficiency. Importantly, the results demonstrate that channel spacing optimization cannot be treated as a purely capacity-driven parameter. Instead, it must be considered as a system-level design factor that accounts for nonlinear effects, inter-channel interactions, and the additional constraints introduced by PQDS-induced payload overhead. These findings confirm that optimal channel spacing in quantum-safe WDM systems is governed not only by spectral efficiency considerations, but also by the combined impact of nonlinear impairments and cryptographic overhead.

The channel spacing plays a crucial role in balancing spectral efficiency and transmission performance in WDM systems. The impact of channel spacing becomes more critical in PQDS-enabled systems, where additional payload overhead further amplifies nonlinear effects.

5.4.4. System Evolution

This section analyzes the combined impact of channel configuration, modulation format, and transmission distance on system performance using BER and Q-factor as functions of fiber length for three configurations: single-channel QPSK, 16-channel WDM QPSK, and 16-channel WDM 16QAM. The results show a consistent degradation in signal quality with increasing transmission distance due to accumulated attenuation, chromatic dispersion, amplified spontaneous emission (ASE) noise, and fiber nonlinearities. This leads to an increase in BER and a corresponding reduction in Q-factor across all configurations. Among the evaluated systems, single-channel QPSK achieves the longest transmission reach (approximately 203 km), demonstrating strong robustness to both linear and nonlinear impairments. This reach decreases to approximately 194 km in the 16-channel WDM QPSK system due to inter-channel interference (ICI) and nonlinear effects such as cross-phase modulation (XPM). The degradation becomes significantly more pronounced in the 16-channel WDM 16QAM configuration, where transmission reach is reduced to approximately 103 km, reflecting the high sensitivity of dense constellations to noise, dispersion, and nonlinear impairments. These results confirm the fundamental trade-off between spectral efficiency and transmission reach. While higher-order modulation formats and multi-channel WDM configurations increase system capacity, they also reduce robustness to channel impairments, limiting achievable transmission distance. Importantly, this trade-off becomes more critical in PQDS-enabled systems, where payload expansion increases transmission duration and accelerates impairment accumulation. As a result, higher spectral efficiency does not necessarily translate into improved system performance under quantum-safe constraints. The results presented in Figure 16 and summarized in

Table 3. Maximum reach and optimum operating points.

No.	System Configuration	Bit Rate	Max. Distance	Optimum Operating Point
1	Single-channel QPSK	40 Gb/s	203 km	Tx = 15 dBm, Rx/LO = 5 dBm
2	16-channel WDM QPSK	40 Gb/s	194 km	Tx = −7 dBm, Rx/LO = 15 dBm
3	16-channel WDM 16QAM	40 Gb/s	103 km	Tx = −10 dBm, Rx/LO = 10 dBm

quantitatively demonstrate this behavior, highlighting that optimal system design requires careful selection of modulation format, channel configuration, and operating conditions to balance capacity, reliability, and security overhead.

The results presented in Figure 16 quantitatively validate the fundamental trade-off between spectral efficiency and transmission reach in coherent optical communication systems. High-order modulation formats, such as 16QAM, achieve increased spectral efficiency by transmitting more bits per symbol; however, this advantage comes at the cost of reduced transmission distance due to their heightened sensitivity to noise, chromatic dispersion, and nonlinear impairments. In contrast, lower-order modulation schemes such as QPSK provide greater robustness against channel impairments, enabling longer transmission distances, albeit with lower spectral efficiency. This inherent trade-off underscores the importance of selecting modulation formats based on system requirements, particularly in long-haul optical communication scenarios. Furthermore, when combined with PQDS-induced payload overhead, this trade-off becomes more critical, as higher spectral efficiency does not necessarily translate into improved overall system performance under constrained transmission conditions. The maximum achievable transmission distances for all evaluated configurations, operating at a data rate of 40 Gb/s and employing FALCON-512, are summarized in Table 3, providing a comparative overview of system performance across different modulation formats and WDM configurations.

The overall findings confirm that transmission distance decreases progressively with increasing modulation order, system complexity, and spectral efficiency, highlighting the need for a careful balance between throughput, channel count, and impairment tolerance in practical long-haul WDM system design.

The system capacity at a transmission distance of 70 km is evaluated for different modulation formats and WDM configurations, as summarized in

Table 4. Achievable system capacity at fixed transmission distance.

No.	System Configuration	Distance	Total Capacity	Optimum Operating Point
1	Single-channel QPSK	70 km	40 Gb/s	Tx = 15 dBm, Rx/LO = 5 dBm
2	16-channel WDM QPSK	70 km	640 Gb/s	Tx = −7 dBm, Rx/LO = 15 dBm
3	16-channel WDM 16QAM	70 km	1280 Gb/s	Tx = −10 dBm, Rx/LO = 10 dBm

. The results clearly demonstrate the relationship between modulation scheme, channel scaling, and achievable system capacity. QPSK transmits 2 bits per symbol, whereas 16QAM transmits 4 bits per symbol, resulting in a twofold increase in spectral efficiency for 16QAM under equivalent operating conditions. Consequently, higher-order modulation formats enable significantly greater data throughput compared to lower-order schemes. The highest overall system capacity is achieved using the 16-channel WDM configuration with 16QAM modulation. This is attributed to the combined effect of increased bits per symbol and parallel transmission across multiple wavelength channels, which together maximize spectral utilization and aggregate data rate. However, this increase in system capacity is accompanied by reduced robustness. Higher-order modulation formats exhibit increased sensitivity to noise, dispersion, and nonlinear impairments, particularly in long-haul transmission scenarios. Therefore, while 16QAM provides superior capacity, its practical deployment requires careful system design and optimization to maintain acceptable performance levels. Furthermore, when PQDS-induced overhead is considered, the effective system capacity may be further constrained due to payload expansion. This highlights the importance of jointly optimizing modulation format, channel configuration, and security mechanisms to achieve a balanced trade-off between capacity, reliability, and quantum-safe operation.

Table 5. Comparative analysis of system configurations under PQDS integration.

Parameter	Single-Channel QPSK	16-Ch WDM QPSK	16-Ch WDM 16QAM
Spectral Efficiency	Low	Medium	High
Maximum Reach (km)	203	194	103
Q-factor for signal power	High (∼25)	Moderate (∼21)	Low (∼5)
BER Performance	Lowest	Moderate	Highest
Sensitivity to Impairments	Low	Medium	High
Impact of PQDS Overhead	Low	Moderate	High
System Complexity	Low	Medium	High
Inter-channel Interference	Low	Moderate	High
Robustness	High	Moderate	Low
Best Use Case	Long-haul links	Balanced	High-capacity
Transmission Time	58 s	38 min	15 min

provides a cross-comparative summary of the evaluated system configurations, highlighting the trade-offs between spectral efficiency, transmission reach, robustness, and sensitivity to PQDS-induced overhead.

5.5. Design Rules and Practical Implications

Based on the obtained results, several key design guidelines can be derived for PQDS-enabled coherent optical systems:

• QPSK for long-haul robustness: QPSK consistently achieves the highest transmission reach and provides superior tolerance to noise, chromatic dispersion, and nonlinear impairments. This makes it the preferred choice for long-haul and reliability-critical applications where stable performance is required.
• 16QAM for capacity-driven deployments: 16QAM offers higher spectral efficiency by transmitting more bits per symbol, enabling increased system capacity. However, its reduced constellation spacing makes it significantly more sensitive to noise and nonlinear effects, limiting its suitability to metro and short-reach scenarios.
• WDM configuration as a critical design parameter: Channel spacing and channel count play a central role in system performance. Increasing channel density improves spectral efficiency but intensifies inter-channel interference (ICI) and nonlinear impairments. Therefore, optimal WDM design requires a balance between capacity and transmission robustness.
• Cross-layer optimization with PQDS integration: PQDS-induced payload expansion increases transmission duration, processing overhead, and latency, thereby amplifying physical-layer impairments. As a result, cryptographic parameters such as signature size must be treated as adaptive design variables rather than fixed constraints. Optimal system performance requires joint optimization of modulation format, launch power, channel configuration, and PQDS scheme selection within a unified cross-layer framework.

6. Implications and Recommendations

The obtained results provide several practical design insights for PQDS-enabled coherent optical communication systems:

• QPSK for long-haul and high-reliability links: QPSK demonstrates superior robustness against both linear and nonlinear impairments in single-channel and WDM configurations. The results show that QPSK can achieve transmission distances of up to approximately 203 km at 40 Gb/s, providing a high performance margin. This makes it well-suited for long-haul communication and reliability-critical applications.
• 16QAM for capacity-driven metro deployments: 16QAM enables higher spectral efficiency by transmitting more bits per symbol, thereby increasing system capacity. However, its reduced constellation spacing significantly increases sensitivity to noise, chromatic dispersion, and nonlinear effects. As a result, the 16-channel WDM 16QAM system achieves a reduced transmission reach of approximately 103 km at 40 Gb/s, making it more suitable for metro and short-reach scenarios.
• WDM channel spacing and capacity trade-off: Channel spacing plays a critical role in determining both system capacity and transmission performance. Reducing channel spacing improves spectral efficiency but intensifies inter-channel interference (ICI) and nonlinear effects. The results show that increasing spacing from 0.2 nm to 0.8 nm improves transmission reach from approximately 95 km to 200 km. Therefore, an intermediate spacing of 0.4 nm provides a balanced trade-off between spectral efficiency and transmission robustness.
• PQDS-aware cross-layer design (payload–modulation alignment): PQDS-induced payload expansion directly affects sequence length, transmission duration, and processing overhead. While higher-order modulation reduces the number of symbols, the overall bit sequence length increases with payload size, leading to longer transmission times and greater exposure to physical-layer impairments. Therefore, payload size and sequence length should be treated as adaptive design parameters jointly optimized with modulation format and system configuration. This cross-layer alignment is essential to minimize latency and maintain stable performance in quantum-safe coherent WDM systems.

7. Limitations

Despite providing a comprehensive analysis of PQDS-aware coherent WDM transmission, this study has several limitations that should be acknowledged. First, the simulations are conducted under idealized conditions, where transmitter, receiver, and digital signal processing (DSP) modules operate using default or near-ideal configurations within the OptiSystem environment. As a result, practical implementation challenges such as hardware imperfections, component mismatches, and calibration inaccuracies are not fully captured. This study is based on simulation results obtained under controlled OptiSystem conditions, which may not fully capture all practical hardware imperfections such as phase noise, component mismatch, and higher-order nonlinear effects. Although a wide range of operating parameters has been explored to reflect diverse system conditions, further work is required to incorporate hardware-aware modeling and experimental validation to enhance real-world applicability. Second, certain physical-layer effects, including polarization-related impairments and higher-order nonlinear phenomena, are not explicitly modeled. In real-world deployments, these factors can introduce additional performance degradation, particularly in dense WDM systems operating near their performance limits. Third, the cryptographic evaluation focuses primarily on computational delay and payload overhead, without considering hardware acceleration techniques or platform-specific optimizations. In practical systems, such factors can significantly influence latency, throughput, and overall system efficiency. Finally, the evaluation of cryptographic latency in this study is limited to algorithmic computational delay, without explicitly considering hardware acceleration techniques such as FPGA-based implementations or optimized digital signal processing (DSP) architectures. In practical deployments, such optimizations can significantly reduce signing and verification latency, thereby mitigating the computational overhead associated with PQDS schemes. However, it is important to note that while hardware acceleration can improve processing speed, it does not eliminate the inherent transmission overhead caused by increased payload size. Therefore, the impact of PQDS on sequence length and transmission duration remains fundamentally unaffected. Future work will investigate hardware-aware implementations to provide a more comprehensive assessment of end-to-end system performance. Future work will extend this study toward more advanced transmission architectures, including coherent optical OFDM and elastic WDM systems. Additionally, incorporating hardware-aware PQDS implementations and more realistic system models will enable a deeper understanding of cross-layer optimization between security mechanisms and physical-layer performance. This will support the development of scalable and efficient quantum-safe optical communication systems.

8. Conclusions

This paper presented a comprehensive cross-layer evaluation of post-quantum digital signature (PQDS) integration within coherent wavelength-division multiplexing (WDM) high-order modulation optical communication systems. Unlike conventional approaches that treat cryptographic mechanisms and physical-layer performance independently, this work established a direct and quantitative link between PQDS-induced payload expansion and key transmission characteristics. Through a structured and systematic simulation framework, multiple system configurations, including single-channel QPSK, 16-channel WDM QPSK, and 16-channel WDM 16QAM, were analyzed under consistent operating conditions. The results demonstrate that PQDS-induced overhead has a measurable and non-negligible impact on system performance, directly influencing bit error rate (BER), Q-factor, transmission reach, and transmission efficiency. The findings reveal that while higher-order modulation formats and dense WDM configurations significantly enhance spectral efficiency, they simultaneously reduce system robustness due to increased sensitivity to noise, dispersion, and nonlinear effects. This trade-off becomes more critical in PQDS-enabled systems, where payload expansion increases transmission duration and amplifies the accumulation of physical-layer impairments. Furthermore, the comparative analysis of PQDS schemes shows that lattice-based approaches, particularly FALCON-512, provide a more suitable balance between computational efficiency and transmission performance due to their compact signature size and reduced overhead. In contrast, hash-based schemes such as SPHINCS+ introduce substantial performance degradation, significantly limiting transmission reach and efficiency. A key contribution of this work is the identification of critical operating regimes where PQDS integration has minimal impact, as well as regimes where performance degradation becomes dominant. These insights enable the formulation of practical design guidelines for selecting modulation formats, channel configurations, and PQDS schemes in quantum-safe optical communication systems. Overall, the results highlight that enabling secure, quantum-resistant optical communication requires a paradigm shift from isolated optimization toward integrated cross-layer co-design. By treating cryptographic overhead as a system-level design parameter, this work provides a structured framework for achieving a balanced trade-off between security, spectral efficiency, and transmission reliability.

Future work will extend this framework to hardware-aware implementations and advanced transmission architectures, including elastic WDM and coherent optical OFDM systems, to further investigate scalability and real-world deployment feasibility.