A Practical Implementation of Post-Quantum Cryptography for Secure Wireless Communication ^†

Abstract

Recent advances in quantum computing have prompted urgent consideration of the migration of classical cryptographic systems to post-quantum alternatives. However, it is impossible to fully understand the impact that migrating to current Post-Quantum Cryptography (PQC) algorithms will have on various applications without the actual implementation of quantum-resistant cryptography. On the other hand, PQC algorithms come with complexity and long processing times, which may impact the quality of service (QoS) of many applications. Therefore, PQC-based protocols with practical implementations across various applications are essential. This paper introduces a new framework for PQC standalone and PQC–AES (Advanced Encryption Standard) hybrid public-key encryption (PKE) protocols. Building on prior results, we focus on securing applications such as file transfer, video streaming, and chat-based communication using enhanced PQC-based protocols. The extended PQC-based protocols use a sequence number-based mechanism to effectively counter replay and man-in-the-middle attacks and mitigate standard cybersecurity attack vectors. Experimental evaluations examined encryption/decryption speeds, throughput, and processing overhead for the standalone PQC and the PQC–AES hybrid schemes, benchmarking them against traditional AES-256 in an existing client–server environment. The results demonstrate that the new approaches achieve a significant balance between security and system performance compared to conventional deployments. Furthermore, a comprehensive security analysis confirms the robustness and effectiveness of the proposed PQC-based protocols across diverse attack scenarios. Notably, the PQC–AES hybrid protocol demonstrates greater efficiency for applications handling larger data volumes (e.g., 10–100 KB) with reduced latency, underscoring the practical necessity of carefully balancing security and operational efficiency in the post-quantum migration process.

1. Introduction

The development of quantum computing is advancing rapidly, transforming theoretical concepts into reality and raising concerns about the security of classical cryptographic systems. Among recent breakthroughs, Equal1 has launched Bell-1: The First Quantum System Purpose-Built for the HPC Era [1], complementing advancements by companies such as IBM and Microsoft. Recent studies, such as Esmailiyan et al.’s work on CMOS position-based charge qubits [2] and Staszewski et al.’s development of cryogenic controllers for quantum dots [3], provide critical insights into the foundational technologies driving this revolution. It is anticipated that fully developed quantum computers will render classical cryptography insecure, potentially compromising all data encrypted using classical methods, including currently encrypted data. Therefore, addressing the imminent risks posed by traditional cryptographic algorithms has become increasingly urgent.

To address this issue and provide quantum-resistant cryptographic algorithms, the National Institute of Standards and Technology (NIST) introduced PQC algorithms [4,5,6], which will ensure that applications and network communications are secure against quantum attacks. However, it is not clear what impact or overhead will be introduced by migrating applications to be PQC-compliant, as PQC algorithms come with large ciphertext and key sizes. It is important to evaluate the practical implementation of PQC algorithms in real-world applications. Therefore, there is a need to investigate the performance trade-offs associated with the practical use of PQC algorithms and to address the unique requirements of emerging technologies such as 5G/6G and IoT. Furthermore, it remains uncertain how to ensure and meet their stringent quality-of-service (QoS) requirements.

NIST has selected four PQC algorithms [7] and released the standards for the selected PQC algorithms, which include one key encapsulation mechanism (KEM), CRYSTALS-Kyber (renamed to Module-Lattice-Based Key-Encapsulation Mechanism (ML-KEM) in the standards) [8], and three digital signature algorithms: CRYSTALS-Dilithium (Module-Lattice-Based Digital Signature Algorithm (ML-DSA)) [9]; FALCON (Fast-Fourier Transform over NTRU-Lattice-Based Digital Signature Algorithm (FN-DSA)), whose draft will be released in FIP206; and SPHINCS+ (Stateless Hash-Based Digital Signature Algorithm (SLH-DSA)) [10]. In addition, as a backup to the ML-KEM algorithm, NIST has selected HQC [11] for the next PQC KEM algorithm to be standardized among the four PQC algorithms in Round 4: BIKE, Classic McEliece, HQC, and SIKE. The draft standard for HQC is expected to be released later.

Additionally, while the NIST standard [8] provides detailed specifications for implementing the ML-KEM algorithm, including key generation, key encapsulation, and decapsulation, it does not finalize specific methods for securing application data. Instead, it recommends the application of established symmetric-key cryptographic techniques, as outlined in other NIST standards, for encrypting and decrypting data to protect against unauthorized access. Therefore, to fully understand the implications of integrating PQC algorithms into various applications, particularly those in IoT, 5G, and 6G systems with stringent QoS requirements, a dedicated PQC-based protocol for securing data transmission is essential.

This paper introduces new and improved PQC-based protocols that extend the PQC standalone and PQC–AES hybrid PKE protocols to secure the communication of application data transmitted between two parties and mitigate common cybersecurity attack vectors. Specifically, we introduce a sequence-number-based mechanism to the PQC-based protocols to prevent replay attacks and man-in-the-middle (MITM) attacks, providing a practical application of PQC algorithms in real-world scenarios. This offers a tangible example of how these quantum-resistant cryptographic methods can be applied to secure communication and their impact on applications.

In the PQC standalone mode, a secure communication channel is established between the sender and receiver through a PQC key encapsulation algorithm. The exchanged public key is used for encryption, while each party uses its secret key for decryption (asymmetric-key cryptography). On the other hand, in the hybrid mode, a two-way shared secret key between the client and server is used to generate a symmetric key (symmetric-key cryptography) as the encryption and decryption keys using a key derivation function (KDF). The encryption and decryption keys are then used by the client and server to encrypt and decrypt their files.

As stated in our prior work [12], the client generates a pair of PQC public and private keys, and the server generates its own keys in a similar manner. Using the PQC KEM, the public key is exchanged, establishing secure communication. Subsequently, the client encrypts its file using the encryption key, based on the protocol being used (PQC standalone or PQC–AES hybrid). Upon receiving the encrypted file, the server decrypts it using the decryption key to access the content. This process ensures end-to-end secure communication between the two parties (i.e., the client and the server). Furthermore, we devise various use-case scenarios for implementing the proposed PQC-based protocols in real-world applications. To integrate these protocols with pre-existing applications, they are designed to mitigate replay and MITM attacks, enhancing connection security. This ensures secure and seamless end-to-end communication between the two parties using the process defined by the proposed protocols. This improvement demonstrates the feasibility of these protocols for real-world secure communications over wireless networks while minimizing their impact on application use cases and network performance.

The goal of our proposed protocols is to enable the practical adoption of the PQC algorithm in real-world applications, balancing its impact on use cases and network performance. Our approach could also benefit 5G and 6G development by guiding the transition to PQC, particularly as there is growing consideration for shifting 5G security to a PKI-based trust model. The proposed protocols adopt the standardized algorithms by NIST, as in [8,9,10], and are designed to facilitate a smooth transition of applications from classical cryptography to quantum resistance utilizing both standalone PQC and PQC–AES hybrid protocols, thereby aligning with NIST’s PQC migration guidelines as stated in SP 800-208 [13].

This paper significantly extends our prior work [12], in which we proposed the first PQC-based and PQC–AES hybrid PKE protocols for wireless file transfer and conducted initial performance benchmarking. In this paper, we expand on that work by introducing a new framework that includes a sequence-number-based mechanism to defend against replay and MITM attacks, along with an expanded security evaluation against common cybersecurity threats. Moreover, we extend the protocols to support a wider range of applications, including chat-based communication, video streaming, and live streaming. We also evaluate system performance under more diverse and realistic conditions and analyze the impact of these protocols on QoS and operational efficiency. This detailed analysis, along with the additional protocol advancements, provides critical insights beyond what was reported in [12] and offers practical guidance for deploying PQC in next-generation communication systems.

The rest of this paper is organized as follows. Section 2 reviews related works on PKE and PQC. Section 3 describes the proposed PQC-based protocols, while Section 4 details the use cases and the implementation of the protocol in applications. In Section 5, we discuss the security analysis of the proposed PQC standalone and PQC–AES hybrid PKE protocols. Section 6 presents a performance evaluation of the proposed method, and Section 7 concludes this paper.

2. Related Works

There has been extensive work, as well as several ongoing works, on the implementation of the PQC algorithm and its variants. In this section, we review some of the existing works and explain the major difference between our paper and such research.

Numerous studies have explored the operation [14,15] and implementation of the PQC KEM algorithm across different hardware platforms. Additionally, most current implementations of the PQC algorithm focus on its integration with Secure Sockets Layer (SSL), Transport Layer Security (TLS) [16,17,18], and Secure Shell (SSH). Furthermore, several works on PKE have been conducted, and there is ongoing work on the implementation of PQC algorithms and their variants.

In [19], Singh et al. proposed a new multivariate public-key cryptosystem based on permutation p-polynomials over finite fields, focusing on the need for secure and efficient public-key systems that can withstand increasing computing power and quantum computing. Marco et al. [20] proposed a generic transformation that achieves post-quantum security for classical signature schemes by hiding the public key with a one-time use of the key pair. Meanwhile, da Silva Lima et al. [21] evaluated the efficiency of the Kyber KEM algorithm in a mobile application, specifically analyzing its performance on the x86 and ARM architectures.

In addition, surveys and reviews highlighting the issues and constraints of migrating applications to be PQC-compliant have been carried out. Giron [22] discussed the challenges and research efforts required for migrating applications to PQC due to the vulnerability of current public-key cryptography schemes to quantum computers. Liu et al. [23] surveyed the performance and optimization of PQC algorithms for Internet of Things (IoT) systems. They highlighted the computational cost challenges of PQC algorithms for resource-constrained IoT devices and reviewed recent proposals for optimization. Similarly, Asif [24] presented a comprehensive review of PQC for the IoT, covering both theoretical and practical aspects. Li et al. [25] reviewed PQC algorithms, focusing on the Kyber algorithm, and discussed the challenges and opportunities of post-quantum security. Balamurugan et al. [26] discussed the need for PQC in light of the threat quantum computing poses to classical encryption schemes.

Fakhruldeen et al. [27] provided a comprehensive introduction to the emerging field of quantum-resistant cryptography, emphasizing its critical importance in safeguarding wireless networks against quantum computing threats. In their paper, they highlighted the vulnerabilities of current cryptographic standards to quantum attacks and explored solutions such as PQC and Quantum Key Distribution (QKD), alongside practical migration strategies like hybrid approaches. They further underscored the role of global standardization efforts, particularly by NIST, in advancing PQC algorithms while addressing challenges in integrating these techniques into existing infrastructures.

Despite these varied approaches, the practical application of the PQC algorithm remains confined to the aforementioned protocols and has not expanded to other commonly used applications. Moreover, directly applying PQC algorithms to various applications and systems may lead to performance issues, as many of these have stringent QoS requirements. Therefore, to fully assess the impact of direct PQC algorithm usage on application scenarios within a wireless network, a protocol that allows a PQC algorithm or hybrid mode to be used for securing application data is essential.

In addition, unlike in most implementations of PQC algorithms, the design and implementation of our proposed protocols extend beyond the integration of PQC into the traditional TLS/SSH to direct application-layer security and real-world scenarios. The use of a sequence-number-based mechanism to counter replay and MITM attacks provides an additional layer of security in addition to providing robust PQC algorithm protection. Unlike previous studies that focus mainly on algorithmic benchmarks or protocol integration at the transport layer, our proposed protocols demonstrate how existing applications can be practically migrated to achieve quantum resistance at the application layer.

3. PQC-Based PKE Protocols for Using the PQC Algorithm in Applications

According to the protocols described in [12], NIST recommends using the strongest algorithm parameters to ensure the effective use of PQC algorithms. The main reason for this is to avoid the cost and resources required to upgrade cryptographic implementations to a higher security level, as it is possible that upgrading may require modifications to the application/software where it will be needed. However, using stronger security parameters may have an adverse effect on application performance. Therefore, it is important to understand the impact of using PQC algorithms in various applications before carefully designing an optimization method or selecting a PQC algorithm for data encryption and decryption in various applications. To determine the impact of using PQC algorithms on applications and various networks, a protocol to encrypt and decrypt data transferred over the network is necessary. Therefore, we propose PQC standalone and PQC–AES hybrid PKE protocols that assume safe key exchange between the client and the server using the PQC KEM algorithm.

3.1. PQC Standalone PKE Protocol

The KEM is often used in conjunction with a PKE scheme to securely exchange encryption keys, which can then be used to encrypt data using the PKE scheme [6]. Therefore, in this paper, we select PKE as the cryptographic scheme for using PQC algorithms in various applications.

In the improved PQC standalone PKE protocol shown in Algorithm 1, both the client (C) and the server (S) generate two pairs of keys using post-quantum cryptographic algorithms. These include a public/private (secret) key pair for the KEM ($pk{k}_c$ and $sk{k}_c$ for the client, and $pk{k}_s$ and $sk{k}_s$ for the server), and a public/private (secret) key pair for the signature ($pk{s}_c$ and $sk{s}_c$ for the client, and $pk{s}_s$ and $sk{s}_s$ for the server). The client begins by signing its public KEM key ($pk{k}_c$) with its private signature key ($sk{s}_c$) to create a digital certificate ($cert\left[pk{k}_c\right]$). This certificate, along with the client’s public signature key ($pk{s}_c$), is sent to the server to establish authenticity (Algorithm 1, lines 3–7).

Upon receiving the certificate and public signature key from the client, the server verifies the authenticity of the certificate using the client’s public signature key ($pk{s}_c$). If the verification is successful, it confirms that the client’s public KEM key ($pk{k}_c$) has not been tampered with. The server then performs encapsulation on the client’s public KEM key ($pk{k}_c$) to generate a shared secret ($s{s}_c$) and a ciphertext ($c{t}_c$). The server also signs its public KEM key ($pk{k}_s$) with its private signature key ($sk{s}_s$) to create its digital certificate ($cert\left[pk{k}_s\right]$). The server sends this certificate, along with its public signature key ($pk{s}_s$) and the ciphertext ($c{t}_c$), back to the client (Algorithm 1, lines 8–18).

The client receives the server’s certificate, public signature key, and ciphertext. It verifies the authenticity of the server’s certificate using the server’s public signature key ($pk{s}_s$). If this verification succeeds, it confirms that the server’s public KEM key ($pk{k}_s$) is legitimate. The client then decapsulates the received ciphertext ($c{t}_c$) using its private KEM key ($sk{k}_c$) to derive a shared secret ($s{s}_c^{\prime }$). Next, the client performs encapsulation on the server’s public KEM key ($pk{k}_s$) to generate another shared secret ($s{s}_s$) and a ciphertext ($c{t}_s$), which it sends back to the server (Algorithm 1, lines 19–30).

Algorithm 1 PQC standalone PKE protocol

1: Input: Data to be transmitted ($dat{a}_c$ from Client, $dat{a}_s$ from Server)   2: Output: Securely exchanged data   3: Client (C):   4: Generate PQC KEM key pair: $(pk{k}_c,sk{k}_c)$   5: Generate PQC Signature key pair: $(pk{s}_c,sk{s}_c)$   6: $cert\left[pk{k}_c\right]\leftarrow \mathrm{Sign}(sk{s}_c,pk{k}_c)$   7: Send $(cert\left[pk{k}_c\right],pk{s}_c,pk{k}_c)$ to Server (S)   8: Server (S):   9: Generate PQC KEM key pair: $(pk{k}_s,sk{k}_s)$ 10: Generate PQC Signature key pair: $(pk{s}_s,sk{s}_s)$ 11: ${\mathrm{valid}}_c\leftarrow \mathrm{Verify}(pk{s}_c,cert\left[pk{k}_c\right],pk{k}_c)$ 12: if ${\mathrm{valid}}_c=\mathrm{TRUE}$ then 13:     $(c{t}_c,s{s}_c)\leftarrow \mathrm{Encapsulate}\left(pk{k}_c\right)$ 14:     $cert\left[pk{k}_s\right]\leftarrow \mathrm{Sign}(sk{s}_s,pk{k}_s)$ 15:     Send $(cert\left[pk{k}_s\right],pk{s}_s,pk{k}_s,c{t}_c)$ to Client (C) 16: else 17:     Abort 18: end if 19: Client (C): 20: ${\mathrm{valid}}_s\leftarrow \mathrm{Verify}(pk{s}_s,cert\left[pk{k}_s\right],pk{k}_s)$ 21: if ${\mathrm{valid}}_s=\mathrm{TRUE}$ then 22:     $s{s}_c^{\prime }\leftarrow \mathrm{Decapsulate}(sk{k}_c,c{t}_c)$ 23:     $(c{t}_s,s{s}_s)\leftarrow \mathrm{Encapsulate}\left(pk{k}_s\right)$ 24:     Send $c{t}_s$ to Server (S) 25:     $se{q}_c\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 26:     $ma{c}_c\leftarrow \mathrm{HMAC}\text{-}\mathrm{SHA}256(s{s}_c^{\prime }\left|\right|s{s}_s,“\mathrm{confirmation}”\left|\right|se{q}_c)$ 27:     Send $ma{c}_c$ to Server (S) 28: else 29:     Abort 30: end if 31: Server (S): 32: $s{s}_s^{\prime }\leftarrow \mathrm{Decapsulate}(sk{k}_s,c{t}_s)$ 33: $se{q}_s\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 34: $ma{c}_s\leftarrow \mathrm{HMAC}\text{-}\mathrm{SHA}256(s{s}_c\left|\right|s{s}_s^{\prime },“\mathrm{confirmation}”\left|\right|se{q}_s)$ 35: if $ma{c}_c=\mathrm{HMAC}\text{-}\mathrm{SHA}256(s{s}_c\left|\right|s{s}_s^{\prime },“\mathrm{confirmation}”\left|\right|se{q}_c)$ then 36:     Send $ma{c}_s$ to Client(C) 37: else 38:     Abort 39: end if 40: Client(C): 41: if $ma{c}_s=\mathrm{HMAC}\text{-}\mathrm{SHA}256(s{s}_c^{\prime }\left|\right|s{s}_s,“\mathrm{confirmation}”\left|\right|se{q}_s)$ then 42:     Secure communication is established. 43: else 44:     Abort 45: end if 46: Client-to-Server Data Transfer: 47: $c{t}_{dat{a}_c}\leftarrow \mathrm{PQC}\text{-}\mathrm{Encrypt}(pk{k}_s,dat{a}_c\left|\right|se{q}_c)$ 48: Send $c{t}_{dat{a}_c}$ to Server (S) 49: Server (S): 50: $dat{a}_c^{\prime }\left|\right|se{q}_c^{\prime }\leftarrow \mathrm{PQC}\text{-}\mathrm{Decrypt}(sk{k}_s,c{t}_{dat{a}_c})$ 51: if $se{q}_c^{\prime }$ is valid (not replayed) then 52:     Process $dat{a}_c^{\prime }$ 53: else 54:     Reject // Replay detected 55: end if 56: Server-to-Client Data Transfer: 57: $c{t}_{dat{a}_s}\leftarrow \mathrm{PQC}\text{-}\mathrm{Encrypt}(pk{k}_c,dat{a}_s\left|\right|se{q}_s)$ 58: Send $c{t}_{dat{a}_s}$ to Client (C) 59: Client (C): 60: $dat{a}_s^{\prime }\left|\right|se{q}_s^{\prime }\leftarrow \mathrm{PQC}\text{-}\mathrm{Decrypt}(sk{k}_c,c{t}_{dat{a}_s})$ 61: if $se{q}_s^{\prime }$ is valid then 62:     Process $dat{a}_s^{\prime }$ 63: else 64:     Reject 65: end if

To ensure that both parties have derived identical shared secrets and to prevent MITM attacks, a robust key confirmation process is employed. After deriving their respective shared secrets, both parties compute message authentication codes (MACs) using “hash-based message authentication code using the secure hash algorithm 256-bit (HMAC-SHA256)” over a combination of their shared secrets, a predefined “confirmation” string, and unique sequence numbers generated independently by each party. The MACs are exchanged between the client and server. After receiving a MAC, each party verifies that it matches its locally computed MAC. If both MACs match, it confirms that no MITM attack has occurred and that both parties share identical secrets (i.e., as shown for the client and the server in Algorithm 1, lines 31–45).

Once mutual confirmation is achieved, secure communication is established. The proposed protocols utilize core PQC primitives (KEMs and digital signatures) for key establishment and authentication, which is similar to the PQC-TLS/PQC-SSH handshake without the additional message negotiation steps (i.e., ClientHELLO/ServerHELLO) required in PQC-TLS. For data transfer from the client to the server, the client encrypts the data ($dat{a}_c$) along with its sequence number ($se{q}_c$) using the server’s public KEM key ($pk{k}_s$). The resulting ciphertext ($c{t}_{dat{a}_c}$) is sent to the server (Algorithm 1, lines 46–48). The server receives and decrypts the ciphertext using its private KEM key ($sk{k}_s$), retrieves both the data and sequence number, and checks whether the sequence number is valid (i.e., not replayed or out of order). If valid, it processes the data; otherwise, it rejects the data to prevent replay attacks (Algorithm 1, lines 49–55).

A similar process is followed for data transfer from the server back to the client. The server encrypts its data ($dat{a}_s$) along with its sequence number ($se{q}_s$) using the client’s public KEM key ($pk{k}_c$) to produce a ciphertext $c{t}_{dat{a}_s}$, which is sent to the client. Upon receiving this ciphertext, the client decrypts it using its private KEM key ($sk{k}_c$), retrieves both the data and the sequence number ($se{q}_s^{\prime }$), and checks whether the sequence number is valid. If the sequence number is valid, the client processes the data; otherwise, it rejects the data to prevent replay attacks (Algorithm 1, lines 56–65).

The proposed protocol introduces various enhancements compared to traditional PQC-based PKE protocols. First, we incorporate explicit certificate validation during the initial handshake steps to ensure that the exchanged KEM public keys are authentic and untampered. Second, we strengthen the replay protection by including unique sequence numbers in data encryption. Finally, by integrating robust HMAC-based key confirmation mechanisms, our proposed protocol significantly reduces vulnerabilities to MITM attacks while ensuring that shared secrets are securely established between legitimate parties.

3.2. PQC–AES Hybrid PKE Protocol

To enhance security and performance, especially against evolving threats, we introduce a PQC–AES hybrid PKE protocol. This approach combines the quantum resistance of PQC with the efficiency and established security of AES.

In this PQC–AES hybrid PKE protocol, as shown in Algorithm 2, we build upon the key exchange process already outlined for the PQC standalone PKE protocol. Following a successful and authenticated key exchange between the client and the server using the steps defined in Algorithm 1 (lines 1–24), both parties transition to utilizing AES encryption for subsequent data transfer. This approach leverages the best attributes of both cryptographic approaches: the quantum-resistant security offered by PQC algorithms during the initial key establishment and the speed and efficiency of AES for the bulk encryption of application data.

After the successful establishment and confirmation of shared secrets $s{s}_c$ and $s{s}_s$, both the client and the server independently derive AES encryption and MAC keys. A KDF is applied to the concatenation of these shared secrets, along with a unique salt and information string, using the HMAC-based extract-and-expand hash-based message authentication code key derivation function (HKDF) with SHA-256. This process yields two essential keys: $K_1$, designated for AES encryption, and $K_2$, designated for HMAC-based message authentication (Algorithm 2, lines 5–6).

For data transfer from the client to the server, the client begins by generating a unique initialization vector, $i{v}_c$, and a sequence number, $se{q}_c$, to provide replay protection. Subsequently, the data, $dat{a}_c$, is encrypted using AES with the derived key, $K_1$, and the initialization vector, $i{v}_c$, and incorporating the sequence number, $se{q}_c$, to form the ciphertext, $c{t}_{dat{a}_c}$. Following this, the client computes a MAC over the ciphertext, $c{t}_{dat{a}_c}$, and the sequence number, $se{q}_c$, using HMAC-SHA-256 with the authentication key, $K_2$, resulting in $ma{c}_{dat{a}_c}$. The client then sends the initialization vector, $i{v}_c$; the ciphertext, $c{t}_{dat{a}_c}$; and the MAC, $ma{c}_{dat{a}_c}$ to the server (Algorithm 2, lines 7–12).

Upon receiving this data, the server begins by verifying the integrity and authenticity of the data. It computes its own MAC over the received ciphertext, $c{t}_{dat{a}_c}$, and the sequence number, $se{q}_c$, using the same HMAC-SHA-256 algorithm and the authentication key $K_2$, generating a server-side MAC. The server then compares the received client-side MAC, $ma{c}_{dat{a}_c}$, with its locally computed MAC. If these MACs match, the server proceeds with decryption, ensuring that the received data has not been tampered with during transmission and that it originates from the authenticated client. The server decrypts the ciphertext, $c{t}_{dat{a}_c}$, using AES with the key, $K_1$, and the initialization vector, $i{v}_c$, and after decryption, it also verifies the sequence number against previously received sequence numbers to detect any replay attacks. If the sequence number is valid and the decryption is successful, the server processes the received data, $dat{a}_c^{\prime }$; otherwise, the server rejects the data to prevent potential security breaches (Algorithm 2, lines 13–23).

Similarly, the procedure is used for secure data transfer from the server to the client. The server also generates a unique $i{v}_S$ and uses $K_1$ to encrypt the data while applying HMAC to ensure authentication of the server by the client. The client also performs decryption in a similar way to ensure that the data is from the intended server (Algorithm 2, lines 24–40).

By combining the strengths of both PQC and AES, the protocol provides resilience against both classical and quantum attacks. Even if the PQC algorithm used for key exchange were compromised, AES encryption would still provide a layer of security. Furthermore, the hybrid approach can improve overall performance since AES encryption and decryption operations are generally faster and more efficient than PQC standalone encryption. Moreover, since AES-256 is a widely adopted and well-analyzed symmetric encryption algorithm, it has been demonstrated to be secure.

In the implementation of the proposed hybrid protocol and conventional AES-256 used in the evaluation, we adopt AES-GCM as the symmetric encryption scheme due to its strong security properties as an authenticated encryption algorithm [28] and its efficient performance. In comparison, AES-CBC requires separate MAC handling, making it more prone to implementation issues. Therefore, AES-GCM is prioritized over other types of symmetric encryption schemes.

Algorithm 2 PQC–AES hybrid PKE protocol

1: Input: Data to be transmitted ($dat{a}_c$ from Client, $dat{a}_s$ from Server)   2: Output: Securely exchanged data   3: Client (C) and Server (S):   4: Perform steps 1–24 of Algorithm 1 to establish shared secrets $s{s}_c$ and $s{s}_s$ (and confirm them).   5: Key Derivation (Both C and S):   6: $(K_1,K_2)\leftarrow \mathrm{HKDF}(s{s}_c\left|\right|s{s}_s,\mathrm{salt},“\mathrm{AES}\mathrm{keys}”)$   7: Client-to-Server Data Transfer:   8: $i{v}_c\leftarrow \mathrm{GenerateInitializationVector}\left(\right)$   9: $se{q}_c\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 10: $c{t}_{dat{a}_c}\leftarrow \mathrm{AES}\text{-}\mathrm{Encrypt}(K_1,i{v}_c,dat{a}_c,\mathrm{mode},se{q}_c)$ 11: $ma{c}_{dat{a}_c}\leftarrow \mathrm{HMAC}\text{-}\mathrm{SHA}256(K_2,c{t}_{dat{a}_c}\left|\right|se{q}_c)$ 12: Send $(i{v}_c,c{t}_{dat{a}_c},ma{c}_{dat{a}_c})$ to Server (S) 13: Server (S): 14: if $ma{c}_{dat{a}_c}=\mathrm{HMAC}\text{-}\mathrm{SHA}256(K_2,c{t}_{dat{a}_c}\left|\right|se{q}_c)$ then 15:     $dat{a}_c^{\prime }\leftarrow \mathrm{AES}\text{-}\mathrm{Decrypt}(K_1,i{v}_c,c{t}_{dat{a}_c},\mathrm{mode},se{q}_c)$ 16:     if $se{q}_c$ is valid (not a replay) AND Decryption successful then 17:         Process $dat{a}_c^{\prime }$ 18:     else 19:         Reject 20:     end if 21: else 22:     Abort 23: end if 24: Server-to-Client Data Transfer: 25: $i{v}_s\leftarrow \mathrm{GenerateInitializationVector}\left(\right)$ 26: $se{q}_s\leftarrow \mathrm{GenerateSequenceNumber}\left(\right)$ 27: $c{t}_{dat{a}_s}\leftarrow \mathrm{AES}\text{-}\mathrm{Encrypt}(K_1,i{v}_s,dat{a}_s,\mathrm{mode},se{q}_s)$ 28: $ma{c}_{dat{a}_s}\leftarrow \mathrm{HMAC}\text{-}\mathrm{SHA}256(K_2,c{t}_{dat{a}_s}\left|\right|se{q}_s)$ 29: Send $(i{v}_s,c{t}_{dat{a}_s},ma{c}_{dat{a}_s})$ to Client (C) 30: Client (C): 31: if $ma{c}_{dat{a}_s}=\mathrm{HMAC}\text{-}\mathrm{SHA}256(K_2,c{t}_{dat{a}_s}\left|\right|se{q}_s)$ then 32:     $dat{a}_s^{\prime }\leftarrow \mathrm{AES}\text{-}\mathrm{Decrypt}(K_1,i{v}_s,c{t}_{dat{a}_s},\mathrm{mode},se{q}_s)$ 33:     if $se{q}_s$ is valid AND Decryption Successful then 34:         Process $dat{a}_s^{\prime }$ 35:     else 36:         Reject 37:     end if 38: else 39:     Abort 40: end if

4. Practical Implementation of PQC-Based PKE Protocols

Since there is no function for data encryption and decryption in current PQC algorithm implementations, we enhance the PQC algorithm implementation by adding encryption and decryption functions. This enables data encryption and decryption in both PQC standalone and PQC–AES hybrid protocols. Both protocols utilize the PQC key encapsulation mechanism (KEM) for secure key exchange between the client and server.

4.1. Post-Quantum Cryptography (PQC) Uses in Applications

To demonstrate the impact of the characteristics of PQC algorithms, PQC-based protocols need to be utilized in real-world applications. However, current implementations of PQC algorithms using PKE are limited to encrypting messages of a fixed size. For instance, CRYSTALS-Kyber, which is based on an IND-CPA-secure PKE scheme, and BIKE, which utilizes PKE from Quasi-Cyclic Moderate Density Parity-Check (QC-MDPC) codes, can only encrypt messages of a fixed length of 32 bytes. HQC, also based on IND-CPA-secure properties, can accept messages of a fixed length of 16 bytes before encoding.

Conversely, when a hybrid approach is employed, the KDF generates symmetric keys of a consistent length for all PQC algorithms. While AES cipher modes typically use key sizes of 128, 192, or 256 bits (16, 24, or 32 bytes), it does not support key sizes larger than 256 bits. Therefore, to use the PQC standalone PKE directly for the encryption and decryption of data that exceeds these fixed sizes without the hybrid implementation, it is necessary to segment the message and reassemble the decrypted segments to reconstruct the original data. The process for this splitting is shown in Figure 1. In this instance, first, the input data is split into blocks of an acceptable size, as shown in Figure 1a. Each block is then encrypted sequentially using the receiver’s public key. The encrypted blocks are sent to the intended party. As shown in Figure 1b, on the receiver side, after the data is received, the encrypted data is processed as a stream. Each block is decrypted using the receiver’s private key and then aggregated to reconstruct the original data.

In our proposed PQC standalone protocol, after establishing a secure communication (i.e., session key) using the standardized PQC KEMs and digital signatures (e.g., CRYSTALS-Kyber, BIKE, and HQC), each segmented message is encrypted within that session using the established encryption and decryption keys. This approach aligns with standard cryptographic practice, where a session key securely negotiated using an IND-CCA2 secure KEM is reused for multiple encryptions within the same session. The security of each encrypted segment is ensured by the PQC algorithm and the sequence number mechanism introduced to counter replay attacks. Moreover, the proposed PQC standalone protocol leverages periodic re-keying when a session is too long. For this, a new set of PQC KEM and signature algorithm keys is generated. The re-keying process is currently not implemented in our evaluation and will be considered as part of future work.

4.1.1. Real-World Scenarios for Proposed PQC-Based Protocols

Consider a real-world scenario, shown in Figure 2, involving two devices representing the client and server, each equipped with the necessary software, including PQC and AES libraries. For this application scenario, we consider four use cases: (i) file transfer, (ii) chat, (iii) video streaming, and (iv) live streaming. Assuming that the client is Alice while the server is Bob, for Alice to send confidential data to Bob using any of these use cases within a PQC-based system to ensure the data remains secure against potential future quantum computer attacks, the following process is used:

• Bob generates a pair of keys using a PQC algorithm: a public key ($pk{k}_b$) and a private key ($sk{k}_b$).
• Bob shares his public key ($pk{k}_b$) with Alice while keeping the private key ($sk{k}_b$) secret.
• Alice prepares the data for transmission and splits it into multiple blocks if it exceeds the maximum byte size allowed by the PQC algorithm, as shown in Figure 1a.
• Alice encrypts each block sequentially using Bob’s public key ($pk{k}_b$). The encrypted blocks are combined into a single encrypted file.
• Alice sends the encrypted file through a secure network.
• The access point routes the encrypted file to Bob without decrypting it.
• Bob receives the encrypted file and uses his private key ($sk{k}_b$) to decrypt each block.
• He reassembles the decrypted blocks to reconstruct the original document, as shown in Figure 1b.

Figure 2. Use cases for PQC algorithm: (a) File transfer, (b) chat, (c) video streaming, (d) live streaming.

Figure 2. Use cases for PQC algorithm: (a) File transfer, (b) chat, (c) video streaming, (d) live streaming.

Using PQC-based protocols, Alice and Bob can ensure that their communication remains secure against threats posed by quantum computing. The workflow of the proposed PQC-based protocols described above can be applied to various real-world scenarios. For file transfers, the protocol ensures the secure transmission and reconstruction of files divided into encrypted blocks. In chat applications, messages can be encrypted as individual small blocks to ensure confidentiality in real-time conversations. For video streaming, secure pre-encryption of video segments guarantees that the data remains protected during transmission. Lastly, live streaming benefits from the protocol by encrypting and transmitting content in near-real time, with a focus on maintaining low latency to preserve streaming quality. These use cases demonstrate the versatility of the PQC-based protocols in addressing diverse application requirements while securing data against quantum threats.

5. Security Analysis of the Proposed Protocols

This section presents a detailed security analysis of the proposed PQC standalone and PQC–AES hybrid PKE protocols. We evaluate the resilience of these protocols against various attack vectors, considering both classical and post-quantum threats.

5.1. Security Analysis of the PQC Standalone PKE Protocol

The PQC standalone PKE protocol relies entirely on the security of the chosen PQC algorithms for both key exchange (KEM) and data encryption. We analyze its strengths and weaknesses, detailing the assumptions underlying its security.

5.1.1. Key Exchange (PQC KEM)

The key exchange phase uses a post-quantum secure KEM, such as CRYSTALS-Kyber, standardized by NIST.

Security Assumption

The security of the key exchange depends on the computational infeasibility of solving the underlying hard mathematical problem (e.g., Module-LWE problem) for quantum adversaries. This is based on cryptanalysis conducted during the NIST PQC standardization process.

Formal Statement

The confidentiality of the shared secret rests on the IND-CCA2 security of the chosen KEM in the quantum random-oracle model. If an adversary solves the Module-LWE problem or breaks the IND-CCA2 security of the KEM, the confidentiality of the shared secret is compromised.

5.1.2. Key Confirmation

Key confirmation prevents MITM attacks, ensuring both parties derive the same shared secret.

Mechanism

After decapsulation, parties derive a shared secret using a KDF. To confirm the shared secret, each party computes and exchanges HMACs (using HMAC-SHA-256) over a pre-defined data structure including nonces or session identifiers. The MACs are verified to ensure that no MITM attacker has replaced either party’s public key.

Security Assumption

The security relies on the following:

• The collision resistance of SHA-256.
• The pseudorandomness and unforgeability properties of HMAC.

Formal Statement

If SHA-256 is collision-resistant and HMAC is unforgeable under chosen-message attacks, an adversary cannot impersonate either party or inject their own public key into the key exchange process.

5.1.3. Data Encryption and Decryption (PQC)

The PQC standalone PKE protocol directly encrypts data using a public-key encryption scheme derived from the chosen KEM.

Security Assumption

The confidentiality of encrypted data depends on the post-quantum security of the public-key encryption scheme derived from the KEM, including resistance to chosen-ciphertext attacks (IND-CCA2 security).

Formal Statement

The confidentiality of transmitted data is guaranteed if the following conditions hold:

• The underlying KEM is IND-CCA2 secure.
• The public-key encryption scheme derived from it inherits this IND-CCA2 security property.

If an adversary breaks these assumptions, they can recover plaintext data from ciphertexts.

5.1.4. Replay Protection

Replay attacks involve an attacker capturing and retransmitting a valid encrypted message to disrupt communication or impersonate a legitimate sender.

Mechanism

To mitigate replay attacks, the protocol implements the following mechanisms:

• Sequence numbers are incorporated into each encrypted message.
• The receiver tracks these sequence numbers to ensure they are unique and monotonically increasing.
• Messages with duplicate or out-of-order sequence numbers are rejected.

Security Assumption

This approach assumes that sequence numbers are generated uniquely and managed correctly by both parties during communication.

Formal Statement

If sequence numbers are generated uniquely and monotonically increasing, replayed messages will be detected and rejected, preventing replay attacks.

5.1.5. Overall Security

The overall security of this protocol is determined by its reliance on post-quantum cryptographic primitives for both key exchange and data encryption.

Advantages

• Post-Quantum Security: Provides resistance against quantum adversaries due to reliance on PQC algorithms.
• Simplified Design: Avoids additional complexity introduced by hybrid schemes.

Limitations

• Single Point of Failure: The protocol’s security entirely depends on the strength of the PQC algorithms used (both KEM and public-key encryption). If these algorithms are compromised, both key exchange and data confidentiality are at risk.
• Performance Overhead: PQC standalone encryption/decryption operations are computationally intensive compared to symmetric cryptography like AES-256.

Formal Statement

The overall security of this protocol is bounded by the security assumptions of its underlying PQC algorithms. If an adversary breaks the post-quantum hardness assumptions (e.g., Module-LWE problem), they can compromise both key exchange and data confidentiality. Unlike hybrid approaches, breaking one component compromises overall security.

5.2. Security Analysis of the PQC–AES Hybrid PKE Protocol

The hybrid protocol leverages the strengths of both post-quantum key exchange (using a KEM like CRYSTALS-Kyber) and a well-established symmetric cipher (AES-256). This defense-in-depth approach enhances overall security.

5.2.1. Security of the Key Exchange (PQC KEM)

The key exchange phase is identical to that described for the PQC standalone PKE protocol (Section 5.1.1). The same security assumptions and formal statements apply.

5.2.2. Key Confirmation

Key confirmation is performed as described for the PQC standalone PKE protocol (Section 5.1.2) to prevent MITM attacks.

5.2.3. Security of Data Encryption (AES-256)

Data encryption uses AES-256, which is well-established and understood.

Security Assumption

AES-256 is considered secure against classical attacks. The hybrid approach aims for defense-in-depth; even if the PQC KEM is broken, the attacker still faces the challenge of breaking AES-256.

Formal Statement

The data encryption phase utilizes AES-256 with a key size of 256 bits. AES-256 is widely considered secure against classical computational attacks, providing 256 bits of security. We assume that the AES-256 implementation is free from side-channel vulnerabilities.

5.2.4. Overall Security Argument

The hybrid protocol combines the strengths of both PQC and AES-256.

Formal Statement

The overall security of the hybrid protocol is bounded by the security of its weakest component. By combining a post-quantum KEM with AES-256, the protocol achieves a defense-in-depth strategy. An attacker would need to break both the PQC KEM and AES-256 to compromise the confidentiality of the data. This provides a higher level of security against both classical and potential future quantum attacks. Since PQC encryption/decryption algorithms are often significantly slower than AES-256, the hybrid approach leverages the best aspects of both: post-quantum key exchange and fast symmetric encryption.

5.3. Robustness Against Common Attack Vectors

To demonstrate the robustness of our proposed protocols, we consider their resilience against several standard cybersecurity attack vectors.

5.3.1. Eavesdropping

Eavesdropping is the primary threat addressed by encryption. Both protocols provide protection against eavesdropping, although with different security assumptions.

PQC Standalone PKE

The PQC standalone PKE protocol protects against eavesdropping by encrypting data using the receiver’s public key, which only the receiver, possessing the corresponding private key, can decrypt. The security relies on the post-quantum security of the underlying PQC algorithm.

PQC–AES Hybrid PKE

The hybrid protocol provides two layers of protection against eavesdropping. First, the AES-256 key is exchanged securely using the PQC KEM. Second, the data itself is encrypted with AES-256. An eavesdropper would need to break both the PQC KEM and AES-256 to recover the plaintext.

5.3.2. Man-in-the-Middle (MITM) Attacks

Both protocols include a key confirmation step using HMAC-SHA-256 to prevent MITM attacks, ensuring that both parties have derived the same shared secret and that an attacker has not successfully impersonated either party during the key exchange.

5.3.3. Replay Attacks

Replay attacks are mitigated differently depending on the AES mode used.

PQC–AES Hybrid PKE (with AES-GCM)

When using AES-GCM, the GCM mode inherently provides protection against replay attacks. The authentication tag incorporates the initialization vector (IV) and ciphertext, so replaying a message with a different IV (which should be unique for each message) will result in authentication failure.

PQC–AES Hybrid PKE (with AES-CBC or Other Modes)

To mitigate replay attacks when not using GCM, we incorporate sequence numbers or timestamps within the HMAC calculation. The receiver tracks these sequence numbers or /timestamps and rejects any messages that are out of order or outside an acceptable time window.

5.3.4. Chosen-Ciphertext Attacks (CCAs)

This is relevant to the PQC KEM used in both protocols. NIST’s selections are generally CCA-secure. The chosen PQC KEM (e.g., CRYSTALS-Kyber) is designed to be IND-CCA2 secure, meaning it is resistant to chosen-ciphertext attacks. This ensures that an attacker cannot gain information about the plaintext even if they can choose ciphertexts to be decrypted.

5.3.5. Side-Channel Attacks

Side-channel attacks exploit physical information leaked during computation (power consumption, timing, etc.). While our current evaluation does not directly address side-channel attacks, we acknowledge their importance, particularly for implementations on embedded devices. Mitigation strategies against side-channel attacks such as constant-time implementations and masking techniques are crucial for real-world deployments and are a subject of ongoing research in the PQC community. We recommend using side-channel-resistant implementations of the chosen PQC algorithms.

5.3.6. Known-Plaintext Attacks

If an attacker somehow knows some of the plaintext and tries to use this to break the encryption, the AES-256 and PQC KEMs are designed to be resistant to known-plaintext attacks.

6. Evaluation

In this section, we conduct a performance evaluation of the proposed protocols through experiments with PQC algorithms. First, we enhance the PQC algorithm implementation by adding encryption and decryption functions, enabling data encryption and decryption using both PQC standalone algorithms and the PQC–AES hybrid protocols. Second, we implement our proposed protocols (i.e., PQC standalone and PQC–AES hybrid PKE protocols). Furthermore, we measure the time required to generate PQC KEM keys and PQC signature keys, as well as the time required to perform encapsulation, decapsulation, and file encryption and decryption, among other operations. We use our extended version (which can now handle data encryption and decryption) of the PQC algorithm API implementation provided by Open Quantum Safe (Liboqs) [29], which is a collection of all the PQC algorithms selected by NIST as one standard library.

6.1. Experimental Settings

In our implementation, we used the four use cases described in Section 4.1.1 (i.e., file transfer, chat, video streaming, and live streaming) and evaluated our protocols based on these cases. The experiment was conducted on two separate machines: one with an Intel^® Core™ i7-10510U CPU, Mouse laptop (sourced: Osaka, Japan) running at a base frequency of 1.80 GHz × 8 and 15.5 GiB of memory, running Ubuntu 20.04 LTS desktop, which served as the server; and another with an Intel^® Core™ i7-10510U CPU, Mouse laptop (sourced: Osaka, Japan) running at a base frequency of 1.80 GHz × 8 and 15.5 GiB of memory, running Ubuntu 20.04 LTS desktop, which was used as the client system.

Table 1. Configuration of client–server system.

Parameter	Value
	Server	Client
PC	Mouse Laptop	Mouse Laptop
OS	Ubuntu 20.04 desktop	Ubuntu 20.04 desktop
Network/Protocol	IEEE 802.11n	IEEE 802.11n
Frequency band	2.4 GHz	2.4 GHz
Maximum data rate	144 Mbps	144 Mbps
Sniffer	Wireshark	Wireshark

presents the PC settings and the configurations of the systems used.

Furthermore, the standardized algorithms selected by NIST were considered. These include the CRYSTALS-Kyber KEM algorithms—ML-KEM-512, ML-KEM-768, and ML-KEM-1024 (referred to as K512, K768, and K1024 hereafter)—as well as alternative KEM algorithms: BIKE (Round 4)—BIKE-L1, BIKE-L3, and BIKE-L5—and HQC–HQC-128, HQC-192, and HQC-256. For the signature algorithms, FALCON (Round 3)variants—FN-DSA-512 and FN-DSA-1024 (referred to as F512 and F1024 hereafter)—were also evaluated. To evaluate the proposed protocols and the impact of PQC algorithms on applications, we used client–server file transfer communication between two different systems using Python 3.8.10 socket programming with Transmission Control Protocol/User Datagram Protocol (TCP/UDP). The client encrypts a file and sends it to the server. TCP traffic generated from the socket programming was sent from the PC2 machine, acting as the client, and received by the PC1 machine, acting as the server. The traffic data was transmitted using IEEE 802.11n wireless LAN in the 2.4 GHz band. The traffic between the client and server was captured using Wireshark [30] from the server system. The experimental setup is shown in Figure 3.

Additionally, we implemented conventional cryptography (i.e., AES-256) to compare and highlight the impact of the PQC algorithm on file transfer. The goal of our evaluation was to facilitate the practical use of the PQC algorithm in real applications and to determine the impact of using the PQC algorithm on the application use case and network. We considered two main evaluation scenarios: (i) the performance of the PQC algorithm to determine how fast it takes to generate PQC algorithm keys, complete the KEM process (encapsulation and decapsulation) (i.e., non-pre-distributed key agreement process), apply digital signatures (signing and verification), and perform encryption and decryption; and (ii) the impact of the PQC algorithm on file transfer applications to determine which PQC algorithm is more suitable for real-time applications without extensive overhead. For this, we measured the processing time for encrypted data transfer, throughput, and round-trip time (RTT).

6.2. Results and Discussion

6.2.1. Performance Results of PQC Algorithm Key Generation

In order to evaluate the performance implications of integrating PQC algorithms, we measured the average key generation time (Avg. Key Gen.), key encapsulation time (Avg. Encap.), and key decapsulation time (Avg. Decap.) for the PQC KEM algorithms, as well as the signature key generation time (Sig. Gen. Time), signing time, and verification time for the PQC signature algorithms across the client and server systems.

Table 2. Average times to generate PQC algorithm KEM keys.

PQC KEM Algorithm	Client			Server
	Avg. Key Gen. (ms)	Avg. Encap. (ms)	Avg. Decap. (ms)	Avg. Key Gen. (ms)	Avg. Encap. (ms)	Avg. Decap. (ms)
BIKE-L1	0.43	0.12	1.64	1.23	0.12	1.52
BIKE-L3	0.80	0.21	5.68	2.63	0.19	5.79
BIKE-L5	1.95	0.36	14.42	5.05	0.33	15.94
HQC-128	3.45	5.76	10.23	5.60	4.03	8.44
HQC-192	4.88	12.83	28.11	9.70	11.99	32.20
HQC-256	8.73	17.96	49.77	14.32	1.81	53.40
K512	0.92	0.10	0.17	1.83	0.10	0.11
K768	0.19	0.09	0.16	0.37	0.10	0.10
K1024	1.00	0.14	0.18	0.49	0.09	0.11

and

Table 3. Average times to generate PQC algorithm signature keys.

PQC Signature	Client			Server
	Avg. Sig. Key Gen. (ms)	Avg. Signing (ms)	Avg. Ver. (ms)	Avg. Sig. Key Gen. (ms)	Avg. Signing (ms)	Avg. Ver. (ms)
F512	7.01	0.33	0.28	9.68	0.37	0.82
F1024	19.00	1.43	0.31	24.51	0.67	0.79

show the average time it took to generate the PQC algorithm keys (PQC KEM and signature keys), encapsulate and decapsulate the KEM keys, and sign and verify PQC signatures on the client and server systems.

For the KEM algorithms, Table 2 shows that for BIKE KEM at all security levels, BIKE-L1 achieved the fastest average key generation time on the client system of $0.43\mathrm{ms}$, while BIKE-L5 required $1.95\mathrm{ms}$, reflecting a progressive increase in computational demand with higher security levels. Conversely, the server-side key generation times for BIKE-L1, BIKE-L3, and BIKE-L5 were $1.23\mathrm{ms}$, $2.63\mathrm{ms}$, and $5.05\mathrm{ms}$, respectively, demonstrating a more pronounced rise compared to the client-side metrics. The encapsulation and decapsulation times also increased with security levels, particularly for the BIKE and HQC algorithms.

Furthermore, as shown in the table, BIKE-L5 decapsulation required $14.42\mathrm{ms}$ on the client system and $15.94\mathrm{ms}$ on the server system, while HQC-256 decapsulation required $49.77\mathrm{ms}$ (client) and $53.40\mathrm{ms}$ (server). However, the CRYSTALS-Kyber algorithms diverged from this trend: K768 achieved the fastest client-side key generation time of $0.19\mathrm{ms}$, surpassing both K512 ($0.92\mathrm{ms}$) and K1024 ($1.00\mathrm{ms}$). On the server side, the CRYSTALS-Kyber results showed that K1024 was the most efficient for key generation ($0.49\mathrm{ms}$), encapsulation ($0.09\mathrm{ms}$), and decapsulation ($0.11\mathrm{ms}$), with K512 and K768 achieving marginally higher times.

Furthermore, Table 3 shows the performance of the PQC signature algorithms. According to the results, F512 required $7.01\mathrm{ms}$ for key generation on the client system, significantly faster than F1024’s $19.00\mathrm{ms}$. A similar trend was observed on the server system, with F512 and F1024 requiring $9.68\mathrm{ms}$ and $24.51\mathrm{ms}$, respectively. The signing times for F512 were $0.33\mathrm{ms}$ (client) and $0.37\mathrm{ms}$ (server), whereas F1024 took $1.43\mathrm{ms}$ (client) and $0.67\mathrm{ms}$ (server), illustrating a reversal in efficiency, favoring the server for higher security levels. The verification times remained consistent for F512 across the systems ($0.28\mathrm{ms}$ client vs. $0.82\mathrm{ms}$ server) but diverged slightly for F1024 ($0.31\mathrm{ms}$ client vs. $0.79\mathrm{ms}$ server). These results underscore the trade-offs between security levels and computational overhead, particularly for server-side operations.

6.2.2. Performance Results of PQC Algorithm Encryption and Decryption

In the hybrid protocol, AES-256 is used for data encryption/decryption. It was expected that the times for these operations should remain consistently low, meaning they should not vary significantly with the chosen PQC KEM, as the underlying symmetric cipher remains the same. To focus on the computational differences introduced by the PQC algorithms themselves, the subsequent figures present the encryption and decryption times specifically for the PQC-only protocol. Therefore, we evaluated the data encryption and decryption times for the PQC KEM algorithms. The encryption process was executed on the client system, while decryption was performed on the server system.

Figure 4 shows the average encryption times for the BIKE, HQC, and CRYSTALS-Kyber algorithms when processing files of sizes $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$. For BIKE-L1, as shown in Figure 4a, the encryption times remained stable across file sizes, with $0.20\mathrm{ms}$ for $1\mathrm{KB}$, $0.199\mathrm{ms}$ for $10\mathrm{KB}$, and $0.148\mathrm{ms}$ for $100\mathrm{KB}$. However, BIKE-L3 exhibited higher encryption times for larger files, increasing from $0.47\mathrm{ms}$ ($1\mathrm{KB}$) to $0.63\mathrm{ms}$ ($10\mathrm{KB}$) before declining to $0.46\mathrm{ms}$ ($100\mathrm{KB}$), while BIKE-L5 ranged from $0.58\mathrm{ms}$ ($1\mathrm{KB}$) to $0.93\mathrm{ms}$ ($10\mathrm{KB}$) and $0.82\mathrm{ms}$ ($100\mathrm{KB}$).

Conversely, the HQC encryption times scaled significantly with both security levels and file sizes, as shown in Figure 4b. HQC-128 averaged $6.13\mathrm{ms}$ ($1\mathrm{KB}$), $5.96\mathrm{ms}$ ($10\mathrm{KB}$), and $5.63\mathrm{ms}$ ($100\mathrm{KB}$), whereas HQC-256 averaged $32.82\mathrm{ms}$ ($1\mathrm{KB}$), $37.04\mathrm{ms}$ ($10\mathrm{KB}$), and $37.54\mathrm{ms}$ ($100\mathrm{KB}$). The CRYSTALS-Kyber algorithms, as shown in Figure 4c, demonstrated modest variations, with K768 achieving the fastest $1\mathrm{KB}$ encryption time of $0.10\mathrm{ms}$, outperforming K512 ($0.15\mathrm{ms}$) and K1024 ($0.13\mathrm{ms}$). For $100\mathrm{KB}$ files, the CRYSTALS-Kyber encryption times remained low compared to those of the other algorithms, ranging from $0.10\mathrm{ms}$ (K768) to $0.17\mathrm{ms}$ (K1024).

Figure 5 shows the average decryption times on the server system. The BIKE decryption times increased sharply as the security level increased, ranging from an average decryption time of $2.17\mathrm{ms}$ ($1\mathrm{KB}$) to $23.85\mathrm{ms}$ ($100\mathrm{KB}$). BIKE-L1 achieved the lowest decryption time across all file sizes, with $2.17\mathrm{ms}$ ($1\mathrm{KB}$), $2.94\mathrm{ms}$ ($10\mathrm{KB}$), and $2.32\mathrm{ms}$ for $100\mathrm{KB}$. BIKE-L5 showed a slight increase in decryption time as the file size increased, with average decryption times of $23.17\mathrm{ms}$ ($1\mathrm{KB}$), $27.70\mathrm{ms}$ ($10\mathrm{KB}$), and $27.85\mathrm{ms}$ ($100\mathrm{KB}$), as shown in Figure 5a. Figure 5b shows that HQC followed a similar pattern to BIKE, ranging from an average decryption time of $2.82\mathrm{ms}$ to $26.18\mathrm{ms}$ for $100\mathrm{KB}$. HQC-128 required an average decryption time of $2.82\mathrm{ms}$ for $100\mathrm{KB}$ files, whereas HQC-192 required $9.38\mathrm{ms}$, which increased sharply to $26.18\mathrm{ms}$ for HQC-256 when decrypting the same file size.

In contrast, Figure 5c shows that the CRYSTALS-Kyber algorithms maintained consistently low decryption times for the same file size (i.e., $100\mathrm{KB}$), with K512 requiring an average decryption time of $0.12\mathrm{ms}$ and K1024 requiring $0.16\mathrm{ms}$. Notably, K768’s decryption time decreased slightly to $0.11\mathrm{ms}$ compared to K512 and K1024, deviating from the upward trend observed in BIKE and HQC. Smaller file sizes ($1\mathrm{KB}$) yielded minimal average decryption times for all algorithms, with K512 and K768 requiring $0.09\mathrm{ms}$. These results reaffirm the computational efficiency of lattice-based algorithms like CRYSTALS-Kyber for both encryption and decryption, particularly when handling larger file sizes, compared to code-based (BIKE) and hash-based (HQC) alternatives.

6.2.3. Impact of PQC Algorithms on Performance in the File Transfer Use Case

To demonstrate the impact of PQC algorithms on various applications, we evaluated the proposed protocols using a file transfer application. We measured the overall processing time, throughput, and RTT as metrics. Since PQC algorithms can be more computationally intensive than traditional algorithms, the requirements to secure the data may adversely affect the performance of the applications and the network’s data-handling capabilities. Therefore, file transfer was selected as the use case to test the application of the proposed protocols.

In our evaluation, we paired F512 and F1024 with each PQC KEM algorithm (the KEM algorithm paired with the F512 and F1024 signature algorithms is denoted as BIKE-L1-F512, BIKE-L3-F512, BIKE-L5-F512, BIKE-L1-F1024, BIKE-L3-F1024, BIKE-L5-F1024, HQC-128-F512, HQC-192-F512, HQC-256-F512, HQC-128-F1024, HQC-192-F1024, HQC-256-F1024, K512-F512, K768-F512, K1024-F512, K512-F1024, K768-F1024, and K1024-F1024) to perform the key exchange process for each algorithm in our proposed protocols. We measured the processing time, throughput, and RTT efficiency of the proposed protocols using various paired KEM and signature PQC algorithms during data transfer across a wireless LAN environment and compared the metrics achieved by different PQC algorithms for data sizes of $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$. The results of the data transfer between the client system and the server, including a key agreement (i.e., the PQC public keys of both parties are exchanged securely) and the actual file transfer, are shown in Figure 6, Figure 7 and Figure 8.

Processing Time

The overall processing times on the server system, segmented into $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes, highlight the performance disparities between PQC–AES hybrid implementations, PQC standalone methods, and conventional AES-256. For $1\mathrm{KB}$ files, as shown in Figure 6a, the hybrid BIKE-L1-F512 protocol recorded an average processing time of $1.56\mathrm{s}$, marginally faster than its PQC standalone counterpart ($1.64\mathrm{s}$), while AES-256 achieved $2.06\mathrm{s}$, demonstrating comparable efficiency to hybrid approaches. Conversely, according to the results in Figure 6b, HQC-256-F512 in PQC standalone mode required $2.79\mathrm{s}$, exceeding AES-256 by 35%, whereas its hybrid variant reduced this to $1.56\mathrm{s}$. For CRYSTALS-Kyber, K512-F512 exhibited inverse behavior, with hybrid processing at $1.73\mathrm{s}$ outperformed by its PQC standalone counterpart ($1.48\mathrm{s}$), suggesting algorithm-specific optimization variances.

Figure 6c shows the processing times for $10\mathrm{KB}$ files. Hybrid methods generally retained advantages: BIKE-L5-F512 hybrid processing required $1.48\mathrm{s}$, significantly lower than the time of its PQC standalone counterpart of $6.08\mathrm{s}$, while AES-256 required $1.82\mathrm{s}$. HQC-256-F512’s PQC standalone time, as shown in Figure 6d, surged to $16.78\mathrm{s}$, 9.2 times slower than that of AES-256, whereas its hybrid implementation reduced this to $2.11\mathrm{s}$. In comparison, K1024-F512 hybrid processing ($2.73\mathrm{s}$) was slightly higher than the PQC standalone variant ($1.55\mathrm{s}$), although both remained within 1.5–2.7 s, aligning closer with the efficiency of AES-256.

Similarly, the $100\mathrm{KB}$ file processing times showed the scalability challenges of PQC standalone methods. As shown in Figure 6f, BIKE-L5-F512 PQC standalone processing reached $32.02\mathrm{s}$, 13.5 times slower than AES-256’s $2.37\mathrm{s}$, while its hybrid variant, as shown in Figure 6e, achieved $1.62\mathrm{s}$, outperforming AES-256 by 34%. HQC-256-F512’s PQC standalone time escalated to $119.18\mathrm{s}$, 50.3 times higher than AES-256, whereas hybrid modes like HQC-128-F512 ($1.77\mathrm{s}$) and K768-F512 ($2.20\mathrm{s}$) demonstrated competitive or superior performance relative to AES-256. These results emphasize hybrid protocols’ potential to mitigate PQC overheads, particularly for larger data volumes, while reaffirming AES-256’s fixed efficiency for mid-sized operations.

Throughput

Furthermore, the server-side throughput results shown in Figure 7, categorized by $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes, illustrate the interplay between ciphertext dimensions and quantum-resistant algorithm performance relative to AES-256. Figure 7a shows results for the $1\mathrm{KB}$ data, confirming that AES-256 achieved a throughput of $0.170\mathrm{Mbps}$, while the PQC standalone methods with larger ciphertexts, such as HQC-128-F1024, reached $7.841\mathrm{Mbps}$, a 46.1-fold increase. This variation arose not from superior efficiency but from expanded ciphertext sizes increasing the throughput metrics. It was observed in the hybrid implementations that the CRYSTALS-Kyber algorithm paired with the FALCON digital signature achieved throughput results comparable to AES-256. The results show total throughput values of K512-F512 ($0.476\mathrm{Mbps}$), K768-F512 ($0.517\mathrm{Mbps}$), K512-F1024 ($0.554\mathrm{Mbps}$), and K768-F1024 ($0.647\mathrm{Mbps}$).

Additionally, Figure 7b shows that at $10\mathrm{KB}$, AES-256 throughput dropped to $0.0035\mathrm{Mbps}$, whereas PQC standalone total throughput increased. K1024-F512 achieved a total throughput of $19.516\mathrm{Mbps}$, reflecting the compounding effect of ciphertext expansion on throughput. BIKE-L1-F512 followed this trend at $10.578\mathrm{Mbps}$, although its larger ciphertexts inherently amplified throughput values despite slower absolute processing speeds. For the hybrid modes, HQC-256-F512 showed a total throughput of $2.005\mathrm{Mbps}$, 565 times higher than that of AES-256. This confirms the higher computational latency of the PQC standalone algorithm compared to the conventional cryptographic algorithm, underscoring the metric’s limitations in isolation.

For $100\mathrm{KB}$ files, as shown in Figure 7c, the AES-256 total throughput further increased to $9.725\mathrm{Mbps}$, while PQC standalone K768-F1024 also increased to $33.320\mathrm{Mbps}$, 3.4 times higher than AES-256. This disparity arose from the file size and ciphertext increase, as PQC algorithms produce larger ciphertexts than AES-256. BIKE-L1-F1024 achieved $11.968\mathrm{Mbps}$ in PQC standalone mode, while its hybrid variant ($7.459\mathrm{Mbps}$) lagged, emphasizing the trade-offs between post-quantum security and protocol complexity. Notably, BIKE-L5-F512’s PQC standalone mode achieved a throughput ($4.476\mathrm{Mbps}$) lower than AES-256 by a factor of 1.9, despite its larger ciphertexts, highlighting vulnerabilities in the scalability of code-based algorithms. These results underscore that while PQC algorithms may exhibit higher throughput due to ciphertext expansion, this metric alone does not capture their operational efficiency or suitability for latency-sensitive applications. The results demonstrate that although larger ciphertext sizes in PQC algorithms result in higher throughput compared to AES-256, this does not directly imply greater efficiency.

Round-Trip Time

The RTT measurements for $1\mathrm{KB}$, $10\mathrm{KB}$, and $100\mathrm{KB}$ file sizes shown in Figure 8 reflect the operational latency introduced by ciphertext expansion and protocol overhead in PQC algorithms, despite their higher throughput metrics. Figure 8a shows the results for $1\mathrm{KB}$ files. AES-256 recorded an RTT of $0.0115\mathrm{s}$, while multiple PQC standalone protocol algorithms achieved slightly lower latencies in comparison. HQC-128-F1024 achieved $0.0022\mathrm{s}$, 5.2 times faster than AES-256, and K512-F512 achieved $0.0024\mathrm{s}$. In the hybrid implementations, however, the algorithm pairs incurred slightly lower latencies than AES-256, while K1024-F512 ($0.0115\mathrm{s}$) achieved similar latency, underscoring the handshake overhead inherent to hybrid protocols.

At $10\mathrm{KB}$, as shown in Figure 8b, AES-256’s RTT decreased to $0.0063\mathrm{s}$, while the PQC standalone protocol algorithms maintained similar trends as with the $1\mathrm{KB}$ file size, except for a few algorithm pairs that showed slightly higher latencies. The PQC standalone protocols achieved higher RTTs for BIKE-L5-F512 ($0.017\mathrm{s}$), BIKE-L3-F1024 ($0.014\mathrm{s}$), BIKE-L5-F1024, HQC-256-F512 ($0.016\mathrm{s}$), HQC-192-F1024 ($0.013\mathrm{s}$), and HQC-256-F1024 ($0.012\mathrm{s}$) compared to the same algorithm pairs in hybrid mode and AES-256. Conversely, it was observed that the remaining algorithm pairs in hybrid mode showed the opposite trend, with slightly higher RTTs (HQC-128-F512 ($0.007\mathrm{s}$), HQC-192-F512 ($0.008\mathrm{s}$), HQC-128-F1024 ($0.007\mathrm{s}$), and K512-F1024 ($0.011\mathrm{s}$)) than those of PQC standalone mode. This showcases the suitability of migrating applications to PQC algorithms while highlighting the processing bottlenecks associated with certain PQC algorithms.

Lastly, Figure 8c shows that for $100\mathrm{KB}$ files, AES-256 achieved $0.0048\mathrm{s}$, outperforming many PQC standalone methods. BIKE-L5-F512 required $0.0117\mathrm{s}$, 2.4 times slower, and HQC-256-F512 reached $0.015\mathrm{s}$, 3.1 times slower. Exceptions included HQC-128-F512 ($0.0028\mathrm{s}$) and K768-F1024 ($0.0039\mathrm{s}$), which were 1.7 times and 1.2 times faster than AES-256, respectively. Notably, in hybrid mode, K768-F1024 incurred a latency of $0.0275\mathrm{s}$ for $100\mathrm{KB}$ files, 5.7 times slower than AES-256, highlighting the compounded overhead of hybrid key exchanges and ciphertext processing. These results emphasize that while PQC algorithms may exhibit competitive throughput due to larger ciphertexts, their RTT performance is more directly tied to computational complexity and protocol design, exposing trade-offs between post-quantum security and real-time application suitability.

6.2.4. Impact of PQC Algorithms on Performance in the Chat-Based Use Case

To further evaluate the performance of our proposed protocols, we implemented the chat-based use case and measured the total processing time that users spent engaged in an active chat session. The total processing time includes the KEM process, the time taken to input the chat message, and the encryption and decryption of the messages in a chat session. The text input time varied slightly for each algorithm, as it would in a real chat, and may account for slight variations in the total processing time (expected to be insignificant). In addition, we measured the RTT, which is the time taken from when a user sends a message to when a response is received (i.e., request and response pairs). The RTT is one of the most crucial metrics for measuring the responsiveness between chat partners. For this evaluation, we created custom messages, which were used across all PQC algorithms within our protocols. We used UTF-8 encoding, in which each ASCII character (including spaces and punctuation) is 1 byte.

Table 4. Examples of text messages.

User 1		User 2
Text Message	Size (Bytes)	Text Message	Size (Bytes)
Hello	5	Hi, how are you?	16
I’m good, thanks. And you?	26	You are welcome. I’m fine too.	30
Are you ready for the demo?	27	Yes, all set. Let’s get started!	32
Great! Here’s the first part of the code. What do you think?	60	Looks interesting. I have a few questions about the results.	60
Sure, I can explain anything about it.	38	Thanks! I want to know how you handled the data preprocessing step.	67
I used the standard normalization method and added custom	57	That makes sense. Let’s move on to the next section.	52

shows the text and size of each message used for client–server communication in the chat use case.

Processing Time

It was observed that the proposed protocols with various PQC algorithms are more suitable for the chat-based use case in comparison to the AES-256 conventional method. Figure 9 shows the performance of the protocols in maintaining end-to-end encryption similar to those in modern chat applications. Figure 9a shows the results when PQC KEM algorithms were paired with the F512 signature algorithm in hybrid mode. A lower processing time between $61\mathrm{s}$ and $73\mathrm{s}$ was achieved by all algorithm pairs than the conventional AES-256 algorithm, which recorded a total processing time of $80\mathrm{s}$. For the BIKE algorithms, BIKE-L5-F512 achieved the lowest time of approximately $61\mathrm{s}$, while for the HQC algorithms, HQC-256-F512 achieved the lowest time ($63\mathrm{s}$), and for the CRYSTALS-Kyber algorithms, K1024-F512 achieved the lowest time ($64\mathrm{s}$).

As shown in Figure 9b, when the signature algorithm F1024 was paired with each KEM algorithm, the total processing time ranged from $64\mathrm{s}$ to $69\mathrm{s}$, with most of the algorithms achieving processing times similar to when the KEM algorithm was paired with F512. BIKE-L3-F1024 ($64\mathrm{s}$) and K1024-F1024 ($64\mathrm{s}$) achieved the lowest processing times compared to the other algorithms. However, slightly higher processing times were observed for the PQC standalone protocol. According to the results in Figure 9c, the CRYSTALS-Kyber algorithms achieved higher processing times for K512-F512 ($77\mathrm{s}$), K768-F512 ($96\mathrm{s}$), and K768-F512 ($74\mathrm{s}$).

A similar trend was observed when F1024 was used, as shown in Figure 9d, with a lower processing time for K768-F1024 ($73\mathrm{s}$), while the time increased for K512-F1024 ($85\mathrm{s}$). The HQC algorithms maintained almost consistent processing times compared to the hybrid protocol, with slight variations of less than $5\mathrm{s}$. Conversely, BIKE experienced higher processing times in the PQC standalone protocol than in the hybrid protocol but lower processing times than the conventional AES-256 algorithm, as shown in Figure 9a,c. With the exception of K768-F512 and K512-F1024, which achieved higher processing times than AES-256 in the PQC standalone protocol, all algorithms in this mode experienced lower overheads and are expected to function effectively without adversely impacting the QoS of a chat-based application.

Round-Trip Time

The end-to-end delay likely to be introduced by the proposed protocols was measured using the average RTT. It was expected that a delay for small, bursty messages, including delays due to encryption, network transit, and decryption, might occur. Therefore, the RTT, which is important in interactive chat, was considered. Figure 10 shows the average RTTs of the end-to-end delay of the proposed protocols. It was observed that the proposed protocols achieved better performance than conventional AES-256, which had an average RTT of $2.44\mathrm{ms}$. According to the results shown in Figure 10a,b for the hybrid protocol, an average RTT of less than $1.5\mathrm{ms}$ was observed for all algorithms, except for K512-F512 ($1.66\mathrm{ms}$), K768-F512 ($1.51\mathrm{ms}$), and K512-F1024 ($1.52\mathrm{ms}$).

In the PQC standalone protocol, a lower delay was observed for most of the PQC algorithms, with the exception of CRYSTALS-Kyber, compared to the hybrid protocol. Figure 10c confirms that the average RTT decreased with the increase in the KEM security level for most of the PQC algorithms, ranging from $0.4\mathrm{ms}$ to $1.7\mathrm{ms}$, except for K768-F512, which had a higher average RTT of $2.10\mathrm{ms}$, slightly closer to that of AES-256 ($2.44\mathrm{ms}$). As shown in Figure 10d, a similarly lower average RTT trend, decreasing as the security level increased, was observed. However, there was a slight variation in the pattern for the BIKE algorithms compared to the hybrid protocol. It was confirmed that BIKE-L5-F1024 increased slightly to $0.99\mathrm{ms}$. It was further confirmed that HQC-256-F1024 achieved the lowest average RTT of $0.28\mathrm{ms}$.

6.2.5. Impact of the PQC Algorithm on Performance in the Video-Streaming Use Case

To demonstrate the impact of PQC algorithms on real-time applications such as video streaming, we evaluated the proposed protocols using a video-streaming application, as described in Section 4.1.1. We measured the overall throughput and retransmission rate as metrics. However, PQC algorithms are not suitable for applications such as video and live streaming in their current form without substantial optimization due to the large ciphertext size, which results in continuous freezes of video frames caused by the frequent retransmission of frames. Therefore, to evaluate the performance of our proposed protocol, we utilized the hybrid protocol and compared its performance with that of conventional AES-256.

In our evaluation, we streamed a short pre-recorded “MP4” video with the following properties: a total size of $17\mathrm{MiB}$, a resolution of 1920 × 1080, 30 frames per second, a bitrate of $2.863\mathrm{Mbps}$, and a duration of 47 s $249\mathrm{ms}$. The server acquired each frame from the video file and encoded it before encrypting it. The encrypted frames were transmitted sequentially as rapidly as possible without limiting the rate. On the other hand, the client (stream viewer) decrypted and decoded each received encrypted frame. These frames were rendered using an OpenCV window for viewing the stream. The display loop included a standard 1 ms delay (i.e., cv2.waitkey(1)), which is necessary for OpenCV to process GUI events and refresh the displayed frame. This delay also allows for user interaction (e.g., capturing user input, such as ‘q’ for quitting the stream). The delay is an attribute of the OpenCV GUI mechanism used for the experiment and is not part of the processing overhead of the proposed protocols.

Throughput

We measured the total throughput, which reflects how many encrypted video frames can be successfully transmitted. Since the user experience in video streaming is important, we measured the metrics from the client side. Figure 11 shows the total throughput achieved for streaming an encrypted video over the network. An almost constant total throughput was observed across all PQC algorithms. The results shown in Figure 11a show an approximate total throughput of $22\mathrm{Mbps}$ for K512-F512 and K768-F512 compared to the other algorithms and conventional AES-256. The other algorithms used in the hybrid protocol achieved a slightly lower approximate total throughput ($20.1\mathrm{Mbps}$) than conventional AES-256 ($20.8\mathrm{Mbps}$).

Furthermore, as shown in Figure 11b, a similar trend was observed for the KEM algorithms paired with F1024, with almost all the algorithms achieving a slightly lower total throughput compared to the performance of AES-256, except for K1024-F1024, which achieved the highest total throughput of $21.6\mathrm{Mbps}$, an $0.8\mathrm{Mbps}$ increase compared to conventional AES-256. The pairs BIKE-L3-F1024, BIKE-L5-F1024, HQC-128-F1024, and K768-F1024 achieved the lowest total throughput of approximately ($19\mathrm{Mbps}$) compared to all the other algorithms.

Retransmission Rate

Additionally, we measured the TCP retransmission rate. A higher retransmission rate indicates an increase in packet loss, often due to network congestion, which can result in freezes or buffering in the video stream. The results shown in Figure 12a confirm lower retransmission rates ranging from $0.05\%$ to $0.20\%$ for most algorithms in the hybrid protocol, except for BIKE-L1-F512, HQC-192-F512, K768-F512, and K1024-F512, which showed retransmission rates above $0.20\%$. BIKE-L5-F512 achieved the lowest retransmission rate, with $0.05\%$ of the total encrypted frames streamed. Similarly, the pairs BIKE-L3-F512 ($0.16\%$), HQC-128-F512 ($0.16\%$), HQC-256-F512 ($0.20\%$), and K512-F512 ($0.15\%$) achieved lower retransmission rates than AES-256. Conversely, BIKE-L1-F512 ($0.22\%$), HQC-192-F512 ($0.25\%$), K768-F512 ($0.27\%$), and K1024-F512 ($0.26\%$) achieved higher retransmission rates than AES-256.

As shown in Figure 12b, in comparison to the algorithms paired with F512, the KEM algorithms paired with F1024 exhibited an opposite trend, with the exception of HQC-192. According to the results, HQC-192-F1024 achieved a retransmission rate of $0.28\%$, which was $0.03\%$ higher than that of HQC-192-F512. Similarly, a downward trend was recorded for CRYSTALS-Kyber as the security level increased, which differs from the retransmission rates observed when the KEM algorithm was paired with F512. K512-F1024 achieved the highest retransmission rates of $0.32\%$, which was higher than that of K512-F512 by $0.17\%$ while K768-F1024 and K1024-F1024 achieved lower retransmission rates of $0.09\%$ and $0.12\%$ than K768-F512 and K1024-F512, respectively. It was further confirmed that BIKE-L5-F1024 ($0.22\%$), HQC-128-F1024 ($0.25\%$), and K512-F1024 ($0.32\%$) all achieved higher retransmission rates than conventional AES-256. This suggests that further optimization may be necessary in order to use these algorithms effectively in real-time applications.

6.2.6. Impact of the PQC Algorithm on Performance in the Live-Streaming Use Case

Similar to video streaming, we evaluated the impact of the proposed protocols using a live-streaming session. We measured the total throughput and retransmission rate, which are essential metrics in real-time applications. The live-streaming use case involved capturing frames from a webcam in real time. For this, a live video was captured from the system’s default webcam using OpenCV’s video interface. A constant live-streaming duration of 30 s was used across all PQC algorithms. This was done to keep the size of the captured file smaller, as it increases exponentially with the number of frames and stream duration. After capturing each frame, the same process used for encoding and encryption, as described in the video-streaming use case, was utilized in the live-streaming use case.

Throughput

Figure 13 shows the total throughputs observed from the live-streaming session. As shown in Figure 13a, an almost constant total throughput was achieved with most of the PQC algorithms across all security levels compared to AES-256 ($9.7\mathrm{Mbps}$). BIKE-L3-F512 achieved the lowest throughput of $8.6\mathrm{Mbps}$, while K512-F512 achieved a total throughput of $9.2\mathrm{Mbps}$, slightly less than AES-256. The other algorithms recorded slightly higher total throughputs. Similarly, as shown in Figure 13b, total throughputs comparable to that of AES-256 were observed across all PQC algorithms, with BIKE-L1-F1024 ($10.1\mathrm{Mbps}$), HQC-128-F1024 ($10.0\mathrm{Mbps}$), and HQC-192-F1024 ($10.3\mathrm{Mbps}$) achieving total throughputs slightly higher than AES-256.

Retransmission Rate

Finally, we measured the retransmission rates of the live stream. According to the results shown in Figure 14a, most of the PQC algorithms in the hybrid protocols are suitable for the live-streaming use case and comparable to AES-256 performance. A lower retransmission rate of less than $0.1\%$ was observed in most algorithms, except for HQC-256-F512 ($0.16\%$) and K512-F512 ($0.67\%$), compared to AES-256 ($0.72\%$), while higher retransmission rates were observed for BIKE-L3-F512 ($2.28\%$) and HQC-128-F512 ($1.46\%$).

Additionally, for KEM algorithms paired with F1024, Figure 14b shows the retransmission rates. BIKE-L1-F1024 ($3.57\%$), BIKE-L3-F1024 ($2.42\%$), and HQC-192-F1024 ($3.06\%$) had higher retransmission rates compared to the other algorithms and AES-256. The lowest retransmission rate was observed for HQC-256-F1024 ($0.1\%$). This suggests that BIKE-L1-F1024, BIKE-L3-F1024, and HQC-192-F1024 may not be suitable for real-time live streaming without freezing and buffering compared to the other algorithms.

6.3. Limitations and Future Directions

In the current implementation of our proposed protocols, use cases in the area of distributed systems and network scenarios with congestion or high mobility are not currently considered. While the proposed protocols are designed for adoption in various use cases, extensive adaptation to mitigate challenges common in these areas requires further study to ensure correct implementation and migration of applications associated with such scenarios to the quantum-secure protocol. To further improve our proposed protocols, as part of our future efforts, we will consider expansion to various systems such as distributed architectures, network scenarios with high overhead impacts, and mobility.

Furthermore, while our current implementation ensures MAC synchronization and consistency between the client and the server, formal modeling remains an open issue. In future work, we will consider the use of formal verification tools to analyze the robustness of MAC synchronization in scenarios where delays, packet loss, or concurrent events may occur. Additionally, we will expand the proposed protocols to include additional network environments and platforms, including energy consumption analysis on ARM and other IoT-specific hardware. Moreover, the current implementation of the proposed protocols focuses on the application layer and does not directly address side-channel attacks at the lower levels. A detailed side-channel analysis using tools such as ChipWhisper and constant-time execution testing in embedded scenarios is necessary and will be considered as part of the expansion of the proposed protocols’ implementation.

7. Conclusions

As part of the advancement toward the post-quantum era, this paper proposed PQC standalone and PQC–AES hybrid protocols to evaluate their practical implementation within real-world secure communication systems. Specifically, the secure exchange of keys is achieved using PQC algorithms (KEM) in either PQC standalone or hybrid protocol modes. In PQC standalone mode, the KEM (public) keys exchanged between the client and server are used for encryption, while each party uses their secret keys for decryption, following the principles of asymmetric-key cryptography. Conversely, PQC–AES hybrid mode uses two-way shared secret keys derived from PQC KEM algorithms to generate encryption and decryption keys using a key derivation function (KDF), thereby reducing the computational overhead for larger data sizes. The proposed PQC-based protocols mitigate replay, MITM attacks, and other conventional cybersecurity attack vectors.

A performance evaluation of the proposed protocols was conducted through experiments using various real-world use cases (file transfer, chat-based communication, video streaming, and live streaming) in a client–server setup. For file transfer, the results confirmed that the hybrid protocols, which leverage the strengths of PQC algorithms, achieved performance relatively closer to that of AES-256 for larger payloads. For $10\mathrm{KB}$ files, the K512-F1024 hybrid mode achieved $0.8\mathrm{Mbps}$ compared to AES-256’s $0.0035\mathrm{Mbps}$, without the significant latency bottlenecks observed in PQC standalone mode, such as $19.5\mathrm{Mbps}$ for K1024-F512 and K1024-F1024. Additionally, it was confirmed that the hybrid protocol achieved better performance in terms of processing time compared to the PQC standalone protocol and AES-256. The processing times for the hybrid protocol were consistently lower across all algorithms, except for HQC192-F512 and K1024-F512, which recorded $2.6\mathrm{s}$ and $5.4\mathrm{s}$ for $1\mathrm{KB}$ files, compared to AES-256’s $2.06\mathrm{s}$. For $100\mathrm{KB}$ files, HQC-192-F1024 and HQC-256-F1024 experienced higher processing times of $2.4\mathrm{s}$ and $3.1\mathrm{s}$ compared to AES-256’s $2.37\mathrm{s}$.

Our findings confirm that PQC standalone algorithms remain effective for lightweight applications with smaller data sizes (e.g., file transfer with smaller size data and chat-based communication), while PQC–AES hybrid protocols provide a scalable and efficient solution for larger payloads (i.e., all use cases), ensuring quantum-resistant security with manageable latency and computational complexity. Using PQC algorithms directly in most applications will adversely impact performance, while the hybrid approach is more suitable for most applications. This reaffirms the necessity of designing and migrating applications to quantum-safe cryptographic protocols tailored to specific application requirements.

In the advancement toward quantum-resistant security, a critical focus should be on addressing stringent QoS requirements in emerging technologies like 5G/6G and IoT. These sectors necessitate a balance between robust security and real-time performance, requiring a hybrid approach that leverages the strengths of both PQC and traditional cryptographic algorithms. Our evaluation results provide valuable insights into the practicality and performance of standalone PQC protocols and can be extended to facilitate secure data transmission in 5G and IoT environments, ensuring a balance between quantum-resistant security and operational efficiency that upholds the stringent QoS requirements inherent in modern wireless networks.

In future work, we will evaluate the complexity and overhead of the proposed protocols. We will extend the implementation of the PQC standalone PKE to include other PQC algorithms such as McEliece, Dilithium, and $SPHINC{S}^{+}$, among others. In addition, we will explore methods for utilizing the PQC standalone PKE in various applications like video streaming and messaging. Furthermore, we will perform an extensive evaluation using various network environments, including 5G/6G.