Hybrid Constellation Shaping 64QAM Based on Hexagonal Lattice of Constellation Subset

: Increasing demand for higher-speed and large-capacity data communications has driven the development of constellation shaping technology. This paper proposes a hybrid constellation shaping scheme for 64-quadrature amplitude modulation (64QAM) based on hexagonal lattice of a constellation subset. The proposed scheme aims to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. The initial quantitative characterization of the constellation is ﬁrstly performed based on the hexagonal lattice structure. Then, the objective function of maximizing constellation ﬁgure of merits (CFM) is utilized to determine the position distribution of constellation points, resulting in the generation of the geometric shaping-64QAM (GS-64QAM) signal. Finally, according to concentric hexagonal layers, all constellation points are divided into multiple subsets where points within the same subset are assigned the same probability, and the hybrid shaping-64QAM (HS-64QAM) signal is generated. To validate the effectiveness of the proposed scheme, the experimental veriﬁcation was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. Experimental results indicate that, at the soft-decision forward error correction threshold, HS-64QAM achieves an optical signal-to-noise ratio (OSNR) gain of 1.9 dB and 4.1 dB over uniform GS-64QAM in back-to-back and 375 km transmission scenarios, respectively. Furthermore, HS-64QAM achieves an OSNR gain of 2.7 dB and 7.6 dB over uniform Square-64QAM in back-to-back and 375 km transmission scenarios, respectively.


Introduction
With the rapid development of global communication, new diversified technologies such as artificial intelligence, cloud computing, and big data have been promoted and integrated [1][2][3]. Optical fiber communication systems are faced with the increasing demands for higher-speed and large-capacity data communication [4][5][6]. To meet the demands mentioned above, higher-order modulation technology has been widely adopted to improve spectral efficiency and channel capacity without taking up additional spectrum [7,8]. However, with increasing the modulation order, the constellation points get closer to each other. Traditional higher-order modulation constellations are particularly sensitive to nonlinearities, resulting in limited transmission distances [9,10]. Among various emerging technologies, constellation shaping technology achieves the shaping gain by designing the

The Initial Quantitative Characterization of the 64QAM Constellation
The hexagonal structure can achieve denser constellation points in the constellation plane with min-ED fixed. Moreover, increasing min-ED reduces the probability of erroneous decisions and improves the error-tolerant performance of the communication system. Given the above reasons, the 64QAM constellation is initialized with a hexagonal lattice structure, where the centroid of each lattice is the initial position of the constellation points, as shown in Figure 1.

The Initial Quantitative Characterization of the 64QAM Constellation
The hexagonal structure can achieve denser constellation points in the constellation plane with min-ED fixed. Moreover, increasing min-ED reduces the probability of erroneous decisions and improves the error-tolerant performance of the communication system. Given the above reasons, the 64QAM constellation is initialized with a hexagonal lattice structure, where the centroid of each lattice is the initial position of the constellation points, as shown in Figure 1.
where the variable h is the number of all constellation points on the second layer inwards from the outside, and M is the total number of 64QAM constellation points (M = 64). Assuming that there are m-layer constellation points in the optimized constellation, where m = 1, 2, · · · , M m , M m is a positive integer. Moreover, the most central point is initialized to the first layer, which typically adopts the origin. Then, the constellation point set Point m,k (x, y) at the m-th layer can be expressed as: where (x m,k , y m,k ) indicates the k-th constellation point on the m-th layer, r m,k and ϕ m,k represent the radius and phase information of the constellation point (x m,k , y m,k ), respectively, r m,k = (x 2 m,k + y 2 m,k ), and θ i denotes the minimum angle from the positive real axis between the constellation points on the m-th layer.
The number H m of all constellation points on the m-th layer satisfies the following conditions: where the variable h is the number of all constellation points on the second layer inwards from the outside, and M is the total number of 64QAM constellation points (M = 64). Therefore, the set S C of all 64QAM constellation points can be obtained as follows: Photonics 2023, 10, 1008 4 of 13

Generation of the GS-64QAM Signal
To better measure the performance of the constellation C, the ratio of min-ED d min (C) to the average power P mean (C) is typically used to define CFM, which is expressed as: GS optimizes the position distribution of constellation points, which improves the CFM without changing the signal transmitted power. To improve the nonlinear tolerance of the 64QAM signal, the GS-64QAM is designed with the maximization of CFM as the objective function. In addition, the normalization of min-ED is commonly adopted for evaluating the constellation performance by CFM.
Combined with Equations (1), (3), and (4), the objective function can be calculated as follows: The specific value of h and corresponding constellation coordinates are obtained after several iterations with the goal of achieving the maximum value of CFM (S C ). Figure 2a shows the 64QAM constellation structure with adjusted coordinates. Since the number of points adjacent to a random constellation point ranges from two to six, gray coding cannot be strictly followed. To retain the superiorities of gray coding as much as possible, quasi-gray (QG) coding is adopted to minimize the Hamming distance of all adjacent constellation points, which decreases the damage of errors. QG coding preferentially encodes the case where the Hamming distance of the innermost point is exactly one and satisfies the requirement that the Hamming distance of the innermost point is smaller than the Hamming distance of the outermost point. Therefore, the set C S of all 64QAM constellation points can be obtained as follows:

Generation of the GS-64QAM Signal
To better measure the performance of the constellation C, the ratio of min-ED min ( ) d C to the average power ( ) mean P C is typically used to define CFM , which is expressed as: GS optimizes the position distribution of constellation points, which improves the CFM without changing the signal transmitted power. To improve the nonlinear tolerance of the 64QAM signal, the GS-64QAM is designed with the maximization of CFM as the objective function. In addition, the normalization of min-ED is commonly adopted for evaluating the constellation performance by CFM .
Combined with Equations (1), (3), and (4), the objective function can be calculated as follows: The specific value of h and corresponding constellation coordinates are obtained after several iterations with the goal of achieving the maximum value of ′( ) C CFM S . Figure 2a shows the 64QAM constellation structure with adjusted coordinates. Since the number of points adjacent to a random constellation point ranges from two to six, gray coding cannot be strictly followed. To retain the superiorities of gray coding as much as possible, quasigray (QG) coding is adopted to minimize the Hamming distance of all adjacent constellation points, which decreases the damage of errors. QG coding preferentially encodes the case where the Hamming distance of the innermost point is exactly one and satisfies the requirement that the Hamming distance of the innermost point is smaller than the Hamming distance of the outermost point.  Then, the GS-64QAM signal is generated according to the QG coding rules. The GS-64QAM constellation diagram is shown in Figure 2b. To achieve symbols of normalized energy, the adjusted coordinates of constellation points in Figure 2a are normalized. Moreover, the normalized coordinates of the GS-64QAM constellation points are obtained, as illustrated in Table 1. Based on the QG coding rules, the sum of the Hamming distances Photonics 2023, 10, 1008 5 of 13 between a selected constellation point and its neighbors is as small as possible. QG coding is a reliable coding method that reduces the misjudgment rate between constellation points. The bits under the QG coding rules and the normalized coordinates of corresponding GS-64QAM constellation points are illustrated in Table 1.

Generation of the HS-64QAM Signal
PS can achieve flexibility in terms of entropy and transmission performance by designing the probability distribution of the constellation. To further improve the constellation shaping gain, PS is optimized on top of GS to achieve a superposition of PS and GS dominance.
In the GS-64QAM constellation as shown in Figure 2b, all constellation points are divided into M m (M m = 6) subsets (layers) according to concentric hexagonal layers centered at the origin, as illustrated in Figure 3. The number of points within the same subset (on the same layer) is H m . In addition, constellation points within the same subset are assigned the same probability. The designation of a probability distribution is completed by directly adjusting the probability of each subset (each layer). Then, constant composition distribution matcher (CCDM) converts uniform bit streams to symbol sequences with expected probability distribution, and HS-64QAM signals are generated.
The probability of constellation points x i on the i-th layer can be described as: where υ represents the shaping parameter and determines the degree of PS, H k denotes the number of constellation points on the k-th layer, P i mean and P k mean indicate the average power of the i-th hexagonal layer and the k-th hexagonal layer, respectively.
Photonics 2023, 10, 1008 6 of 13 P i mean can be given by the ratio of the power of all points on the i-th layer to the total number H i of constellation points on the i-th layer, which can be given by: where Po i,j represents the j-th constellation points on the i-th layer, Re(·) and Im(·) represent the real and imaginary parts of a complex constellation point Po i,j , respectively, and the corresponding entropy H can be calculated by: The change of the value υ can achieve the adjustment of the signal entropy and further realize the comprehensive consideration of the transmission performance and signal entropy H according to practical requirements of the system. By combining Equations (6) and (7), the optimized probability distribution of the GS-64QAM constellation is obtained. The symbol sequence of the expected probability distribution is further obtained by CCDM, and then the HS-64QAM signal is generated. The probability of constellation points i x on the i-th layer can be described as: where υ represents the shaping parameter and determines the degree of PS, k H denotes the number of constellation points on the k-th layer, 1 (Re( ) Im( ) ), 1, 2, , where , i j Po represents the j-th constellation points on the i-th layer,

Experimental Setup
To verify the effectiveness of the proposed scheme, an experiment was conducted on a 120 Gbit/s transmission over a 5 × 75 km single mode fiber (SMF) with the proposed HS-64QAM in a coherent optical fiber communication system, as illustrated in Figure 4. The transmission performance of HS-64QAM is compared with that of uniform S-64QAM and uniform GS-64QAM.
At the transmitter end, the pseudo-random bit sequence (PRBS) was generated and used to generate different 64QAM signal sequences (HS-64QAM with H = 5.7 bits/symbol, and uniform S-64QAM, uniform GS-64QAM). The shaping factor v of HS-64QAM was 0.1308. To ensure a net rate of 120 Gbit/s, the baud rate multiplied by the entropy H should be the same. The above three 64QAM signals should have a baud rate of 21.06 GBaud, 20 GBaud, and 20 GBaud, respectively. The above sequences were successively imported into the arbitrary waveform generator (AWG) to complete the digital-to-analog conversion, where the sampling rate of the AWG was set to 50 GSa/s, and the peak value of the AWG output voltage was 200 mVpp. Analog signals were then amplified by two electrical amplifiers (EA) (SHF809) and fed into the IQ modulator (FTM7960). An external cavity Photonics 2023, 10, 1008 7 of 13 laser (ECL) with 100 kHz linewidth and 15 dBm signal output power was used to generate the continuous optical wave. Optical waves at 1550 nm were loaded into an IQ modulator for electrical-to-optical conversion. Finally, an erbium-doped fiber amplifier (EDFA) with a noise figure of 5.5 dB and a variable optical attenuator (VOA) were used to adjust the TOP transmitted into the fiber link.  At the transmitter end, the pseudo-random bit sequence (PRBS) was generated and used to generate different 64QAM signal sequences (HS-64QAM with H = 5.7 bits/symbol, and uniform S-64QAM, uniform GS-64QAM). The shaping factor v of HS-64QAM was 0.1308. To ensure a net rate of 120 Gbit/s, the baud rate multiplied by the entropy H should be the same. The above three 64QAM signals should have a baud rate of 21.06 GBaud, 20 GBaud, and 20 GBaud, respectively. The above sequences were successively imported into the arbitrary waveform generator (AWG) to complete the digital-to-analog conversion, where the sampling rate of the AWG was set to 50 GSa/s, and the peak value of the AWG output voltage was 200 mVpp. Analog signals were then amplified by two electrical amplifiers (EA) (SHF809) and fed into the IQ modulator (FTM7960). An external cavity laser (ECL) with 100 kHz linewidth and 15 dBm signal output power was used to generate the continuous optical wave. Optical waves at 1550 nm were loaded into an IQ modulator for electrical-to-optical conversion. Finally, an erbium-doped fiber amplifier (EDFA) with a noise figure of 5.5 dB and a variable optical attenuator (VOA) were used to adjust the TOP transmitted into the fiber link.
The transmission link was a 375 km SMF consisting of five spans G.652 fiber, and the length of each span was 75 km. Moreover, each span was equipped with an 18 dB gain EDFA to compensate for the power loss due to signal attenuation over the span.
At the receiver end, an ECL with a linewidth of 100 kHz was utilized as the local oscillator (LO). The received signal was detected by the integrated coherent receiver (ICR) to achieve the phase diversity of the signal. After that, ICR and LO achieved optical-toelectrical conversion. The digital phosphor oscilloscope (DPO72004C) with a sampling rate of 100 GSa/s was used to implement analog-to-digital conversion. Then, the sampled data sequences were processed by offline digital signal processing (DSP). At the same time, an optical spectrum analyzer (OSA) was employed to monitor the OSNR of received optical signals. Figure 5a shows the DSP flow chart for uniform received signals (uniform S-64QAM and uniform GS-64QAM). The processing flow includes lowpass filtering, amplitude normalization, Gram-Schmidt orthogonalizing process (GSOP), chromatic dispersion (CD)/nonlinear compensation, clock recovery using Gardner algorithm, adaptive equalization using constant modulus algorithm (CMA), frequency offset estimation (FOE), carrier phase estimation (CPE) based on blind phase search (BPS), synchronization, 64QAM demapping, and bit error rate (BER) calculation. Figure 5b shows the DSP flow chart for received HS-64QAM signals, which consists of lowpass filtering, normalization, GSOP, CD/nonlinear compensation, clock recovery using Gardner, adaptive equalization using CMA, FOE, CPE, BPS, synchronization, 64QAM demapping, de-distribution matcher (de-DM), and BER calculation. The transmission link was a 375 km SMF consisting of five spans G.652 fiber, and the length of each span was 75 km. Moreover, each span was equipped with an 18 dB gain EDFA to compensate for the power loss due to signal attenuation over the span.
At the receiver end, an ECL with a linewidth of 100 kHz was utilized as the local oscillator (LO). The received signal was detected by the integrated coherent receiver (ICR) to achieve the phase diversity of the signal. After that, ICR and LO achieved optical-toelectrical conversion. The digital phosphor oscilloscope (DPO72004C) with a sampling rate of 100 GSa/s was used to implement analog-to-digital conversion. Then, the sampled data sequences were processed by offline digital signal processing (DSP). At the same time, an optical spectrum analyzer (OSA) was employed to monitor the OSNR of received optical signals. Figure 5a shows the DSP flow chart for uniform received signals (uniform S-64QAM and uniform GS-64QAM). The processing flow includes lowpass filtering, amplitude normalization, Gram-Schmidt orthogonalizing process (GSOP), chromatic dispersion (CD)/nonlinear compensation, clock recovery using Gardner algorithm, adaptive equalization using constant modulus algorithm (CMA), frequency offset estimation (FOE), carrier phase estimation (CPE) based on blind phase search (BPS), synchronization, 64QAM demapping, and bit error rate (BER) calculation. Figure 5b shows the DSP flow chart for received HS-64QAM signals, which consists of lowpass filtering, normalization, GSOP, CD/nonlinear compensation, clock recovery using Gardner, adaptive equalization using CMA, FOE, CPE, BPS, synchronization, 64QAM demapping, de-distribution matcher (de-DM), and BER calculation. Figure 6 shows the BERs of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus OSNR in the 120 Gbit/s coherent optical communication system for the BTB situation. A VOA and an EDFA were applied to adjust the OSNR for the BER measurement. The recovered constellations for the three 64QAMs at 20 dB OSNR are plotted in the insets of Figure 6a-c. It is shown that constellations of uniform GS-64QAM and uniform S-64QAM are not conducive to the decision compared to the proposed HS-64QAM with the same transmission power. When the OSNR is 20 dB, the BER in Figure 6a-c is 0.0237, 0.0158, and 0.0029, respectively. For all the three modulation formats, the BER decreases as the OSNR gradually increases. It can be clearly seen that HS-64QAM outperforms the uniform GS-64QAM and the uniform S-64QAM in the measured OSNR range of 16 dB to 21.5 dB for the BTB situation. Furthermore, the soft-decision forward error correction (SD-FEC) threshold of 2 × 10 −2 and hard-decision  Figure 6 shows the BERs of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus OSNR in the 120 Gbit/s coherent optical communication system for the BTB situation. A VOA and an EDFA were applied to adjust the OSNR for the BER measurement. The recovered constellations for the three 64QAMs at 20 dB OSNR are plotted in the insets of Figure 6a-c. It is shown that constellations of uniform GS-64QAM and uniform S-64QAM are not conducive to the decision compared to the proposed HS-64QAM with the same transmission power. When the OSNR is 20 dB, the BER in Figure 6a-c is 0.0237, 0.0158, and 0.0029, respectively. For all the three modulation formats, the BER decreases as the OSNR gradually increases. It can be clearly seen that HS-64QAM outperforms the uniform GS-64QAM and the uniform S-64QAM in the measured OSNR range of 16 dB to 21.5 dB for the BTB situation. Furthermore, the soft-decision forward error correction (SD-FEC) threshold of − × 2 2 10 and hard-decision forward error correction (HD-FEC) threshold of − × 3 3.8 10 are commonly used to evaluate BER performance for optical communication systems. Therefore, these two values are applied to measure the performance in the experimental verification.

Results and Analysis
As shown in Figure 6, with an OSNR of 19.6 dB, the BER of uniform GS-64QAM can reach below the SD-FEC threshold, while the BER performance of HS-64QAM and uniform S-64QAM is 0.0043 and 0.03, respectively. It can be clearly seen that a considerably better OSNR sensitivity can be obtained by introducing the proposed scheme. Taking the SD-FEC threshold as a reference, the HS-64QAM signal obtains an OSNR gain of 1.9 dB and 2.7 dB, respectively, compared to the uniform GS-64QAM signal and the uniform S-64QAM signal. When BER = 10 −2 , HS-64QAM obtains an OSNR gain of 2 dB and 2.8 dB over uniform GS-64QAM and uniform S-64QAM, respectively. When the OSNR is greater than 19.7 dB, the BER of HS-64QAM reaches below the HD-FEC threshold. Utilizing the proposed scheme can enhance the system performance for 64QAM. This is because the proposed HS-64QAM scheme realizes the combination of GS and PS dominance, which further improves the constellation shaping gain.   As shown in Figure 6, with an OSNR of 19.6 dB, the BER of uniform GS-64QAM can reach below the SD-FEC threshold, while the BER performance of HS-64QAM and Photonics 2023, 10, 1008 9 of 13 uniform S-64QAM is 0.0043 and 0.03, respectively. It can be clearly seen that a considerably better OSNR sensitivity can be obtained by introducing the proposed scheme. Taking the SD-FEC threshold as a reference, the HS-64QAM signal obtains an OSNR gain of 1.9 dB and 2.7 dB, respectively, compared to the uniform GS-64QAM signal and the uniform S-64QAM signal. When BER = 10 −2 , HS-64QAM obtains an OSNR gain of 2 dB and 2.8 dB over uniform GS-64QAM and uniform S-64QAM, respectively. When the OSNR is greater than 19.7 dB, the BER of HS-64QAM reaches below the HD-FEC threshold. Utilizing the proposed scheme can enhance the system performance for 64QAM. This is because the proposed HS-64QAM scheme realizes the combination of GS and PS dominance, which further improves the constellation shaping gain. Figure 7 shows the BER curves of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus OSNR in the 120 Gbit/s coherent optical communication system for the 375 km transmission situation. The recovered constellations for the three 64QAMs at 23 dB OSNR are plotted in the insets of Figure 7a-c. When the OSNR is 23 dB, the BERs of the proposed HS-64QAM with H = 5.7 bits/symbol, uniform GS-64QAM, and uniform S-64QAM are 0.0024, 0.019, and 0.032. It is also shown that the recovered constellation for the proposed HS-64QAM at the same transmission power is more favorable to be distinguished compared to uniform S-64QAM and uniform GS-64QAM.    As shown in Figure 7, the BER of the three 64QAMs decreases as the OSNR gradually increases. It can be clearly seen that the proposed HS-64QAM scheme is superior to the uniform GS-64QAM and the uniform S-64QAM throughout the entire OSNR range from 17 dB to 30 dB for the 375 km transmission situation. Since the uniform GS-64QAM constellation is optimized, the min-ED of the uniform GS-64QAM constellation is the largest. Probabilistically shaped on the basis of uniform GS-64QAM, the proposed HS-64QAM can effectively reduce the transmission power. Compared with uniform GS-64QAM and uniform S-64QAM, HS-64QAM obtains 4.1 dB and 7.6 dB OSNR gains, respectively, at the BER of the SD-FEC threshold. When BER = 10 −2 , HS-64QAM obtains a 5.6 dB OSNR gain over uniform GS-64QAM. In addition, when the OSNR is 21.9 dB, the BER of HS-64QAM reaches the HD-FEC threshold, while the best BER performance of uniform GS-64QAM and uniform S-64QAM is 0.026 and 0.04, respectively. Experimental results in both BTB and 375 km transmission scenarios indicate that the proposed HS-64QAM scheme is more reliable than the uniform GS-64QAM and the uniform S-64QAM when adapted to transmission systems with the same transmission power. Figure 8 shows the BER curves of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus TOP in the 120 Gbit/s coherent optical communication system for the 375 km transmission situation. The TOP transmitted into the SMF fiber link was managed by a VOA and an EDFA to measure BER curves. The recovered constellations for the three 64QAMs at −6 dBm TOP are plotted in the insets of Figure 8a-c. When the TOP into the SMF is −6 dBm, the BERs of the proposed HS-64QAM with H = 5.7 bits/symbol, uniform GS-64QAM, and uniform S-64QAM are 0.0115, 0.028, and 0.036. It is worth observing from the diagrams of recovered constellations that the points in the corners of uniform S-64QAM are blended together and hard to distinguish. However, the outermost constellation points of the proposed scheme have higher discrimination, which again indicates the superiority of the proposed scheme. BER decreases as TOP increases in the measured TOP range from −11 dBm to −4 dBm. The reason is that the OSNR of the received signal is positively correlated with TOP transmitted into the SMF, and the amplifier spontaneous emission noise is the main factor affecting BER over the entire range of TOP measurements.
Probabilistically shaped on the basis of uniform GS-64QAM, the proposed HS-64QAM can effectively reduce the transmission power. Compared with uniform GS-64QAM and uniform S-64QAM, HS-64QAM obtains 4.1 dB and 7.6 dB OSNR gains, respectively, at the BER of the SD-FEC threshold. When BER = 10 −2 , HS-64QAM obtains a 5.6 dB OSNR gain over uniform GS-64QAM. In addition, when the OSNR is 21.9 dB, the BER of HS-64QAM reaches the HD-FEC threshold, while the best BER performance of uniform GS-64QAM and uniform S-64QAM is 0.026 and 0.04, respectively. Experimental results in both BTB and 375 km transmission scenarios indicate that the proposed HS-64QAM scheme is more reliable than the uniform GS-64QAM and the uniform S-64QAM when adapted to transmission systems with the same transmission power. Figure 8 shows the BER curves of 64QAM (uniform S-64QAM, uniform GS-64QAM, and HS-64QAM with H = 5.7 bits/symbol) versus TOP in the 120 Gbit/s coherent optical communication system for the 375 km transmission situation. The TOP transmitted into the SMF fiber link was managed by a VOA and an EDFA to measure BER curves. The recovered constellations for the three 64QAMs at −6 dBm TOP are plotted in the insets of Figure 8a-c. When the TOP into the SMF is −6 dBm, the BERs of the proposed HS-64QAM with H = 5.7 bits/symbol, uniform GS-64QAM, and uniform S-64QAM are 0.0115, 0.028, and 0.036. It is worth observing from the diagrams of recovered constellations that the points in the corners of uniform S-64QAM are blended together and hard to distinguish. However, the outermost constellation points of the proposed scheme have higher discrimination, which again indicates the superiority of the proposed scheme. BER decreases as TOP increases in the measured TOP range from −11 dBm to −4 dBm. The reason is that the OSNR of the received signal is positively correlated with TOP transmitted into the SMF, and the amplifier spontaneous emission noise is the main factor affecting BER over the entire range of TOP measurements.  As illustrated in Figure 8, when achieving the same BER performance, the TOP required for HS-64QAM is optimal, followed by uniform GS-64QAM and then uniform S-64QAM. Taking the 4 × 10 −2 BER threshold as a reference, the TOP performance of the HS-64QAM outperforms uniform GS-64QAM and uniform S-64QAM by 2.11 dBm and 2.71 dBm, respectively. When TOP is −8.8 dBm, BER of HS-64QAM reaches the SD-FEC threshold, while the best BER performance is 0.043 and 0.048 for uniform GS-64QAM and uniform S-64QAM, respectively. Furthermore, the BER of HS-64QAM reaches the 10 −2 threshold at TOP = −4.9 dBm. This is due to the superiority of the proposed HS-64QAM scheme, which reduces signal transmission power and confirms high-capacity transmission performances.

Discussion
The reason why the proposed HS-64QAM scheme can achieve a high performance is that the proposed scheme combines the advantages of GS and PS to achieve a higher constellation shaping gain. By utilizing the hexagonal lattice structure, which is the densest pattern in the complex plane, denser constellation points are achieved within a fixed min-ED constellation plane for 64QAM. This densification enhances the error tolerance compared to the traditional S-64QAM. Moreover, the transmission link is limited in distance due to the non-desired tolerance to nonlinearity in the traditional 64QAM scheme. The proposed HS-64QAM scheme optimizes the coordinates and probabilistic distribution of constellation points. Compared with the traditional S-64QAM, GS-64QAM achieves a certain CFM gain. On the basis of GS, HS-64QAM realizes probabilistic optimization and the average-signalpower reduction. In this way, not only is the resistance to nonlinearity improved, but also constellation shaping gain is enhanced. Consequently, we can conclude that the proposed scheme exhibits superior transmission performance compared to S-64QAM.
As can be seen from Figures 6 and 7, the proposed HS-64QAM scheme consistently achieves superior transmission performance as the number of OSNRs increases in both BTB and 375 km transmission scenarios. In addition, it can also be seen from Figure 8 that the BER performance of the proposed scheme can satisfy the SD-FEC threshold with TOP above −8.8 dBm. These results highlight the significant potential of the proposed HS-64QAM scheme in enhancing the performance of high-capacity coherent optical communication systems. With the in-depth development of future communication technologies such as the fifth/sixth generation, the explosion of network traffic will require further improvement of transmission capacity. In this case, the proposed HS-64QAM scheme has been demonstrated to be superior and feasible. Therefore, the proposed scheme holds substantial application value in long-haul, high-sensitivity, high-speed, and even ultra-high-speed optical communications.
To address the diverse requirements of practical application scenarios, the proposed HS-64QAM scheme can be extended to other modulation orders. More schemes such as the HS-16QAM scheme, the HS-32QAM scheme, and the HS-128QAM scheme will be explored. In order to verify the effectiveness of the proposed schemes, we will perform verifications based on actual scenarios in subsequent studies. At the same time, different application scenarios will be considered to ensure the reliability of the above schemes. Moreover, machine learning technology has gained widespread adoption in the field of optical communications, presenting extensive opportunities for achieving higher-speed and more flexible data transmission. The utilization of data training sets allows for advanced optimization and enhanced performance in optical communication systems. By combining the proposed HS-64QAM scheme with machine learning technology, the intelligent and adaptive constellation shaping can be realized at the transmitter side. The approach will involve adaptively adjusting the geometric and probabilistic distributions of the constellation based on channel characteristics. Therefore, combining the proposed hybrid constellation shaping scheme with machine learning technology is also one of the promising directions for future research.

Conclusions
A novel HS-64QAM based on hexagonal lattice of a constellation subset is proposed to enhance the nonlinear tolerance of higher-order modulated signals and further improve the constellation shaping gain. Firstly, the initial quantitative characterization of the constellation is performed on the basis of the hexagonal lattice structure. Then, the position distribution of corresponding constellation points is obtained by maximizing the CFM as the objective function to generate GS-64QAM. Finally, HS-64QAM is obtained by dividing all constellation points into multiple subsets according to concentric hexagonal layers, where points within the same subset are endowed with the same probability. In addition, the experimental verification of the proposed HS-64QAM scheme was demonstrated in a 120 Gbit/s multi-span coherent optical communication system. Experimental results indicate that HS-64QAM achieves an OSNR gain of 1.9 dB and 4.1 dB compared to uniform GS-64QAM in the BTB and transmission scenarios, respectively, taking the SD-FEC threshold as a reference. Moreover, HS-64QAM achieves an OSNR gain of 2.7 dB and 7.6 dB compared with uniform S-64QAM in BTB and transmission scenarios, respectively. Therefore, the proposed scheme has demonstrated its superiority and can be used to realize high-speed optical transmission over longer distance and with higher sensitivity, providing a viable means for the future development of ultra-high-speed optical communication networks.