Vortex Beam Transmission Compensation in Atmospheric Turbulence Using CycleGAN

Abstract

To improve the robustness of vortex beam transmission and detection in the face of atmospheric turbulence and to guarantee accurate recognition of orbital angular momentum (OAM), we present an end-to-end dynamic compensation technique for vortex beams using an improved cycle-consistent generative adversarial network (CycleGAN). This approach transforms the problem of vortex beam distortion compensation into one of image translation. The Pix2pix and CycleGAN models were extended with a structural similarity loss function to constrain turbulence distortion compensation in luminance, contrast and structure. Experiments were designed to evaluate the compensation performance from subjective and objective indicators. The simulation results demonstrate that the optical OAM intensity map is very similar to that of the target OAM light after compensation. The mean value of structural similarity is close to 1. The recognition accuracy of the OAM is improved by 4.4% compared to no distortion compensation, demonstrating that the improved CycleGAN-based compensation scheme can guarantee excellent detection accuracy without reconstructing the wavefront and saving optical hardware. The method can be implemented in real-time optical communications in atmospheric turbulence environments.

1. Introduction

It is well known that the electromagnetic wave carries energy, momentum and angular momentum. The angular momentum has a spin part associated with polarization and an orbital part associated with spatial distribution. The existing information modulation in communications is mainly carried out in the time, frequency and polarization domains; vortex beams carrying orbital angular momentum (OAM) provide a new dimension for communication [1,2,3,4,5]. The OAM has an infinite number of eigenmodes, theoretically allowing the construction of infinite dimensional Hilbert space, which greatly increases the capacity of communication systems [6]. The OAM multiplexing can greatly broaden the bandwidth of wireless communication and thus improve the spectrum utilization [7].

Unfortunately, OAM will suffer heavy distortion from atmospheric turbulence (AT) when it is used in free space optical communication [8,9,10,11,12]. The AT will disturb the wave front phase, damage the orthogonality among OAM modes and further degrade the communication performance and detection accuracy. To improve the robustness of the vortex beam communication and detection system to the influence of AT distortion and ensure the subsequent high-precision OAM mode identification, it is necessary to dynamically compensate for the distorted beam.

The vortex beam turbulence distortion compensation methods are mainly divided into those with and without a probe and wavefront sensor. In 2007, a Gaussian mutation genetic algorithm to correct the wavefront phase of the vortex beam was proposed. Skipping the reconstruction of the wavefront phase, the scheme directly controls the deformable mirror for correction and iteratively finds the optimal surface shape of the deformable mirror for wavefront correction [13]. In [14], a wavefront correction method based on the Shack–Hartmann wavefront sensor was proposed, which improved the purity of the fundamental mode beam at the receiving end and improved the accuracy of OAM mode identification. The wavefront correction technique based on the simulated annealing algorithm was proposed, and the Zernike polynomials were used to represent the wavefront phase [15]. Ren et al. used the Gerchberg–Saxton (GS) phase recovery algorithm to deal with the wavefront correction at the receiving end of the vortex optical communication system [16]. They also launched a fundamental mode Gaussian beam as a probe beam at the transmitting end for the wavefront correction problem [17]. A vortex beam wavefront correction technique based on stochastic parallel gradient descent was investigated in [18]. Fu et al. proposed a wavefront correction technique based on the GS phase recovery algorithm in the framework of a vortex optical communication system with probe beams which has a good effect on probe beam, single-mode and superposition vortex beam correction [19].

The abovementioned vortex beam wavefront correction methods are almost based on the adaptive optics system without wavefront detection. The control algorithm does not have learning and memory capabilities, and it may fall into the local optima. With the development of machine learning, deep learning-based vortex beam wavefront correction has attracted more and more attention. Deep learning-based wavefront compensation does not require a Shack–Hartmann wavefront sensor, which is not subject to turbulence intensity constraints. A novel joint AT detection and adaptive demodulation technique based on the convolutional neural network (CNN) was proposed for the OAM-based free-space optical communication [20]. Tian et al. mapped the distributed features learned from the OAM light intensity distribution to the sample label space based on the CNN. The Zernike coefficients of the first 2–400 order are output and converted into the control signal of the spatial light modulator for correction [21]. Based on CNN, Liu et al. proposed a scheme to predict the turbulent phase screen by training the mapping relationship between the intensity distribution of vortex beam and the turbulent phase, which has a strong generalization performance [22]. In [23], a CNN-based wavefront reconstruction technique to train the mapping relationship between the vortex beam intensity map and the first 20-order Zernike coefficients was carried out. A six-layer CNN model to achieve high recognition accuracy of Laguerre–Gaussian beams under different ATs was designed in [24]. The LG beam OAM-mode detection by CNN was studied experimentally in [25]. Zhou et al. realized the identification of OAM in AT based on ShuffleNet V2 for single and multiplexing positive OAM modes [26]. In [27], a vortex phase modulation method was introduced to achieve both integer topological charges and fractional ones even under a strong turbulence intensity and long transmission distance.

The core of the existing deep learning-based wavefront correction method is the wavefront reconstruction algorithm. Based on the probe beam, the wavefront is reconstructed by training the mapping relationship between the light intensity map of the probe beam and the turbulent Zernike coefficient. However, the method of wavefront reconstruction based on CNN ignores a lot of high frequency information to some extent and cannot be applied to non-circular aperture reconstruction, and the reconstruction capability is limited. As a hotspot in machine learning, the generative adversarial network (GAN) adopts adversarial training methods to generate more realistic samples [28,29,30]. GAN can be utilized in adaptive optics compensation to implement image-to-image translation. Based on condition GAN, Pix2pix trained the data in pairs, and the L1 loss function was added to realize domain adaptation, which is a classical image translation algorithm [31]. In [32], CycleGAN was proposed which only requires data from two domains, without requiring strict correspondence between them as Pix2pix does, making the application of the CycleGAN more widespread. To address the existing deep learning-based vortex beam wavefront correction technique that requires probe beams and numerous optical hardware resources, as well as reconstructing the wavefront before correction, the paper proposes an end-to-end turbulence distortion compensation technique based on improved CycleGAN to directly output the compensated OAM light intensity map.

In this paper, a vortex beam distortion compensation technique in AT based on improved CycleGAN is proposed from the perspective of image processing for OAM communication. The structural similarity (SSIM) loss function term is added to the Pix2pix and CycleGAN loss function to realize end-to-end AT distortion compensation in Section 2. The experimental results on the mean structural similarity (MSSIM), vortex beam intensity distributions and difference with and without compensation, and OAM state detection accuracy are presented in Section 3. Section 4 is devoted to the conclusion.

2. Atmospheric Turbulence Compensation Scheme

2.1. Atmospheric Turbulence Distortion Theory

Due to the existence of temperature gradient, AT causes the atmospheric refractive index to be randomly distributed, which will lead to scintillation and phase fluctuation of the vortex beam passing through the turbulence. The existing refractive index spatial power spectrum functions mainly include the Kolmogorov spectrum [33], Tatarskii spectrum [34], Hill–Andrews spectrum [35], etc. Based on the power exponent 2/3 law in inertial region, the Kolmogorov spectrum can be expressed as follows:

$${\varphi }_{\mathrm{n}}(\kappa )=0.033C_n^2{k}^{-11/3},\begin{array}{ccc}& & \end{array}1/L_0\le k\le 1/l_0$$

(1)

where $C_n^2$ represents the atmospheric refractive index structure constant, which is used to measure the turbulence intensity. The larger the value, the more serious the distortion of the beam passing through the turbulence. Moreover, $k$ is the wavenumber, and $L_0$ and $l_0$ represents the outer and inner scale.

When simulating the influence of AT distortion, the perturbation of the vortex beam by a single-phase screen can be used to simulate the phase change of the vortex beam after passing through a turbulent channel at a certain distance, as shown in Figure 1. The whole transmission process is equivalent to many phase screens with a certain distance [36]. There are two main methods to generate a random phase screen: the Zernike polynomial expansion method and the power spectrum inversion method [37,38].

The specific steps to describe the random distribution of phase disturbance in a Cartesian coordinate system are as follows: First, generate a complex random number matrix with a mean of 0 and variance of 1; use the phase power spectrum function, ${\varphi }_{\phi }(k)$, conforming to the Kolmogorov spectrum to filter; and then use the fast Fourier transform (FFT) to map and finally get the turbulent phase screen in the frequency domain. The relationship between the phase power spectrum function, ${\varphi }_{\phi }(k)$, and the refractive index spectral density function, ${\varphi }_n(k)$, is as follows:

$${\varphi }_{\phi }(k)=2\pi k_0^2\Delta z{\varphi }_n(k)$$

(2)

where $\Delta z$ represents the interval between the phase screens. The wavenumber is $k_0=2\pi /\lambda$, and the wavelength of the light beams is $\lambda$. The variance of the frequency domain distribution of the phase screens is as follows:

$${\sigma }^2(k)={(\frac{2\pi }{N\Delta x})}^2{\varphi }_{\phi }(k)$$

(3)

where N represents the grid number of the complex random number matrix, and $\Delta x$ represents the grid spacing. The fast Fourier transform is used to calculate the turbulent phase screen as follows:

$$\phi (x,y)=FFT(C\cdot \sigma (k))$$

(4)

where C represents the complex random number of N × N matrix, and FFT is the fast Fourier transform.

2.2. Vortex Beam Distortion Compensation Based on Pix2pix

Different from traditional optical communication systems, vortex beam communication loads information into different OAM states for transmission. Here, OAM mode recognition based on deep learning is realized by classifying the two-dimensional OAM light intensity maps. Without predicting the complete wavefront phase, only generating the complete OAM light intensity map for subsequent mode identification is enough. This provides a theoretical basis for the mapping of the distorted OAM beam intensity map and the target OAM beam intensity map based on the GAN training in this paper. The Pix2pix and CycleGAN flowchart adopted U-Net [39] architecture, which is more suitable for the phase recovery task, as shown in Figure 2.

The framework of end-to-end distortion compensation based on Pix2pix is shown in Figure 3. After the receiver captures the distorted OAM light intensity map, it is fed into the Pix2pix network, and the compensated OAM light intensity map is directly output for subsequent state detection tasks. Each sub-layer of the generator and discriminator of Pix2pix consists of three parts: a convolution layer, a batch normalization layer and an activation function layer. The size of the convolution kernel of the convolution layer is 4 × 4. The generator contains a total of 16 layers of the network and adopts a U-Net-like structure. The E1 to E8 layers down-sample the input OAM light intensity distribution features through convolution, and the interlayer activation function selects Leaky Relu. D8 to D1 layers perform an up-sampling operation on the input OAM light intensity distribution features through deconvolution, and the activation function between layers selects Relu. The corresponding down-sampling E layers and up-sampling D layer use skip connections to transfer the high-frequency information learned in the encoder layer to the corresponding decoder layer to prevent the loss of important high-frequency image information in the down-sampling process. The compensated OAM light intensity map output by the generator is normalized and sent to the discriminator, and its authenticity is judged in combination with the target OAM light intensity map. The discriminator contains 5 layers of network, and the activation function between layers selects the Leaky Relu.

During the training process, the size of the OAM light intensity map is uniformly set to 256 × 256 × 3, the input image is flipped horizontally with a certain probability and the Xavier method is used to evenly distribute the network weight coefficients. The input of the E1 layer is 256 × 256 × 3. After the “convolution-batch normalization-function activation” operation, the size of the output feature map becomes 128 × 128 × 3. From layer E2 to E8, the size of the feature image doubles after each layer. The output feature map of the E8 layer is 1 × 1 × 256 in size and is fed into the D8 layer. After each layer from layer D8 to D1, the size of the feature image is doubled. Finally, the output size of the D8 layer is 256 × 256 × 3, which restores the original image size. The OAM light intensity map generated by the generator is input into the discriminator. The first four layers of the discriminator are used to extract features, and then a convolution layer with a convolution kernel size of 4 × 4 is used for down-sampling to reduce the resolution of the feature map to 1 × 1. The final output vector size is 1 × 1 × 2, representing the discrimination result. The Adam optimizer is used during training, and the Adam momentum term is set to 0.9. The learning rates of the generator and discriminator are both set to 0.002, and the regularization weight in the objective function of the generator is set to 100. A total of 50 epochs were trained, and the batch size was set to 2.

Traditionally, the performance of AT distortion compensation is evaluated by using the Strehl Ratio, mean square error and peak-to-valley values. The purpose of the vortex beam AT compensation is to provide a high-quality light intensity distribution for subsequent OAM state detection. Therefore, the loss function of Pix2pix is modified here. In addition to Pix2pix’s the traditional condition GAN loss function and the $L_1$ loss function, a SSIM loss function term is introduced. The two intrinsic parts of the Pix2pix loss function evaluate the difference between the distribution of the generated data and the real data at the pixel level. Meanwhile the SSIM loss function can assess the image quality of the generated vortex beam in terms of luminance, contrast and structure [40]. The mean structural similarity (MSSIM) is adopted to measure the image quality after compensation.

SSIM is an evaluation criterion that is used to measure the similarity between two images. Here, the two images are intensity distributions of the undistorted target vortex beam and the distorted vortex beam. SSIM includes three indicators, luminance $l(x,y)$, contrast $c(x,y)$ and structure $s(x,y)$, which are expressed as follows [40]:

$$l(x,y)=\frac{2{\mu }_x{\mu }_y+c_1}{{\mu }_x^2+{\mu }_y^2+c_1}$$

(5)

$$c(x,y)=\frac{2{\sigma }_x{\sigma }_y+c_2}{{\sigma }_x^2+{\sigma }_y^2+c_2}$$

(6)

$$s(x,y)=\frac{{\sigma }_{xy}+c_3}{{\sigma }_x{\sigma }_y+c_3}$$

(7)

where $c_1={(k_{1}L)}^2$, $c_2={(k_{2}L)}^2$ and $c_3=c_2/2$. L represents the range of image pixel values, and $k_1$ and $k_2$ are 0.01 and 0.03, respectively. Moreover, x and y represent two image signals; ${\mu }_x$ and ${\mu }_y$ refer to the average pixel values of image x and image y; ${\sigma }_x$ and ${\sigma }_y$ refer to the standard deviation of pixel values of image x and image y, respectively; and ${\sigma }_{xy}$ is the covariance of pixel values of image x and image y. Assuming that the image signal is discrete, the related parameters above are calculated as follows:

$${\mu }_x=\frac{1}{MN}{\displaystyle \sum _{j=1}^M{\displaystyle \sum _{i=1}^{N}y(i,j)}}$$

(8)

$${\sigma }_x^2=\frac{1}{MN-1}{\displaystyle \sum _{j=1}^M{\displaystyle \sum _{i=1}^N{(y(i,j)-{\mu }_x)}^2}}$$

(9)

$${\sigma }_{xy}=\frac{1}{MN-1}{\displaystyle \sum _{j=1}^M{\displaystyle \sum _{i=1}^N(y(i,j)-{\mu }_x)(G(x(i,j))-{\mu }_y)}}$$

(10)

Substituting the above parameters into Pix2pix network assessment Equations $\mathrm{SSIM}(x,y)=l{(x,y)}^{\alpha }\cdot c{(x,y)}^{\beta }\cdot s{(x,y)}^{\gamma }$,where $\alpha ,\beta ,\gamma$ represent the proportion of different features in SSIM measurement, usually taken as $\alpha =\beta =\gamma =1$, the SSIM index can be written as follows [40]:

$$\mathrm{SSIM}(x,y)=\frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}$$

(11)

The value of SSIM is between 0 and 1. The closer the value is to 1, the better the turbulence distortion compensation effect is, and the higher the image quality of the light intensity distribution of vortex beam after compensation. The loss function term of SSIM can be written as follows:

$$L_{SSIM}(G)=E_{x,y~p_{data(x,y)}}[1-SSIM(y,G(x))]$$

(12)

The final total loss function includes three parts: traditional condition GAN loss, $L_1$ loss function and SSIM loss:

$$L=\arg \underset{G}{\min }\underset{D}{\max }{L}_{cGAN}(G,D)+\alpha L_{L1}(G)+\beta L_{SSIM}(G)$$

(13)

where $\alpha$ represents the weight of generator $L_1$ loss function, and $\beta$ represents the weight of generator structural similarity loss function.

2.3. Vortex Beam Distortion Compensation Based on Improved CycleGAN

The principle of end-to-end distortion compensation based on CycleGAN is shown in Figure 4. The structure of the CycleGAN generator and discriminator, the parameters of each layer and the parameter settings in the training process are the same as Pix2pix. For vortex beam AT distortion compensation, image x of the distortion domain X is the distorted OAM light intensity map captured by the receiver, and image y of the target domain Y is the undistorted target OAM light intensity map. During training, firstly, the distorted OAM light intensity map x passes through the generator G from the X domain to the Y domain and then passes through the generator F from the Y domain to the X domain to obtain the reconstructed OAM light intensity map F(G(x)) in the distortion domain. It makes the reconstructed OAM light intensity map as close to the original image x as possible. The same is true for the process of migration from the target domain Y to the distortion domain X, so that the reconstructed target OAM light intensity map is as close as possible to the original image y, and the reconstruction loss is based on the loss. The discriminators of the two unidirectional GANs were used to discriminate the reconstructed image and the original image, respectively.

The same as the Pix2pix in Section 2.2, the SSIM loss function is added to the generator loss of the CycleGAN, namely the improved CycleGAN. Since CycleGAN consists of two symmetrically mirrored classical GANs, a SSIM loss function term is also added to the generator loss function for each of the two directions:

$$\begin{array}{ll}{L}_{SSIM}(G,F)& =E_{x~P_{data(x)}}[1-SSIM(x,F(G(x)))]\\ & +E_{y~P_{data(y)}}[1-SSIM(y,G(F(y)))]\end{array}$$

(14)

For the vortex beam distortion compensation, the total loss function of the improved CycleGAN includes the loss of the two classical GANs, the loss of the generator $L_1$ in both directions and the loss of the SSIM, written as follows:

$$\begin{array}{ll}L& =\arg \underset{G,F}{\min }\underset{D_X,D_Y}{\max }{L}_{GAN}(G,D_Y,X,Y)+L_{GAN}(F,D_X,Y,X)\\ & +\alpha L_{cycle}(G,F)+\beta L_{SSIM}(G,F)\end{array}$$

(15)

where $\alpha$ denotes the weights of the total $L_1$ loss function term in both directions, and $\beta$ represents the weights of the total SSIM function loss term in both directions.

3. Results and Discussions

3.1. Evaluation Index of Turbulence Distortion Compensation

The core of the previous wavefront correction scheme is the wavefront reconstruction algorithm. It adopts many quality evaluation indicators such as Strehl Ratio, mean square error and peak to valley values to measure the performance of the wavefront correction system and wavefront reconstruction error fluctuation. We utilize the improved CycleGAN to directly train the end-to-end mapping between the light intensity distribution of vortex beam with and without compensation. It skips the wavefront reconstruction and dynamic compensation of optical devices and turns the problem of vortex beam turbulence distortion compensation into an image repair problem. Combined with the image translation study, the effect of turbulence distortion compensation can be evaluated by the image restoration method. Thus, MSSIM is adopted here to evaluate the compensation scheme.

The turbulence distortion compensation evaluation index used in this paper can be divided into subjective evaluation and objective evaluation. Subjective evaluation refers to the subjective feeling of the human eye on the compensation effect of light intensity distribution, and it is a subjective evaluation of the similarity between the light intensity distribution of the compensated vortex beam and the target vortex beam. However, whether in the actual scientific research or industrial implementation, objective evaluation criteria are still needed. Thus, MSSIM, OAM state detection probability distribution at the receiving end before and after compensation and the OAM state detection accuracy with and without compensation were selected to evaluate the compensation performance of the proposed scheme.

3.2. Simulation Dataset Construction

The experimental data is generated by physical simulation, and the OAM state set with a unified image style (petal distribution) is selected. We selected the following 10 superposition states of OAM: {$1,-1$}, {$1,-2$}, {$2,-2$}, {$2,-3$}, {$3,-3$}, {$3,-4$}, {$4,-4$}, {$4,-5$}, {$5,-5$} and {$6,-6$}, as shown in Figure 5. The distorted OAM light intensity map captured at the receiving end and the corresponding undistorted target OAM light intensity map form an image pair. The wavelength is 0.6328 um, and the beam waist radius is 0.3 m. The AT distortion compensation experiments in the medium turbulence and the strong turbulence environments were carried out, respectively.

In a medium turbulence, the atmospheric refractive index structure constant $C_n^2$ ranges from $1.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$ to $5.5\times {10}^{-14} {\mathrm{m}}^{-2/3}$ at intervals of $5.0\times {10}^{-15} {\mathrm{m}}^{-2/3}$. In a strong turbulence, $C_n^2$ ranges from $1.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ to $5.5\times {10}^{-13} {\mathrm{m}}^{-2/3}$, at intervals of $5\times {10}^{-14} {\mathrm{m}}^{-2/3}$. When simulating the effects of AT distortion, ten turbulence-phase screens with a certain interval are obtained by dividing the transmission distance into ten equal parts based on the power spectral inversion method. Under different ATs, 200 light intensity maps are generated for each OAM state. The data set of medium and strong turbulence-mixed OAM states contains 20,000 samples and is divided into a training and test set at a ratio of 8:2.

3.3. Analysis of Simulation Results

MSSIM was used to evaluate the compensation performance of the vortex beam turbulence distortion compensation based on Pix2pix, CycleGAN, improved Pix2pix and improved CycleGAN. Here, improved Pix2pix and improved CycleGAN represent the network with the addition of a SSIM loss function term, marked as Pix2pix + SSIM_loss and CycleGAN + SSIM_loss, respectively. In both turbulent environments, as shown in Figure 6, the MSSIM increases with the training process and tends to be above 90%, indicating that the intensity distribution of the compensated vortex beam is close to that of the target vortex beam. Compared with Pix2pix, the MSSIM of CycleGAN is higher for the intensity distribution of the compensated vortex beam in strong turbulence. Because CycleGAN uses two unidirectional GANs with mirror symmetry, it is more suitable for cases where the data are not one-to-one symmetrical to the pixel points. Moreover, the MSSIM of the Pix2pix + SSIM_loss and CycleGAN + SSIM_loss method was improved obviously by adding the SSIM loss function term, especially in strong turbulence. As shown in

Table 1. Comparison of the compensation performance based on various methods.

Distortion Compensation Network	MSSIM (Medium Turbulence)	MSSIM (Strong Turbulence)
Pix2pix	0.944	0.903
Pix2pix + SSIM_Loss	0.949	0.922
CycleGAN	0.952	0.918
CycleGAN + SSIM_Loss	0.954	0.931

, the MSSIM using the CycleGAN + SSIM_loss compensation method has a 1% and 2.8% improvement compared with that of Pix2pix in medium and strong AT, respectively.

The effect of vortex beam AT compensation based on CycleGAN is shown in Figure 7 for when the transmission distance is 4000 m and $C_n^2=3.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ in strong turbulence. The first column is the intensity distribution of the vortex beam captured at the receiving end, and the second column is the intensity distribution of the target vortex beam (ground truth). The third and fourth columns are the intensity distributions of the compensated vortex beam by CycleGAN and improved CycleGAN (CycleGAN + SSIM_Loss), respectively. It can be observed that the intensity distribution of the compensated vortex beam is very close to that of the target vortex beam. In addition, when the SSIM loss function term is added to the loss function of CycleGAN, namely CycleGAN + SSIM_Loss, the intensity distribution of the compensated vortex beam is much closer to ground truth and provides shaper edges for the petal-like intensity distribution, which is very important for the subsequent OAM state detection.

The differences between the intensity distributions of the compensated vortex beam and the target vortex beam are shown in Figure 8. The turbulence parameters are the same as in Figure 7. The OAM state is {$1,-1$}, {$1,-2$}, {$3,-3$} and {$6,-6$}. The first column is the intensity distribution of the vortex beam captured at the receiving end, and the second column is the intensity distribution of the target vortex beam (ground truth). Meanwhile, the third and fourth columns are the differences of CycleGAN and CycleGAN + SSIM_Loss compared with the ground truth, respectively. Notably, the more the pixel points of the difference image are biased towards black, the closer the two images are, and the better the distortion compensation is. Therefore, the improved CycleGAN (CycleGAN + SSIM_Loss) has a good compensation performance for subsequent OAM communication and detection.

AT adds a perturbation phase to the vortex beam during transmission, causing phase distortion and intermodal crosstalk, which results in reduced mode purity. In addition to comparing the intensity distribution of the vortex beam before and after compensation, the effect of distortion compensation can also be tested by comparing the OAM probability distribution of the detected vortex beam before and after compensation. Under three atmospheric refractive index structure constants, namely $1.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, $3.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$ and $5.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, the distortion compensation model trained in medium turbulence is selected to compensate. Meanwhile, atmospheric refractive index structure constants $1.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$, $3.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ and $5.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ are selected for the strong turbulence experiment.

The OAM state detection network selects the basic CNN network. The data set is constructed by the 10 superimposed vortex beams’ light intensity distribution of {$1,-1$}, {$1,-2$}, {$2,-2$}, {$2,-3$}, {$3,-3$}, {$3,-4$}, {$4,-4$}, {$4,-5$}, {$5,-5$} and {$6,-6$}. When the transmission distance is 4000 m, the OAM state detection model is trained under six different turbulence intensities ($C_n^2$: $1.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, $3.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, $5.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, $1.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$, $3.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ and $5.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$). Under each turbulence intensity, 2000 light intensity distributions are simulated for each OAM state. The data set contains a total of 20,000 light intensity distributions of the vortex beam and is divided into a training set and test set in the ratio of 8:2.

The transmitter emits 5000 vortex beams with OAM = {4,$-4$}, and the CCD camera at the receiving end captures the light intensity distribution of vortex beams. The OAM state probability distributions before and after compensation are shown in Figure 9 based on the end-to-end turbulence distortion compensation scheme proposed in this paper, under two different $C_n^2$ scenarios, with a 4000 m transmission distance. It can be seen from Figure 9a that when $C_n^2=3.0\times {10}^{-14} {\mathrm{m}}^{-2/3}$, the probability of OAM = {4,$-4$} before and after compensation is above 96.5%. Although it increases slightly after compensation, the difference is not significant. In the strong turbulent environment, $C_n^2=3.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$ shown in Figure 9b, the probability of OAM = {4,$-4$} increased by 4.6% compared with that without compensation, and it further increased by 1.9% after adding the SSIM loss function. Moreover, the comparison between Figure 9a,b shows that the probability of OAM = {4,$-4$} is significantly reduced in strong turbulence with $C_n^2=3.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$, and the dispersion of the OAM state is more serious. It also demonstrated the necessity of distortion compensation for communication and detection in strong turbulence.

The comparison of the OAM state detection accuracy before and after compensation is shown in Figure 10. The detection accuracy of the OAM state was obviously improved under different turbulence intensity after compensation. Especially in strong turbulence, when $C_n^2=5.0\times {10}^{-13} {\mathrm{m}}^{-2/3}$, the detection accuracy of the OAM state is only 85.5% without compensation, but the detection accuracy can be improved to 89.1% after CycleGAN compensation. After using CycleGAN + SSIM_loss compensation, we obtain a further 0.8% increase in the detection accuracy. Since the experiments in this paper are based on simulation data, the compensation effect will be more obvious in a real vortex optical communication system.

4. Conclusions

The paper proposes an end-to-end vortex beam turbulence distortion compensation method based on the improved CycleGAN, skipping wavefront reconstruction and directly training the mapping relationship between the light intensity distributions of distorted vortex beam and undistorted target vortex beam. Whether it is affected by turbulence distortion is regarded as two image styles, turning the vortex beam turbulence distortion compensation problem into an image-to-image translation problem without the need for probe beams and wavefronts reconstruction, which saves optical hardware and improves the compensation performance. The frameworks of Pix2pix and CycleGAN were analyzed, and the SSIM loss function was introduced to assess the image quality from luminance, contrast and structure. The feasibility of the distortion compensation scheme and the compensation performance proposed in this paper was verified according to two aspects: light intensity image comparison and difference before and after compensation, and the MSSIM and OAM state recognition accuracy with and without compensation. The results show that the intensity distribution of the compensated vortex beam of the improved CycleGAN is very close to that of the undistorted target vortex beam, and the MSSIM is close to 1. In addition, the detection accuracy of OAM using CycleGAN + SSIM_loss is improved by 4.4% compared with that before compensation in strong turbulence. This research provides a potential basis for the vortex optical communication applications in interference environments.