A Poisson Shot Noise Limited MMSE Precoding for Photon-Counting MIMO Systems with Reinforcement Learning

Abstract

With the development of the Internet of Things (IoT), most communication systems are difficult to implement on a large scale due to their high complexity. Multiple-input multiple-output (MIMO) precoding is a generally used technique for improving the reliability of free-space optical (FSO) communications, which is a key technology in the 6G era. However, traditional MIMO precoding schemes are typically designed based on the assumption of additive white Gaussian noise (AWGN). In this paper, we present a novel MIMO precoding method based on reinforcement learning (RL) that is specifically designed for the Poisson shot noise model. Unlike traditional MIMO precoding schemes, our proposed scheme takes into account the unique statistical characteristics of Poisson shot noise. Our approach achieves significant performance gains compared to existing MIMO precoding schemes. The proposed scheme can achieve the bit error rate (BER) of ${10}^{-5}$ in a strong turbulence channel and exhibits superior robustness against imperfect channel state information (CSI).

1. Introduction

With the massive growth of data transmission, the future Internet of Things (IoT) will face more severe challenges. Free-space optical (FSO) communication is crucial in the forthcoming 6G era, with its characteristics of wide bandwidth, high speed, and high capacity [1]. Multiple-input multiple-output (MIMO) technology has been proven by multiple studies to be equally applicable to FSO systems. In FSO communication, MIMO is applied to reduce turbulence-induced fading. Instead of using a single transmitter and receiver, FSO MIMO systems employ multiple transmitters and receivers along with an array of optical elements such as laser diodes and detectors. These elements are strategically positioned to create multiple optical channels for data transmission. The application of MIMO to FSO systems offers significant advantages in terms of data rate, reliability, and robustness [2,3].

Ultraviolet (UV) communication has the characteristics of support for non-line-of-sight (NLOS) communication [4], and good confidentiality performance, which has attracted extensive attention. UV communication can be utilized in various scenarios. On-the-move communication is made possible through UV-C links, enabling devices in vehicles or drones to communicate seamlessly. Additionally, UV communication proves beneficial in IoT applications, facilitating communication between devices in harsh environments. Moreover, UV communication enables machine-to-machine communication, allowing machines or sensors to exchange data even in environments with high interference or restricted line-of-sight [5].

The atmospheric turbulence effect caused by random fluctuations in the atmosphere can seriously affect the communication quality of FSO systems. To solve these problems, precoding techniques are used at the transmitter to combat turbulence [6,7,8]. However, it should be pointed out that traditional precoding methods are derived from the additive white Gaussian noise (AWGN) model. Optical communication systems use light to transmit data, and light consists of discrete particles known as photons. The arrival of photons at the receiver’s detector follows a Poisson process, where photons are emitted randomly over time. The Poisson shot noise model accurately captures this discrete nature of optical signals, making it a better fit for optical communication systems. In other words, the optical communication models are generally modeled as signal-related Poisson shot noise [9,10]. Precoders and detectors designed under the AWGN model are no longer suitable for photon-counting communication systems. The precoder and combiner should be redesigned for the Poisson shot noise model.

The studies regarding the combination of communication and artificial intelligence (AI) have attracted widespread attention from scholars [11,12,13], which is a novel way to solve problems in mobile communication. The authors of [11] jointly design the active beamforming and passive beamforming to maximize the sum rate, using the deep deterministic policy gradient (DDPG) algorithm. The action space of this algorithm is designed by the beamforming matrix and the phase shift matrix. In [12], the authors utilize soft actor–critic (SAC) algorithm to design active analog precoder and passive beamformer. The authors of [13] propose a DRL-based precoding framework in both codebook-based and non-codebook-based MIMO precoding systems and examine the performance of the DQN and DDPG algorithms. The advantage of combining reinforcement learning (RL) and mobile communication is that different features can be extracted from a large number of raw data, and by learning to continuously adjust the parameter settings in the internal structure, RL can flexibly approximate the mathematical model of the simulated communication environment and deal with some complex physical channels. RL is applied in practice in various domains such as robotics, navigation, and smart grids. In robotics, RL can be used to train robotic arms to perform tasks like opening doors and picking up objects. RL is also used in navigation systems to optimize routes and make decisions based on real-time data. However, there are some potential challenges in applying RL. One challenge is the long time it takes for RL algorithms to converge and learn something meaningful. This restricts the use of RL techniques in real-time learning scenarios. Another challenge is the need for large numbers of data for training RL models. RL algorithms require extensive exploration of the environment to learn optimal policies, which can be time-consuming and resource-intensive. Additionally, RL algorithms may struggle with partial observability and uncertainty in complex environments, which can affect their performance and ability to make accurate decisions [14,15].

The non-convex problem and coupling constraints, as studied in this paper, pose considerable challenges to finding an optimal solution. In order to address this, we propose a novel MIMO precoding scheme based on RL, which provides a highly effective approach for jointly optimizing both the precoding and detection matrix of the transceiver and receiver, respectively. The proposed RL-based solution offers a number of key advantages over traditional optimization methods, such as greater flexibility and the adoption of the variation of system dynamics. Specifically, the RL framework allows the system to learn from past experiences and optimize the transmission and reception processes in an iterative and adaptive manner. Overall, our findings highlight the potential of RL-based methods for addressing complex optimization problems in FSO communication systems.

In this study, a joint precoder and combiner optimization based on RL is proposed. We analyze the bit error rate (BER) performance at different system parameters and compare it with the conventional schemes. The simulation results show that our scheme is able to obtain a lower BER performance compared to the conventional scheme. The main contributions of this research include:

1.

Our proposal involves the introduction of a state-of-the-art NLOS UV MIMO system, which is capable of photon-counting while operating under the influence of Poisson shot noise. To optimize the system performance, we formulate a novel optimization problem that addresses the design of both the precoding and combining matrices, utilizing the well-established minimum mean square error (MMSE) criterion.

2.

By employing a methodical approach to deriving and reformulating key aspects of our expected MMSE precoding design, we successfully transform it into an optimization problem that is conducive to efficient computation using the multi-agent deep deterministic policy gradient (MADDPG) algorithm [16]. Through this novel approach, we are able to attain the global optimal precoding matrix and combining matrix.

3.

The proposed system exhibits strong robustness. To evaluate the efficacy of our approach in realistic settings, we conducted experiments considering varying degrees of channel state information (CSI). The simulation results indicate our scheme is capable of achieving a BER of less than ${10}^{-4}$ even when the CSI is imperfect.

The rest of this paper is organized as follows. The related work is presented in Section 2. Section 3 introduces the system model. In Section 4, we present the RL-based precoder and combiner design, in which a specific expression of the optimization problem is given by a detailed mathematical derivation. In Section 5, we propose a method to solve the optimization problem based on the MADDPG algorithm. The simulation results are given in Section 6. Finally, we conclude in Section 7.

2. Related Work

Recently, various studies have been published in the area of photon-counting systems. In [17], the design of a MIMO system under the Poisson model is considered, in which a pulse-position modulation (PPM) modulation is used at the transmitter side, a maximum likelihood (ML) detection algorithm under the Poisson model is derived, and optimal as well as sub-optimal decoders are given. In [18], a receiver based on the linear least mean square error (MSE) criterion for FSO communication is presented. The system BER performance is analyzed under on–off keying (OOK) and PPM modulation. In [19], the statistical behavior of underwater fading with different probability density functions (PDFs) is studied. In [20], the authors present a composite quantum iterative multistage measurer and MIMO detector at the receiver. In [21], the analytical error probabilities for ML detection in turbulent and non-turbulent cases are derived. In [22], to minimize the probability of detection errors given relay forwarding power budget, a counting and forward relay framework for NLOS communication is presented. In [23], the communication capacity and performance of the Poisson model are investigated. And an upper limit of capacity and a lower limit of error probability are proposed. In [24], the photonic information rate is investigated for a single-photon avalanche diode (SPAD) array, and the effect of dead time on the system is considered. The results show that the photon-counting distribution can be regarded as a Gaussian distribution for sufficiently large arrays.

The conventional millimeter-wave hybrid precoding design method mainly decouples the hybrid precoding problem into transmitter-side hybrid precoding design and receiver-side hybrid combiner design, and it regards these two subproblems as matrix decomposition problems, respectively. The hybrid precoding problem is solved by minimizing the Euclidean distance between the product of the analog and digital matrices and the optimal all-digital precoding matrix under unconstrained conditions. In [25], a hybrid precoding method based on the alternating directional multiplier method (ADMM) is proposed. In [26], the symbol-level precoding of a MU-MISO downlink system is investigated. In [27], the objective function of the considered precoding scheme is highly nonconvex and has some complicated constraints, and the authors design a new method based on the ADMM. The convergence conditions of the proposed method are given by considering the order of iterations of the variables. In [28], the design of the symbol-level precoder for MU-MISO downlink communication is investigated, and its decision boundary is studied through the minimum maximum fairness design. To deal with this problem, the dimensionality of the variables is firstly reduced by solving a relaxation problem, after which the ADMM framework is used to efficiently solve the problem.

UV has the characteristics of high interference immunity, good confidentiality, and support for non-visual transmission, thus attracting the attention of a large number of researchers. In [29], the authors calculate the optical loss in the FSO communication system operating in the non-visual range, where scattering effects were taken into account, and the obtained results were compared with experimental data at 265 nm (solar-blind UV region). In [30], a 1 × 4 communication system was built. The experimental results show that equal-gain combining can provide significant diversity gain, provided that the transmit elevation angle is small or the transceiver distance is short. In [31], wireless UV communication is applied to UAV communication systems to solve the problem of directional awareness. In [32], the spatial diversity technique for the NLOS UV system is investigated, and the BER for different transmitter and receiver configurations is derived.

Recently, the concept of intelligent communication has attracted a considerable amount of attention, and its use of machine learning-based methods to deal with optimization problems in communication systems has achieved superior performance. In [33], an optimal hybrid precoding scheme based on hybrid cross entropy (HCE) is designed to maximize the total achievable rate. In [34], the authors design the selection and precoding matrices jointly for millimeter-wave systems. The proposed framework contains a neural network (NN) based on deep reinforcement learning (DRL) and a deep deployment NN. In [35], a user grouping algorithm based on channel gain and correlation is designed, and then a beam space orthogonal simulation precoder is obtained using DRL-based beam selection.

3. System Model

This section introduces the theoretical basis of the photon-counting MIMO system; the system model of each technology will be further explained below.

As presented in Figure 1, the considered photon-counting MIMO downlink system has $N_t$ light-emitting diodes (LEDs) in the base station (BS) that transmit $N_s$ data streams collaboratively to the $N_r$ photo detector (PD). Assume that the number of data streams and PDs is identical ($N_s=N_r$) and that the data streams are uncorrelated. The precoded signal is $\mathbf{W}\mathit{s}$, where $\mathit{s}={\left[s_1,\cdots ,s_{N_s}\right]}^T$ represents the data streams, and $\mathbf{W}\in {\mathbb{R}}^{N_t\times N_s}$ is the precoding matrix. To ensure the non-negativity of the signals, a direct current (DC) bias $\mathit{b}={\left[b_1,\cdots ,b_{N_t}\right]}^T$ is appended to the transmit symbols through bias-ttee circuits. Therefore, the input signal of the LED is $\mathit{x}=\mathbf{W}\mathit{s}+\mathit{b}$, driving the LEDs to transmit the signal through the FSO turbulent fading channels.

Free-space light transmits data to the PD through an optical medium via LEDs. FSO systems encounter several challenges during transmition, such as shadows when the lighting location is blocked from an area, which leads to information loss. The channel irradiance between the i-th PD and the j-th LED is denoted by $g_{i,j}$, which obeys the gamma–gamma distribution with the PDF:

$$\mathrm{Pr}\left(g_{i,j}\right)=\frac{2{\left(\alpha \beta \right)}^{\frac{\left(\alpha +\beta \right)}{2}}}{\gamma \left(\alpha \right)\gamma \left(\beta \right)}{g_{i,j}}^{\frac{\left(\alpha +\beta \right)}{2}-1}{K}_{\alpha -\beta }\left(2\sqrt{\alpha \beta g_{i,j}}\right),g_{i,j}>0,$$

(1)

where $\gamma \left(·\right)$ is the standard gamma function, and $\alpha ,\beta$ are the distribution shaping parameters satisfying:

$$\alpha ={\left[exp\left(\frac{0.49{\chi }^2}{{\left(1+0.18d^2+0.56{\chi }^{12/5}\right)}^{7/6}}\right)-1\right]}^{-1},$$

(2)

$$\beta ={\left[exp\left(\frac{0.51{\chi }^2{\left(1+0.69{\chi }^{12/5}\right)}^{-5/6}}{{\left(1+0.9d^2+0.62d^2{\chi }^{12/5}\right)}^{5/6}}\right)-1\right]}^{-1},$$

(3)

where $d={\left(k{D}^2/4L\right)}^{1/2}$, ${\chi }^2=0.5C_n^2{k}^{7/6}{L}^{11/6}$, $k=2\pi /\lambda$ is the light wave factor; $\lambda$ represents the wavelength; D is the diameter of the aperture of the condenser at the receiver; L is the distance of the link; and $C_n^2$ represents the index value of the refractive structure parameter and is a function of altitude h.

The i-th PD’s channel matrix is ${\mathit{g}}_i=\left[g_{1,1},\cdots ,g_{1,N_t}\right]$; hence, the channel matrix between the user and the BS is denoted by

$$\mathbf{G}={\left[{\mathit{g}}_1^T,\cdots ,{\mathit{g}}_{N_r}^T\right]}^T.$$

(4)

The photon-counting process model is adopted at the receiver. The random variable corresponding to the number of photons detected by the j-th PD, denoted by $z_j$, follows the Poisson distribution with the PDF:

$$\mathrm{Pr}\left(z_j|\mathit{s}\right)=\frac{{\left[{\mathit{g}}_j\left(\mathbf{W}\mathit{s}+\mathit{b}\right)+n_b\right]}^{z_j}}{z_j!}{e}^{-\left[{\mathit{g}}_j\left(\mathbf{W}\mathit{s}+\mathit{b}\right)+n_b\right]}.$$

(5)

In (5), $n_b=\eta \frac{P_{b}T}{hf}$ is the number of noise photons generated by background radiation, where $\eta$ is the PD efficiency, $P_b$ is the background radiation power, T is the duration of a symbol, h is Planck’s constant, and f is the center frequency.

After removing the DC bias, the received signal follows:

$$\mathrm{Pr}\left(y_j=r_j|\mathit{s}\right)=\frac{{\left({\mathit{g}}_j\mathbf{W}\mathit{s}+n_b\right)}^{r_j}}{r_j!}{e}^{-\left({\mathit{g}}_j\mathbf{W}\mathit{s}+n_b\right)}.$$

(6)

The received signal from the transmitter is typically subject to various forms of interference and distortion during propagating through the channel. This results in a degraded signal at the receiver, which requires further promotion in order to extract useful information. Hence, a combiner matrix is typically used to combine the received signals from PD at the receiver. The combiner matrix ${\mathbf{F}}^T\in {\mathbb{R}}^{N_s\times N_r}$ is used to combine the received signal across $N_r$ PDs.

Assuming independent numbers of received photons across all PDs, the combined signal ${\mathbf{F}}^T\mathbf{y}$ can be used to estimate the coded bit. This involves decoding the received signal and extracting the transmitted information from it.

4. Problem Formulate

In this section, we design the precoder and combiner according to the MMSE principle with detailed mathematical derivation and propose the form of the optimization problem. By solving this optimization problem, we are able to derive the optimal precoder and combiner that minimize the MSE of the received and transmitted signals.

We take the MSE as the performance measure and optimization objective for the joint precoder and combiner design, which is defined as

$$\begin{array}{cc}\hfill J_{\mathrm{MSE}}& =\mathbb{E}\left\{\left|\right|{\mathbf{F}}^T\mathit{y}-\mathit{s}{\left|\right|}^2\right\}\hfill \\ \hfill & =\mathbb{E}\left\{\mathrm{tr}\left({\mathbf{F}}^T\mathit{y}-\mathit{s}\right){\left({\mathbf{F}}^T\mathit{y}-\mathit{s}\right)}^T\right\}\hfill \\ \hfill & =\mathrm{tr}\left\{{\mathbf{F}}^T\mathbb{E}\left({\mathit{yy}}^T\right)\mathbf{F}-2{\mathbf{F}}^T\mathbb{E}\left({\mathit{ys}}^T\right)\right\}+\Theta ,\hfill \end{array}$$

(7)

where the expectation is taken over variables $\mathit{y}$ and $\mathit{s}$. Consider OOK modulation in the BS, and equal probability for the code bit $\mathit{s}$, i.e., $\mathrm{Pr}\left(s=-1\right)=\mathrm{Pr}\left(s=1\right)=\frac{1}{2}$ [36]. Hence,

$$\Theta =\mathrm{tr}\left\{\mathbb{E}\left({\mathit{ss}}^T\right)\right\}=N_s.$$

(8)

Define the correlation matrix ${\mathbf{R}}_{\mathit{yy}}=\mathbb{E}\left({\mathit{yy}}^T\right)\in {\mathbb{R}}^{N_r\times N_r},{\mathbf{R}}_{\mathit{ys}}=\mathbb{E}\left({\mathit{ys}}^T\right)\in {\mathbb{R}}^{N_r\times N_s}$. Then, we have

$$\begin{array}{c}\hfill J_{\mathrm{MSE}}=\mathrm{tr}\left\{{\mathbf{F}}^T{\mathbf{R}}_{\mathit{yy}}\mathbf{F}-2{\mathbf{F}}^T{\mathbf{R}}_{\mathbf{zs}}\right\}+N_s.\end{array}$$

(9)

First, we have that ${\mathbf{R}}_{\mathit{yy}}$ is given by

$$\begin{array}{c}\hfill {\mathbf{R}}_{\mathit{yy}}=\mathbb{E}\left[\begin{array}{cccc}{y}_1^2& y_1{y}_2& \cdots & y_1{y}_{N_r}\\ y_2{y}_1& y_2^2& \cdots & y_2{y}_{N_r}\\ ⋮& ⋮& \ddots & ⋮\\ y_{N_r}{y}_1& y_{N_r}{y}_2& \cdots & y_{N_r}^2\end{array}\right].\end{array}$$

(10)

For $1⩽i⩽N_r$, we have

$$\begin{array}{c}\hfill \mathbb{E}\left(y_i^2\right)=\sum _{\mathit{s}}\mathbb{E}\left(y_i^2|\mathit{s}\right)\mathrm{Pr}\left(\mathit{s}\right).\end{array}$$

(11)

Define a set ${\mathcal{S}}_0=\{s^{\left(j\right)}\in \{-1,1\},j=1,\cdots ,2^{N_s}\}$ with alphabet size $2^{N_s}$, where each element of ${\mathcal{S}}_0$ has the a priori probability $1/2^{N_s}$. Then, (11) can further expand as

$$\begin{array}{c}\hfill \mathbb{E}\left(y_i^2\right)=\sum _{j=1}^{2^{N_s}}\mathbb{E}\left(y_i^2|{\mathit{s}}^{\left(j\right)}\right)\mathrm{Pr}\left({\mathit{s}}^{\left(j\right)}\right).\end{array}$$

(12)

Based on (6), we have $\mathbb{E}\left(y_i|{\mathit{s}}^{\left(j\right)}\right)={\mathit{g}}_i\mathbf{W}{\mathit{s}}_j+n_b$. Since $\mathbb{E}\left(X^2\right)={\mathbb{E}}^2\left(X\right)+\mathrm{Var}\left(X\right)$ for a randome varaible X, (12) is equal to

$$\begin{array}{cc}\hfill \mathbb{E}\left(y_i^2\right)& =\sum _{j=1}^{2^{N_s}}\left\{{\left[{\mathit{g}}_i\mathbf{W}{\mathit{s}}^{\left(j\right)}+n_b\right]}^2+{\mathit{g}}_i\mathbf{W}{\mathit{s}}^{\left(j\right)}+n_b\right\}\mathrm{Pr}\left({\mathit{s}}^{\left(j\right)}\right)\hfill \\ \hfill & =\frac{1}{2^{N_s}}\sum _{j=1}^{2^{N_s}}\left\{{\left[{\mathit{g}}_i\mathbf{W}{\mathit{s}}^{\left(j\right)}+n_b\right]}^2+{\mathit{g}}_i\mathbf{W}{\mathit{s}}^{\left(j\right)}+n_b\right\}\hfill \\ \hfill & ={\mathit{g}}_i{\mathbf{WW}}^T{\mathit{g}}_i^T+n_b^2+n_b.\hfill \end{array}$$

(13)

Note that we have used the fact that ${\sum }_{j=1}^{2^{N_s}}{\mathit{s}}^{\left(j\right)}{\left[{\mathit{s}}^{\left(j\right)}\right]}^T=2^{N_s}{\mathbf{I}}_{N_s}$ and ${\sum }_{j=1}^{2^{N_s}}{\mathit{s}}^{\left(j\right)}=\mathbf{0}$ based on our definition.

For $1⩽i⩽N_r,1⩽j⩽N_r,i\ne j$, we have

$$\begin{array}{c}\hfill \mathbb{E}\left(y_i{y}_j\right)=\mathbb{E}\left(y_i\right)\mathbb{E}\left(y_j\right)=\sum _{\mathit{s}}\mathbb{E}\left(y_i|\mathit{s}\right)\mathrm{Pr}\left(\mathit{s}\right)\sum _{\mathit{s}}\mathbb{E}\left(y_j|\mathit{s}\right)\mathrm{Pr}\left(\mathit{s}\right)=0.\end{array}$$

(14)

According to (13) and (14), we have that

$$\begin{array}{cc}\hfill {\mathbf{R}}_{\mathit{yy}}& =\left[\begin{array}{cccc}{\mathit{g}}_1{\mathbf{WW}}^T{\mathit{g}}_1^T+n_b^2+n_b& 0& \cdots & 0\\ 0& {\mathit{g}}_2{\mathbf{WW}}^T{\mathit{g}}_2^T+n_b^2+n_b& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & {\mathit{g}}_{N_R}{\mathbf{WW}}^T{\mathit{g}}_{N_r}^T+n_b^2+n_b\end{array}\right]\hfill \\ \hfill & =\mathrm{diag}\left(\mathrm{diag}\left({\mathbf{GWW}}^T{\mathit{g}}^T+n_b^2+n_b\right)\right).\hfill \end{array}$$

(15)

Next, ${\mathbf{R}}_{\mathit{ys}}$ is given by

$$\begin{array}{c}\hfill {\mathbf{R}}_{\mathit{ys}}=\mathbb{E}\left(\begin{array}{cccc}{y}_1{s}_1& y_1{s}_2& \cdots & y_1{s}_{N_s}\\ y_2{s}_1& y_2{s}_2& \cdots & y_2{s}_{N_s}\\ ⋮& ⋮& \ddots & ⋮\\ y_{N_r}{s}_1& y_{N_r}{s}_2& \cdots & y_{N_r}{s}_{N_s}\end{array}\right).\end{array}$$

(16)

For $1⩽i⩽N_r,1⩽j⩽N_s$, we have

$$\begin{array}{cc}\hfill \mathbb{E}\left(y_i{s}_j\right)& =\mathbb{E}\left(y_i{s}_j|s_j=-1\right)\mathrm{Pr}\left(s_j=-1\right)+\mathbb{E}\left(y_i{s}_j|s_j=1\right)\mathrm{Pr}\left(s_j=1\right)\hfill \\ \hfill & =-\frac{1}{2}\mathbb{E}\left(y_i|s_j=-1\right)+\frac{1}{2}\mathbb{E}\left(y_i|s_j=1\right)\hfill \\ \hfill & ={\mathit{g}}_i{\mathbf{We}}_j.\hfill \end{array}$$

(17)

Note that we use the conclusion that $\mathrm{E}\left(y_i|s_j=1\right)={\mathit{g}}_i{\mathbf{We}}_j+n_b$ and $\mathrm{E}\left(y_i|s_j=-1\right)=-{\mathit{g}}_i{\mathbf{We}}_j+n_b$, and the proof is given in Appendix A. According to (17), we have that

$$\begin{array}{cc}\hfill {\mathbf{R}}_{\mathit{ys}}& =\left(\begin{array}{cccc}{\mathit{g}}_1{\mathbf{We}}_1& {\mathit{g}}_1{\mathbf{We}}_2& \cdots & {\mathit{g}}_1{\mathbf{We}}_{N_s}\\ {\mathit{g}}_2{\mathbf{We}}_1& {\mathit{g}}_2{\mathbf{We}}_2& \cdots & {\mathit{g}}_2{\mathbf{We}}_{N_s}\\ ⋮& ⋮& \ddots & ⋮\\ {\mathit{g}}_{N_r}{\mathbf{We}}_1& {\mathit{g}}_{N_r}{\mathbf{We}}_2& \cdots & {\mathit{g}}_{N_r}{\mathbf{We}}_{N_s}\end{array}\right)=\mathbf{GW}.\hfill \end{array}$$

(18)

According to (8), (15), and (18), the closed form of the MSE is

$$\begin{array}{c}\hfill J_{\mathrm{MSE}}=\mathrm{tr}\left\{{\mathbf{F}}^T\mathrm{diag}\left(\mathrm{diag}\left(\mathit{g}{\mathbf{WW}}^T{\mathit{g}}^T+n_b^2+n_b\right)\right)\mathbf{F}-2{\mathbf{F}}^T\mathbf{GW}\right\}+N_s.\end{array}$$

(19)

In order to make the LED work within a linear dynamic range, the transmit signal satisfies

$$\begin{array}{c}\hfill I_L⩽e_i^T\mathbf{W}\mathit{s}+b_i⩽I_H,\end{array}$$

(20)

where $I_L$ and $I_H$ represent the minimum and maximum currents corresponding to the linear dynamic range, respectively. Since

$$\begin{array}{c}\hfill -\left|\right|e_i^T{\mathbf{W}\left|\right|}_1⩽e_i^T\mathbf{W}\mathit{s}⩽\left|\right|e_i^T\mathbf{W}{\left|\right|}_1,\end{array}$$

(21)

adding DC bias to (21) yields

$$\begin{array}{c}\hfill -\left|\right|e_i^T{\mathbf{W}\left|\right|}_1+b_i⩽e_i^T\mathbf{W}\mathit{s}+b_i⩽\left|\right|e_i^T\mathbf{W}{\left|\right|}_1+b_i.\end{array}$$

(22)

To satisfy the constraint in (20), we have

$$\begin{array}{c}\hfill \left\{\begin{array}{c}-\left|\right|e_i^T\mathbf{W}{\left|\right|}_1+b_i⩾I_L\hfill \\ \left|\right|e_i^T\mathbf{W}{\left|\right|}_1+b_i⩽I_H,\hfill \end{array}\right.\end{array}$$

(23)

(23) can be further expressed as

$$\begin{array}{c}\hfill \left|\right|e_i^T\mathbf{W}{\left|\right|}_1⩽D,\forall i=1,\cdots ,N_T,\end{array}$$

(24)

where D is given by

$$\begin{array}{c}\hfill D=min\left(I_H-b_i,b_i-I_L\right).\end{array}$$

(25)

The optimization problem can thus be formulated as

$$\begin{array}{cc}\hfill \underset{\mathbf{W},{\mathbf{F}}^T}{\arg }min& \mathrm{tr}\left\{{\mathbf{F}}^T\mathrm{diag}\left(\mathrm{diag}\left(\mathit{g}{\mathbf{WW}}^T{\mathit{g}}^T+n_b^2+n_b\right)\right)\mathbf{F}-2{\mathbf{F}}^T\mathbf{GW}\right\}\hfill \\ \hfill \mathrm{s}.\mathrm{t}.& \left|\right|e_i^T\mathbf{W}{\left|\right|}_1⩽D,\forall i=1,\cdots ,N_t\hfill \\ \hfill & \left|\right|{\mathbf{e}}_i^T\mathbf{W}{\left|\right|}_1+b_i⩽\frac{\eta P_{max}T}{h\upsilon },\forall i=1,\cdots ,N_t,\hfill \end{array}$$

(26)

where $P_{max}$ is the maximum linear optical power of the LEDs. For optical communications, the transmit signals must be non-negative. The emitting power of the LED also needs to be within its dynamic range [$P_{min}$, $P_{max}$]. These factors lead to the constrains.

5. Precoder and Combiner Design Based on Reinforcement Learning

The optimization problem mentioned in Equation (26) is obviously a NP-hard problem, which cannot be solved via conventional optimization solvers. Hence, we proposed an RL-based approach to jointly solve the precoding matrix and the combining matrix. Several researchers of beamforming employed the deep Q network (DQN) to find optimal solutions. DQN is designed to solve tasks with discrete action space. To utilize DQN in continuous action space, we need to discretize the continuous action space, which will make the action space grow exponentially with the size of network [37]. While the DDPG algorithm can be used in continuous action space, the MADDPG can in addition reduce the action dimension of single agent. In the case of the joint design of the precoding matrix and combining matrix, MADDPG may be appropriate as it allows for decentralized decision-making while considering global state information. MADDPG enables each agent to interact with the environment and learn its own strategy based on local observations and rewards while also considering the joint actions and rewards of other data streams. The choice of MADDPG for jointly designing precoding and combining matrices may be based on its ability to handle multi-agent scenarios, its abikity to handle continuous action spaces, and its use of target networks and experience replay for stability and efficiency. To employ an RL method, we model the process of generating the precoding and combining matrix into the Markov decision process (MDP). Then, we apply the MADDPG algorithm to explore the optimal policy of the MDP. The elements of MDP and the MADDPG algorithm will be presented in the following, respectively. In the context of optimizing precoding and combining matrices, MADDPG can be used to train a network of agents, where each agent represents a communication node equipped with multiple antennas. The agents collaborate to learn the optimal strategies for precoding and combining matrices, taking into account the interactions and dependencies among the nodes.

Figure 2 is the framework of the MADDPG-based joint precoding and combing algorithm, which consists of two sets of the same number of agents and an interactive environment. The agents receive the states and rewards transmitted by the environment and output corresponding actions, which are recombined into a precoding/combing matrix, respectively, allowing the environment to compute and output new states and rewards.

5.1. Agent

Since the dimensions of the precoding and combining matrices to be optimized do not coincide, two sets of agents with different action dimensions are set to form the part of solving the precoding and combining matrices. For the state of each agent, characterized as the L2-norm value of each row of the interactively generated precoding matrix and combine matrix, the set of state dimensions and the set of action dimensions of the agent are denoted as $di{m}_{state}=\left\{d_1^{state},d_2^{state},\cdots ,d_{2N_s}^{state}\right\}$ and $di{m}_{action}=\left\{d_1^{action},d_2^{action},\cdots ,d_{2N_s}^{action}\right\}$.

5.2. State Space

In the t-th iteration, given the environment state $s_t^i\in {\mathbb{R}}^{di{m}_i^{state}}$ of the current agent i, the joint state space of multiple agents at iteration step t is represented as ${\mathit{s}}_t=\left\{s_t^1,s_t^2,\cdots ,s_t^{2N_s}\right\}$. One precoding and combining agent only obtains its own state information, which is the L2-norm value of the row vector corresponding to the ${\mathbf{W}}^{\mathbf{T}}$ or ${\mathbf{F}}^{\mathbf{T}}$ matrix according to the agent allocation situation, while the states as well as actions information of other agents are unknown. That is, the state observed by the agent i is $s_t^i=\left\{\left|\right|{\mathbf{e}}_i\mathbf{W}\left|\right|\right\}$ or $s_t^i=\left\{\left|\right|{\mathbf{e}}_i\mathbf{F}\left|\right|\right\}$.

5.3. Action Space

The action output of one agent constitutes the elements of one row of the matrix. The action outputs of all $2N_s$ agents compose the whole matrix. The elements of the matrix are continuous real numbers, and in the agent action design, the action output range is [−1,1], that is, the matrix elements are normalized to the range [−1,1], and the action dimension of the agent is designed according to $di{m}_{action}$. In the t-th iteration, the actor network of the i-th agent outputs the action $a_t^i\in {\mathbb{R}}^{di{m}_t^{action}}$ according to the current policy. The joint action is denoted as ${\mathit{A}}_t=\left\{a_t^1,a_t^2,\cdots ,a_t^{2N_s}\right\}$. The agents interact with the environment through the joint action and obtain the reward ${\mathbf{R}}_t$ and the $(t+1)$-th state information ${\mathit{s}}_{t+1}$. For the reward, all of the agents obtain the same reward from the interaction with the system environment to prompt cooperative behaviors among the $2N_s$ agents.

5.4. Reward Function

The focus of whether RL can better solve the optimization problem lies in the reward function. In the precoding optimization problem of the downlink MIMO system, the joint optimization problem of the precoding matrix and combining matrix can be reduced to a minimization MSE problem. The optimization problem is shown in (26).

Due to the existence of constraints, the reward function designed in this paper is divided into two parts: one is the desired optimization objective, and another one is the penalty term for not satisfying the constraints. The first part of the reward function is

$$\begin{array}{c}\hfill R_1=\mathrm{tr}\left\{{\mathbf{F}}^T\mathrm{diag}\left(\mathrm{diag}\left(\mathit{g}{\mathbf{WW}}^T{\mathit{g}}^T+n_b^2+n_b\right)\right)\mathbf{F}-2{\mathbf{F}}^T\mathbf{GW}\right\}.\end{array}$$

(27)

The second part of the reward function is the penalty term, which is set to $-\beta$ for any constraint that is not satisfied. Since the MSE value should be positive, consider increasing the penalty when $MSE<0$. This penalty term is set to $-10\ast \beta$, which means that

$$\begin{array}{c}\hfill penalt{y}_1=\left\{\begin{array}{c}-\beta ,Constraint1notmet\\ 0,Constraint1met\end{array}\right.\end{array}$$

(28)

$$\begin{array}{c}\hfill penalt{y}_2=\left\{\begin{array}{c}-\beta ,Constraint2notmet\\ 0,Constraint2met\end{array}\right.\end{array}$$

(29)

$$\begin{array}{c}\hfill penalt{y}_{mse}=\left\{\begin{array}{c}-10\ast \beta ,ConstraintMSEnotmet\\ 0,ConstraintMSEmet\end{array}\right..\end{array}$$

(30)

The second part of the reward function can be expressed as

$$\begin{array}{c}\hfill R_2=penalt{y}_1+penalt{y}_2+penalt{y}_{mse}.\end{array}$$

(31)

In summary, the ultimate reward of MDP can be represented as the linear combination of optimization objective and the penalty term.

$$\begin{array}{c}\hfill Reward=R_1+R_2.\end{array}$$

(32)

5.5. Joint Precoding and Combining Algorithm Based on MADDPG

MADDPG is actor–critic algorithm that utilizes the idea of centralized training and decentralized decision-making. The actor makes a decision over time steps, while the critic evaluates the value of the decision. Assume the set $\mu =\left\{{\mu }_1,\cdots ,{\mu }_{2N_s}\right\}$ is the policies of all agents and set $\theta =\left\{{\theta }_1,\cdots ,{\theta }_{2N_s}\right\}$ as the parameters of corresponding policies. For the i-th agent’s policy ${\mu }_i$, the object function $\mathcal{I}\left({\theta }_i\right)$ is the expected reward. The gradient of the object function w.r.t ${\theta }_i$ can be depicted as:

$$\begin{array}{c}\hfill {\nabla }_{\left({\theta }_i\right)}\mathcal{I}\left({\mu }_i\right)={\mathbb{E}}_{\left(\mathit{S},\mathit{A}\sim \mathcal{D}\right)}\left[{\nabla }_{\left({\theta }_i\right)}{\mu }_i\left(a_i|s_i\right){\nabla }_{\left(a_i\right)}{Q}_i^{\mu }\left(\mathit{S},a_1,...,a_i\right)\right]{|}_{\left(a_i={\mu }_i\left(s_i\right)\right)},\end{array}$$

(33)

where $\mathit{S}=\left(s_1,\cdots ,s_{2N_s}\right)$, $\mathit{A}=\left(a_1,\cdots ,a_{2N_s}\right)$. $\mathcal{D}$ is the replay buffer, which stores the transition $\left(\mathit{S},\mathit{A},\mathit{R},{\mathit{S}}^{\prime }\right)$, where ${\mathit{S}}^{\prime }$ is the next time step state and $R=(r_1,...,r_{2Ns})$. $Q_i^{\mu }(\mathit{S},a_1,...,a_i)$ is the Q-value function. The Q-function can be updated by minimizing the function below:

$$\begin{array}{c}\hfill \mathcal{L}\left({\theta }_i\right)={\mathbb{E}}_{\mathit{S},\mathit{A},\mathit{R},{\mathit{S}}^{\prime }}\left[{\left(Q_i^{\mu }\left(\mathit{S},a_1,...,a_i\right)-y\right)}^2\right].\end{array}$$

(34)

In (34), $y=r_i+\gamma Q_i^{{\mu }^{\prime }}\left({\mathit{S}}^{\prime },a_1^{\prime },...,a_i^{\prime }\right){|}_{a_i^{\prime }={\mu }_i^{\prime }\left(s_i\right)}$, $\gamma$ is the Q-value discounting factor, and ${\mu }^{\prime }$ represents the policies with the delayed network parameter ${\theta }^{\prime }$. Each agent maintains two sets of actor–critic network pairs, known as the behavior pair and the target pair. Relative to the behavior pair, the target pair of networks makes a soft replacement with parameters. We show the algorithm as a pseudo-code type in Algorithm 1.

The hyperparameters of MADDPG are shown in

Table 1. Hyperparameters of MADDPG.

Parameter	Value
Episode M	600
Time step N	1000
Gaussian random process $\mathcal{N}$	$\mathcal{N}$ (0, 0.5)
Minibatch size	100
Discouting factor ${\rm Y}$	0.9
Soft replacement factor $\tau$	0.001
Learning rate of behavior critic	0.0001
Learning rate of behavior actor	0.00001
Replay buffer $\mathcal{D}$ size	100,000
Layers and Hiddern Units $8\times 2$	(4, 200)
Layers and Hiddern Units $6\times 2$, $4\times 2$	(3, 128)

.

Algorithm 1: MADDPG for joint precoding and combining optimization
1	Start;
2	Initialize the environment of downlink MIMO communication system;
3	Randomly initialize the weights of behavior actor–critic pair;
4	for episode = $1\to M$ do$\phantom{(}$
5			Initialize a Gaussian random process $\mathcal{N}$;
6			Reset the photon-counting MIMO downlink system environment and obtain initial global state $\mathit{s}$;
7			for $t=1\to N$ do
8					Each agent selects action $a_i={\mu }_i\left(s_i\right)+{\mathcal{N}}_t$;
9					Reformulate actions $\mathit{A}=\left(a_1,\cdots ,a_{2N_s}\right)$ into precoding matrix $\mathbf{W}$ and combining matrix ${\mathbf{F}}^T$ and obtain new state ${\mathit{S}}^{\prime }$;
10					Store $\left(\mathit{S},\mathit{A},\mathit{R},{\mathit{S}}^{\prime }\right)$ in replay buffer $\mathcal{D}$;
11					$\mathit{S}\leftarrow {\mathit{S}}^{\prime }$;
12					for agent $i=1\to 2N_s$ do
13							Randomly sample a minibatch of transitions $\left(\mathit{S},\mathit{A},\mathit{R},{\mathit{S}}^{\prime }\right)$ from $\mathcal{D}$;
14							Update the critic network by (34);
15							Update the actor network by (33);
16					Update the target network parameters of each precoding and combining agent i;
17					${\theta }_i^{\prime }\leftarrow \tau {\theta }_i+\left(1-\tau \right){\theta }_i^{\prime }$;
18	End;

6. Result and Discussion

In this section, we present the simulation results to estimate the BER performance of the proposed RL-based MMSE precoding scheme in the UV NLOS MIMO communication systems. For the simulation setup, the PD efficiency is set to $\eta =0.06$, and the wavelength of the UV light is set to $266$nm [38]. We use the gamma–gamma channel model as the channel fading model. We set background radiation fixed at −188.18 dBJ and increase ${}_{E}b$ from −160 dBJ to −155 dBJ. A single user is considered with two data streams. MIMO configurations with $2\times 8,2\times 6,2\times 4$ are set for simulation. Each data point in Figures 4–9 is the average of five independent tests, and different random seeds are used for the tests.

Figure 3 shows the average reward in the $95\%$ confidence interval for five runs to show statistical significance. As shown in Figure 3, the average reward converges with the number of training increases. This reward includes the optimization objective’s MSE and the penalty of constraint conditions. In the later stage of training, the penalty term is 0, and the entire reward represents MSE, which shows the effectiveness of our proposed scheme.

The impact of the number of LEDs is analyzed in Figure 4, where the background radiation is $n_b=-188.18$ dBJ. The increase in the number of LED arrays ($N_t$) helps to reduce the BER. In particular, for a $2\times 8$ MIMO configuration, the proposed scheme achieves an average BER of ${10}^{-5}$ at an energy per bit of $E_b=-155$ dBJ, providing a performance gain of about 2 dB over the $2\times 4$ system. This indicates that the proposed scheme can efficiently utilize the spatial diversity. The result suggests that the proposed scheme can provide reliable and efficient data transmission even under low receive energy conditions.

Figure 5 presents the performance comparison between our scheme and traditional AWGN-based precoding schemes in a severe fading channel with $N_t=8$ and $N_r=2$. For traditional schemes, we considered different schemes at the receiver side and transmitted side [39,40,41]: MMSE-MMSE, ZF-MMSE, and SVD-MMSE. The background radiation is $n_b=-188.18$ dBJ. It is clear that the AWGN-based MMSE precoding schemes perform poorly in the Poisson shot-noise photon-counting systems. The MMSE precoding scheme in the Gaussian system can only achieve a BER of ${10}^{-3}$ when $E_b=-155$ dBJ, while our proposed scheme can reach ${10}^{-5}$.

For the computational complexity analysis, we exploit the number of float number operations with Big-O notation. According to the formula, we can know that the computational complexity of the MMSE-MMSE, ZF-MMSE, and SVD-MMSE scheme is $\mathcal{O}\left(N_r^3+N_t{N}_r^2\right)$. Since the actor network of MADDPG uses the multi-layer perceptron structure, the computational complexity of the MADDPG generating precoding matrix and combining per execution for one agent is given by [42]:

$$\begin{array}{cc}\hfill & \mathcal{O}\left(N_s{N}_{eval}{N}_{step}\left(di{m}_{state}{n}_1+\sum _l^{L-1}{n}_l{n}_{l+1}+n_{L}di{m}_{action}\right)\right)\hfill \\ \hfill & =\mathcal{O}\left(N_s{N}_{eval}{N}_{step}\gamma \left(\frac{di{m}_{state}}{di{m}_{action}}+\left(L-1\right)\gamma +1\right){\left(di{m}_{action}\right)}^2\right)\hfill \\ \hfill & =\mathcal{O}\left(N_s{N}_{eval}{N}_{step}{\gamma }^2\left(L-1\right){\left(di{m}_{action}\right)}^2\right)\hfill \\ \hfill & =\mathcal{O}\left(N_s{N}_{eval}{N}_{step}{\gamma }^2\left(L-1\right)\left(N_t^2+N_r^2\right)\right),\hfill \end{array}$$

(35)

where $N_{eval}$ is the number of evaluation episodes; $di{m}_{action}$ > $di{m}_{state}$, $N_{step}$ is the step number; L is the number of hidden layers; and $n_l$ is the nodes number of l-th hidden layer. $n_l=\gamma di{m}_{action}$ for $1⩽l⩽L$. Note that $\gamma$ is a scaling factor, which depends on the dimension of states, for the hidden-layer nodes.

Figure 6 shows the BER of the proposed MMSE scheme under three different turbulence fading conditions, where $\alpha =2/3,1.5,2.1$ and $\beta =2/3,1.5,2.4$. It is observed that the proposed MMSE scheme could achieve a low BER as $E_b$ increases, under each of the turbulence fading conditions. However, under stronger turbulence, a higher $E_b$ is required to maintain the same BER target.

Figure 7 shows the BER performance of the proposed scheme with different DC bias. As shown in Figure 7, when the DC bias is 1, the BER cannot decrease. This is because the bias is too small to allow the LED to operate in a linear range. Compare the other three curves; it is obvious that a smaller DC bias tends to provide a better performance. This is because when the total power is given, the less the power is applied to biasing, the more power can be saved for transmitting useful signals.

In FSO communication systems, CSI plays a critical role in designing and optimizing transmission schemes. However, CSI estimation errors can occur due to various factors, such as outdated or inaccurate feedback from the users. To investigate the BER performance of the proposed MIMO precoding scheme in the presence of imperfect CSI, the channel irradiance is modeled as ${\widehat{g}}_{i,j}=\delta g_{i,j}$, where the estimation error $\delta$ is independently and uniformly randomly distributed within the interval $\left[-\delta ,\delta \right]$, and $\delta$ is the maximum error percentage. The CSI at the transmitter is modeled as a mapping $\mathit{g}\to \mathcal{G}$, where $\mathcal{G}$ is an arbitrary subset of ${\mathbb{R}}^{N_r\times N_t}$.

To evaluate the robustness of the proposed MIMO precoding scheme, Figure 8 shows the BER performance in the presence of different levels of CSI imperfection in a strong gamma–gamma fading channel, where $N_r=2$ and $N_t=8$. In Figure 8, the value of $\delta =1$ indicates perfect CSI, and it is observed that the proposed scheme is resilient to imperfect CSI. More specifically, it is clear that our scheme achieves a BER of ${10}^{-3}$ or lower at $E_b=-155$ dBJ even when the estimation error percentage is as high as $\delta =0.6$. These results indicate the robustness of the proposed precoding scheme in practical communication systems.

The background light intensity is a key factor affecting the performance of the FSO system. Figure 9 shows the system BER performance under different background light intensities (i.e., $-188.18$ dBJ, $-180$ dBJ, $-175$ dBJ, and $-170$ dBJ). It is clear that the proposed scheme can resist strong background light interference. Specifically, under the strong background light condition of $n_b=-175$ dBJ, a BER ${10}^{-4}$ can be achieved.

7. Conclusions

We presented a novel MIMO precoding scheme based on RL, jointly optimizing both the precoding and combining matrix of the transceiver and receiver, respectively. The simulation results indicate that the proposed RL-based MIMO precoding scheme outperforms existing methods under a wide range of operating conditions, delivering significant gains in terms of BER performance. The proposed system exhibits strong robustness and can achieve the BER of less than ${10}^{-4}$ even when the CSI is imperfect. The proposed solution offers a highly effective approach for jointly optimizing both the precoding and detection matrix of the transceiver and receiver.

The dead time in photon counting systems refers to the minimum time delay that must occur between the detection of two consecutive photons by a detector. During this dead time period, the detector is unable to register or detect any subsequent photons. This study does not consider the impact of dead time. In the future, we can analyze the system throughput with precoding and the impact of dead time. In addition, the system considered in this article is a single user system, which can also be extended to multi-user systems in the future. It is worth exploring how to eliminate the inter-user interference in multi-user systems by designing precoding and detection matrices.